A method and apparatus for estimating road surface adhesion coefficient based on image and point cloud fusion

Abstract

This invention discloses a method and apparatus for estimating road surface adhesion coefficient based on image and point cloud fusion, comprising: acquiring sensor signals from a vehicle and images and road surface point clouds in front of the vehicle; determining a region of interest (ROI) for road surface adhesion coefficient estimation based on the sensor signals and vehicle kinematics; performing a first estimation of the road surface adhesion coefficient based on the ROI and the image to obtain a first estimation result of the road surface adhesion coefficient; performing a second estimation of the road surface adhesion coefficient based on the ROI and the road surface point clouds to obtain a second estimation result of the road surface adhesion coefficient; and fusing the first and second estimation results of the road surface adhesion coefficient to obtain a target estimation result of the road surface adhesion coefficient. This invention fuses the acquired images and LiDAR point clouds to estimate the road surface adhesion coefficient, covering more complex scenarios and effectively improving the reliability of the estimation method under different environmental conditions while ensuring real-time performance and accuracy.

Description

Technical Field

[0001] This invention relates to the field of vehicle technology, and in particular to a method and apparatus for estimating road surface adhesion coefficient based on image and point cloud fusion. Background Technology

[0002] With the surge in the number of cars, traffic safety issues arising from the varying skill levels and complex psychology of drivers are becoming increasingly prominent. Active vehicle safety technology has become a crucial element in addressing this complex problem, especially in adverse road conditions such as snow, ice, or muddy surfaces. Vehicles often face the risk of locking up, skidding, loss of control, rollover, and complete instability, causing tires to operate near or even beyond their design limits, leading to traffic accidents. Therefore, real-time and accurate sensing of the road surface adhesion coefficient is a vital guarantee for active vehicle safety technology and an important support for coordinating the control of various functional modules of the vehicle chassis.

[0003] In existing technologies, road surface adhesion coefficient estimation mainly suffers from two problems: First, the dynamic-based estimation method requires the tire to be subjected to sufficient excitation to achieve good accuracy, generally exceeding 70% of the tire adhesion limit, and requires the accumulation of a certain amount of measurement data for effective estimation, which is not easy to meet real-time requirements; Second, the estimation based on images and point clouds is limited by environmental factors. Estimating the road surface adhesion coefficient by cameras under backlight and night conditions is extremely challenging, while lidar requires the establishment of a prior database and cannot cope with the influence of random environmental factors such as debris on the estimation. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the related art.

[0005] Therefore, one objective of this invention is to propose a road surface adhesion coefficient estimation method based on image and point cloud fusion. This method estimates the road surface adhesion coefficient by fusing the acquired image with the lidar point cloud. While fully leveraging the high detection accuracy and real-time performance of image detection, it utilizes the light-insensitive nature of point clouds to redundantly supplement image detection, thereby covering more complex scenes. Under the premise of ensuring real-time performance and accuracy, it effectively improves the reliability of the estimation method under different environmental conditions.

[0006] Another objective of this invention is to propose a road surface adhesion coefficient estimation device based on image and point cloud fusion.

[0007] To achieve the above objectives, one embodiment of the present invention proposes a method for estimating road surface adhesion coefficient based on image and point cloud fusion, comprising:

[0008] Acquire vehicle sensor signals and images of the area in front of the vehicle, as well as road point clouds;

[0009] Based on the sensor signals, the region of interest for estimating the road surface adhesion coefficient is determined using vehicle kinematics;

[0010] Based on the region of interest and the image, a first estimate of the road surface adhesion coefficient is performed to obtain a first estimate result of the road surface adhesion coefficient.

[0011] A second estimation of the road surface adhesion coefficient is performed based on the region of interest and the road surface point cloud, resulting in a second estimation result of the road surface adhesion coefficient.

[0012] The first and second estimates of the road surface adhesion coefficient are fused to obtain the target estimate of the road surface adhesion coefficient.

[0013] The road surface adhesion coefficient estimation method based on image and point cloud fusion in this invention may also have the following additional technical features:

[0014] Further, the sensor signals include the current vehicle speed and front wheel steering angle; the step of determining the region of interest for estimating the road adhesion coefficient based on the sensor signals using vehicle kinematics includes:

[0015] Determine the current location of the vehicle;

[0016] Based on the vehicle's current position, current speed, and front wheel angle, the predicted trajectory of the vehicle is obtained through the vehicle's kinematic equations.

[0017] Based on the predicted trajectory and preset width, the region of interest for estimating the road surface adhesion coefficient is determined.

[0018] Further, the first estimation of the road surface adhesion coefficient based on the region of interest and the image, to obtain a first estimation result of the road surface adhesion coefficient, includes:

[0019] The image is then converted to obtain the corresponding bird's-eye view;

[0020] The road surface in the area of ​​interest in the bird's-eye view is detected by a target deep learning network model, and the first classification result of the road surface type is obtained.

[0021] Based on the first classification result of the road surface type, a first estimate of the road surface adhesion coefficient is determined.

[0022] Furthermore, before detecting road surfaces within the region of interest in the bird's-eye view using a target deep learning network model to obtain a road surface type classification result, the method further includes:

[0023] Obtain the first training data, and filter the first training data to obtain the second training data;

[0024] The second training data is labeled to obtain the third training data;

[0025] Based on the third training data, the corresponding training dataset and validation dataset are obtained;

[0026] The initial deep learning network model is trained based on the training dataset to obtain the trained deep learning network model.

[0027] The trained deep learning network model is validated using the validation dataset to obtain the target deep learning network model.

[0028] Further, the second estimation of the road surface adhesion coefficient based on the region of interest and the road surface point cloud, to obtain a second estimation result of the road surface adhesion coefficient, includes:

[0029] Construct a road surface point cloud feature library;

[0030] Calculate the similarity between the road surface point cloud within the region of interest and each road surface point cloud feature in the road surface point cloud feature library;

[0031] Based on the similarity, a second classification result for the road surface type is determined;

[0032] Based on the second classification result of the road surface type, a second estimate of the road surface adhesion coefficient is determined.

[0033] Further, calculating the similarity between the road surface point cloud within the region of interest and each road surface point cloud feature in the road surface point cloud feature library includes:

[0034] Determine the first reflection intensity of the road surface point cloud within the region of interest;

[0035] Based on the first reflection intensity and the second reflection intensity of each road surface point cloud feature in the road surface point cloud feature library, the similarity between the road surface point cloud in the region of interest and each road surface point cloud feature in the road surface point cloud feature library is calculated using a similarity formula.

[0036] Furthermore, the process of fusing the first and second estimates of the road surface adhesion coefficient to obtain a target estimate of the road surface adhesion coefficient includes: fusing the first and second estimates of the road surface adhesion coefficient using a weighted least squares method to obtain a target estimate of the road surface adhesion coefficient.

[0037] To achieve the above objectives, another embodiment of the present invention proposes a road surface adhesion coefficient estimation device based on image and point cloud fusion, the device comprising:

[0038] The acquisition module is used to acquire sensor signals from the vehicle and images and road point clouds in front of the vehicle;

[0039] The determination module is used to determine the region of interest for estimating the road surface adhesion coefficient based on the sensor signals and vehicle kinematics;

[0040] The first estimation module is used to make a first estimation of the road surface adhesion coefficient based on the region of interest and the image, and obtain a first estimation result of the road surface adhesion coefficient.

[0041] The second estimation module is used to perform a second estimation of the road surface adhesion coefficient based on the region of interest and the road surface point cloud, and obtain a second estimation result of the road surface adhesion coefficient.

[0042] The fusion module is used to fuse the first and second estimation results of the road surface adhesion coefficient to obtain the target estimation result of the road surface adhesion coefficient.

[0043] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0044] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0045] Figure 1 This is a flowchart of a road surface adhesion coefficient estimation method based on image and point cloud fusion according to an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of a vehicle kinematic model according to an embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram of a road surface adhesion coefficient estimation device based on image and point cloud fusion according to an embodiment of the present invention. Detailed Implementation

[0048] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0049] The following description, with reference to the accompanying drawings, describes a method and apparatus for estimating road surface adhesion coefficient based on image and point cloud fusion according to an embodiment of the present invention.

[0050] First, the method for estimating road surface adhesion coefficient based on image and point cloud fusion according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0051] Figure 1 This is a flowchart of a road surface adhesion coefficient estimation method based on image and point cloud fusion according to an embodiment of the present invention.

[0052] like Figure 1 As shown, the road surface adhesion coefficient estimation method based on image and point cloud fusion includes the following steps:

[0053] Step S1: Acquire the vehicle's sensor signals and the image and road point cloud in front of the vehicle;

[0054] In one embodiment of the present invention, the aforementioned sensor signals may include the current vehicle speed and the front wheel steering angle. In another embodiment of the present invention, vehicle sensor signals can be acquired via sensors.

[0055] Furthermore, in one embodiment of the present invention, an in-vehicle camera can be used to capture images of the area in front of the vehicle. The image resolution is 1920×1080, and the acquisition frequency is 30 frames per second. The in-vehicle camera is positioned below the rearview mirror inside the windshield. The camera angle is adjusted so that the lowest point of the field of view is at the edge of the hood, and the road surface occupies more than three-fifths of the entire image, ensuring that the road surface area in the captured image exceeds 50 meters.

[0056] Furthermore, in one embodiment of the present invention, a vehicle-mounted LiDAR can be used to collect point clouds in front of the vehicle, with a horizontal viewing angle of 120 degrees, a vertical viewing angle of 25 degrees, and a collection frequency of 10 Hz. The LiDAR is positioned in front of the center of the vehicle's front bumper and is not tilted in any direction.

[0057] Furthermore, in one embodiment of the present invention, an inertial navigation system can be used to acquire clock source signals via the GPRMC protocol at a frequency of 100Hz, and the entire system uses the clock source signals as a time reference for timestamp recording.

[0058] In one embodiment of the present invention, the image of the vehicle can be transmitted to the vehicle-mounted industrial control computer via USB, the point cloud of the vehicle can be transmitted to the vehicle-mounted industrial control computer via Ethernet, the vehicle speed and front wheel angle signals can be transmitted to the industrial control computer via CAN interface, the inertial navigation system transmits signals to the industrial control computer via serial port, and the industrial control computer records the UTC time collected by the inertial navigation system to the image, point cloud and vehicle speed and front wheel angle timestamps, providing a reference for subsequent time and space synchronization.

[0059] Step S2: Based on sensor signals, determine the region of interest for estimating the road surface adhesion coefficient using vehicle kinematics;

[0060] In one embodiment of the present invention, after obtaining the sensor signal through the above steps, the region of interest for estimating the road surface adhesion coefficient can be determined based on the sensor signal and vehicle kinematics.

[0061] Specifically, in one embodiment of the present invention, the method for determining the region of interest for estimating the road surface adhesion coefficient based on sensor signals and vehicle kinematics may include the following steps:

[0062] Step S21: Determine the current location of the vehicle;

[0063] Step S22: Based on the vehicle's current position, current speed, and front wheel angle, obtain the predicted trajectory of the vehicle using the vehicle kinematic equations;

[0064] Step S23: Based on the predicted trajectory and preset width, determine the region of interest for estimating the road surface adhesion coefficient.

[0065] In one embodiment of the present invention, Figure 2 This is a schematic diagram of a vehicle kinematic model proposed in an embodiment of the present invention. Figure 2 As shown, the vehicle's kinematic equations are:

[0066]

[0067] Where (x,y) is the position of the vehicle's center of mass in the world coordinate system, v is the vehicle speed, and δ f Let be the front wheel steering angle, and a and b be the distances from the center of mass to the front and rear axles, respectively.

[0068] And, the position of the left front wheel (x) fL ,y fL ) and right front wheel position (x fR ,y fR This can be represented as:

[0069]

[0070] Where B is the front track width and ψ represents the vehicle yaw angle.

[0071] In one embodiment of the present invention, the current position is used as a reference, and the current vehicle speed and front wheel angle are fixed inputs. The trajectories of the left and right front wheels are predicted through the above steps to obtain the predicted trajectory of the vehicle. The prediction distance is set to 10m, and the vehicle speed is set to 5km / h when the vehicle speed is less than 5km / h.

[0072] In one embodiment of the present invention, the predicted trajectories of the left and right front wheels are used as the center line, and the preset width is twice the width of the front tires, which is determined as the region of interest for estimating the road surface adhesion coefficient.

[0073] Step S3: Based on the region of interest and the image, perform a first estimate of the road surface adhesion coefficient to obtain the first estimate result of the road surface adhesion coefficient;

[0074] In one embodiment of the present invention, after determining the region of interest as described above, a first estimate of the road surface adhesion coefficient can be made based on the region of interest and the image to obtain a first estimate result of the road surface adhesion coefficient.

[0075] Specifically, in one embodiment of the present invention, the method for obtaining a first estimate of the road surface adhesion coefficient based on the region of interest and the image may include the following steps:

[0076] Step S31: Convert the image to obtain the corresponding bird's-eye view;

[0077] Step S32: Detect the road surface in the area of ​​interest in the bird's-eye view using the target deep learning network model to obtain the first classification result of the road surface type;

[0078] Step S33: Based on the first classification result of the road surface type, determine the first estimated result of the road surface adhesion coefficient.

[0079] In one embodiment of the present invention, the method for converting an image to obtain a corresponding bird's-eye view may include: converting the image to a bird's-eye view through inverse perspective transformation based on vehicle motion correction. Converting the image from a front view to a bird's-eye view can reduce the influence of background elements such as the sky on road surface classification. However, when a vehicle is in motion, its direction of travel is often not parallel to the road direction. To make the main road direction in the inverse perspective transformation result more parallel to the image coordinate system, compensation is needed for the vehicle's pitch and yaw angles. Therefore, inverse perspective based on vehicle motion correction can be:

[0080]

[0081] Where θ represents the vehicle pitch angle, x0(r,c) and y0(r,c) represent the uncorrected horizontal and vertical coordinates mapped from the image coordinate system to the world coordinate system, and x c (r,c) and y c (r,c) are the corrected coordinates.

[0082] In one embodiment of the present invention, a corresponding bird's-eye view can be obtained based on the corrected coordinates obtained through the above steps.

[0083] Furthermore, in one embodiment of the present invention, before detecting the road surface in the region of interest in the bird's-eye view using a target deep learning network model to obtain a first classification result of the road surface type, the above method may further include the following steps:

[0084] Step 1: Obtain the first training data and filter it to obtain the second training data;

[0085] Step 2: Label the second training data to obtain the third training data;

[0086] Step 3: Based on the third training data, obtain the corresponding training dataset and validation dataset;

[0087] Step 4: Train the initial deep learning network model based on the training dataset to obtain the trained deep learning network model;

[0088] Step 5: Validate the trained deep learning network model using the validation dataset to obtain the target deep learning network model.

[0089] In one embodiment of the present invention, road surface images under various weather conditions are collected as first training data, and the images in the first training data are filtered by clarity to remove blurry images to obtain second training data.

[0090] Furthermore, in one embodiment of the present invention, the image in the second training data can be labeled using labelImg image annotation software to obtain the third training data by annotating the location and type information of road surface features in the image. The road surface types are divided into eight categories, namely dry asphalt road surface, wet asphalt road surface, dry cement road surface, wet cement road surface, paved road surface, loose snow road surface, compacted snow road surface, and ice film road surface.

[0091] In one embodiment of the present invention, the third training data can be divided into a training dataset and a validation dataset for training an initial deep learning network model according to a preset ratio. For example, assuming the preset ratio is 7:3, that is, 70% of the third training data is randomly divided into a training dataset and 30% into a validation dataset.

[0092] Furthermore, in one embodiment of the present invention, the method for training an initial deep learning network model based on a training dataset to obtain a trained deep learning network model may include the following steps:

[0093] Step a: Input the data from the training dataset into the initial deep learning network model in batches to obtain the classification results of the predicted road surface type;

[0094] Step b: Input the predicted road surface type classification result and the actual road surface type classification result into the loss function to obtain the corresponding loss value;

[0095] Step c: Update the network parameters in the initial deep learning network model using the loss value until the number of iterations of the network parameters reaches the preset number, and then obtain the trained deep learning network model.

[0096] In one embodiment of the present invention, the deep learning network model described above may be YOLOv10-S.

[0097] Furthermore, in one embodiment of the present invention, the images in the training set can be divided into 2... n Zhang is a batch, where 0≤n≤6 and n is an integer. The training data is used to train the model in the above steps in batches. The preset number of times can be set as needed, for example, the preset number of times is 50200.

[0098] Furthermore, in one embodiment of the present invention, the loss function described above is the same as that in the prior art, and will not be described in detail here.

[0099] Furthermore, in one embodiment of the present invention, after obtaining the trained deep learning network model through the above steps, the trained deep learning network model can be verified using a verification dataset. If the road feature recognition accuracy of each image in the verification dataset is greater than 90%, the trained deep learning network model is determined as the target deep learning network model; otherwise, the model needs to be trained again based on the previous training, and so on until the accuracy requirement is met.

[0100] Furthermore, in one embodiment of the present invention, after obtaining the target deep learning network model through the above steps, the road surface in the area of ​​interest in the bird's-eye view can be detected by the target deep learning network model to obtain the first classification result of the road surface type.

[0101] Furthermore, in one embodiment of the present invention, after obtaining the first classification result of the road surface type through the above steps, the first estimated result of the road surface adhesion coefficient is determined by the mapping relationship between the first classification result and the road surface adhesion coefficient. Considering that the road surface adhesion coefficient is related to factors such as road surface wear, tire wear, and air temperature and humidity, and that the range of road surface adhesion coefficient values ​​differs at different vehicle speeds, the range of road surface adhesion coefficients for each road surface type at different speeds is determined by referring to the publicly available standard "GA / T643-2006 Technical Appraisal of Vehicle Speed ​​in Typical Traffic Accident Patterns". The peak road surface adhesion coefficient is selected as the final detection result. Table 1 is a mapping relationship comparison table between road surface type and peak road surface adhesion coefficient proposed in this embodiment of the present invention.

[0102] Table 1

[0103]

[0104]

[0105] Based on the first classification result of the road surface type, the first estimated result of the road surface adhesion coefficient is obtained by querying the corresponding relationship in Table 1. The first estimated result is represented by a vector [index]. c ,conf c ,e c Save as [,v,UTC], where index c For road surface type, conf c e represents the confidence level. c This represents the peak road adhesion coefficient, v represents the vehicle speed, and UTC represents the timestamp.

[0106] Step S4: Based on the region of interest and the road surface point cloud, perform a second estimation of the road surface adhesion coefficient to obtain the second estimation result of the road surface adhesion coefficient;

[0107] In one embodiment of the present invention, after determining the region of interest through the above steps, a second estimation of the road surface adhesion coefficient can be performed based on the region of interest and the road surface point cloud to obtain a second estimation result of the road surface adhesion coefficient.

[0108] Specifically, in one embodiment of the present invention, the method for obtaining a second estimate of the road surface adhesion coefficient based on the region of interest and the road surface point cloud may include the following steps:

[0109] Step S41: Construct a road surface point cloud feature library;

[0110] Step S42: Calculate the similarity between the road surface point cloud within the region of interest and the features of each road surface point cloud in the road surface point cloud feature library;

[0111] Step S43: Determine the second classification result of the road surface type based on similarity;

[0112] Step S44: Based on the second classification result of the road surface type, determine the second estimate of the road surface adhesion coefficient.

[0113] In one embodiment of the present invention, lidar point cloud data of different road surfaces under different lighting and weather conditions during the day and night can be collected, and a road surface point cloud feature library can be constructed based on the reflection intensity of each point cloud data.

[0114] In one embodiment of the present invention, when a lidar scans a road surface, the reflection intensity of the point cloud follows a Gaussian distribution. However, due to the presence of traffic markings and other material objects on the road surface, the probability distribution of the reflection intensity of the point cloud contains noise. Therefore, the probability density function f(I) of the reflection intensity I can be expressed as:

[0115]

[0116] Where, α i Let μ be the mixing coefficient of the i-th Gaussian distribution, with values ​​ranging from [0,1], and N be the probability density function of the Gaussian distribution. i and σ i denoted as the mean and standard deviation of the i-th Gaussian distribution, respectively.

[0117] Furthermore, in one embodiment of the present invention, assuming the amount of point cloud data collected is n, the set of corresponding reflection intensities is I = (I1, I2, ..., I...). j ,...,I n ) T The Gaussian model response Φ is defined as:

[0118]

[0119] in,

[0120] In one embodiment of the present invention, (α) can be obtained through maximum likelihood estimation. i ,μ i ,σ i The iterative solution formula for ) is:

[0121]

[0122] Specifically, when the iteration termination condition of the above iterative solution formula is triggered or the maximum number of iterations is reached, the corresponding (α) is obtained. i ,μ i ,σ i ), and based on (α) i ,μ i ,σ i The reflection intensity of the point cloud is determined. Since the reflection intensity differs greatly between day and night, a point cloud sub-feature library for typical road surfaces in two time periods is established based on the reflection intensity of the point cloud.

[0123] Furthermore, in one embodiment of the present invention, after constructing the road surface point cloud feature library through the above steps, it is necessary to calculate the similarity between the road surface point cloud in the region of interest and the features of each road surface point cloud in the road surface point cloud feature library in order to determine the second classification result of the road surface type.

[0124] Specifically, in one embodiment of the present invention, the method for calculating the similarity between the road surface point cloud within the region of interest and the features of each road surface point cloud in the road surface point cloud feature library may include the following steps:

[0125] Step 1: Determine the first reflection intensity of the road surface point cloud within the region of interest;

[0126] Step 2: Based on the first reflection intensity and the second reflection intensity of each road surface point cloud feature in the road surface point cloud feature library, calculate the similarity between the road surface point cloud in the region of interest and each road surface point cloud feature in the road surface point cloud feature library using the similarity formula.

[0127] In one embodiment of the present invention, the first reflection intensity of the road surface point cloud in the region of interest can be determined by the probability density function of the reflection intensity described above. The embodiment of the present invention will not be elaborated here.

[0128] Furthermore, in one embodiment of the present invention, the similarity between the road surface point cloud within the region of interest and each road surface point cloud feature in the road surface point cloud feature library can be calculated using a similarity formula determined based on Jensen-Shannon divergence, wherein the similarity formula is:

[0129] S i =1-JSD i (f scan (I),f i (I))

[0130] in,

[0131] And, f scan (I) represents the first reflection intensity of the road surface point cloud within the region of interest, f i (I) represents the second reflection intensity of the i-th road surface point cloud in the road surface point cloud feature library, S i The similarity between the road surface point cloud within the region of interest and the i-th road surface point cloud in the road surface point cloud feature library is given.

[0132] Furthermore, in one embodiment of the present invention, after obtaining the similarity between the road surface point cloud within the region of interest and the features of each road surface point cloud in the road surface point cloud feature library through the above steps, the similarity can be sorted in descending order, and the maximum similarity S is selected. max The corresponding road surface type is used as the second classification result for the road surface type. Specifically, after completing the road surface detection in the daytime and nighttime sub-feature databases through the above steps, the results obtained are merged to obtain the second classification result for the road surface type. Furthermore, in one embodiment of the present invention, after obtaining the second classification result for the road surface type through the above steps, the second estimated result of the road surface adhesion coefficient can be obtained by querying Table 1. The second estimated result can be a vector [index...] l ,conf l ,e l [,v,UTC], where index l For road surface type, conf l Indicates confidence level, conf l =S max e lThis represents the peak road adhesion coefficient, v represents the vehicle speed, and UTC represents the timestamp.

[0133] Step S5: The first and second estimation results of the road surface adhesion coefficient are fused to obtain the target estimation result of the road surface adhesion coefficient.

[0134] In one embodiment of the present invention, after obtaining the first and second estimation results of the road surface adhesion coefficient through the above steps, the first and second estimation results of the road surface adhesion coefficient can be fused to obtain the target estimation result of the road surface adhesion coefficient.

[0135] Specifically, in one embodiment of the present invention, the method of fusing the first and second estimation results of the road surface adhesion coefficient to obtain the target estimation result of the road surface adhesion coefficient may include: fusing the first and second estimation results of the road surface adhesion coefficient using a weighted least squares method to obtain the target estimation result of the road surface adhesion coefficient.

[0136] In one embodiment of the present invention, the target estimation result of the road surface adhesion coefficient obtained through the above steps is stored as a vector [e,conf,v,UTC], where e is the fused road surface adhesion coefficient, conf is the confidence level of the fused road surface adhesion coefficient, v is the vehicle speed, and UTC is the timestamp. The fused road surface adhesion coefficient e is:

[0137] e = (conf c +conf l ) -1 (conf c ·e c +conf l ·e l );

[0138] The confidence level (conf) after fusion is:

[0139]

[0140] The road surface adhesion coefficient estimation method based on image and point cloud fusion proposed in this embodiment of the invention estimates the road surface adhesion coefficient by fusing the acquired image with the lidar point cloud. While giving full play to the high detection accuracy and real-time performance of image detection, it utilizes the light-insensitive characteristic of point cloud to redundantly supplement image detection, thereby covering more complex scenes. Under the premise of ensuring real-time performance and accuracy, it effectively improves the reliability of the estimation method under different environmental conditions.

[0141] Next, referring to the accompanying drawings, a road surface adhesion coefficient estimation device based on image and point cloud fusion according to an embodiment of the present invention is described.

[0142] Figure 3 This is a schematic diagram of a road surface adhesion coefficient estimation device based on image and point cloud fusion according to an embodiment of the present invention.

[0143] like Figure 3 As shown, the road surface adhesion coefficient estimation device 10 based on image and point cloud fusion includes: an acquisition module 301, a determination module 302, a first estimation module 303, a second estimation module 304, and a fusion module 305, wherein...

[0144] The acquisition module 301 is used to acquire the vehicle's sensor signals and images and road point clouds in front of the vehicle;

[0145] The determination module 302 is used to determine the region of interest for estimating the road surface adhesion coefficient based on sensor signals and vehicle kinematics;

[0146] The first estimation module 303 is used to perform a first estimation of the road surface adhesion coefficient based on the region of interest and the image, and obtain the first estimation result of the road surface adhesion coefficient.

[0147] The second estimation module 304 is used to perform a second estimation of the road surface adhesion coefficient based on the region of interest and the road surface point cloud, and obtain the second estimation result of the road surface adhesion coefficient.

[0148] The fusion module 305 is used to fuse the first and second estimation results of the road surface adhesion coefficient to obtain the target estimation result of the road surface adhesion coefficient.

[0149] Furthermore, the sensor signals include the current vehicle speed and the front wheel steering angle; the aforementioned determining module 302 is specifically used for:

[0150] Determine the vehicle's current location;

[0151] Based on the vehicle's current position, current speed, and front wheel angle, the predicted trajectory of the vehicle is obtained through the vehicle's kinematic equations.

[0152] Based on the predicted trajectory and preset width, the region of interest for estimating the road surface adhesion coefficient is determined.

[0153] Furthermore, the aforementioned first estimation module 303 is specifically used for:

[0154] The image is converted to obtain the corresponding bird's-eye view;

[0155] The target deep learning network model is used to detect the road surface in the area of ​​interest in the bird's-eye view, and the first classification result of the road surface type is obtained.

[0156] Based on the first classification result of road surface type, the first estimate of the road surface adhesion coefficient is determined.

[0157] Furthermore, the above-mentioned device is also used for:

[0158] Obtain the first training data and filter it to obtain the second training data;

[0159] Label the second training data to obtain the third training data;

[0160] Based on the third training data, the corresponding training dataset and validation dataset are obtained;

[0161] The initial deep learning network model is trained based on the training dataset to obtain the trained deep learning network model.

[0162] The trained deep learning network model is validated using a validation dataset to obtain the target deep learning network model.

[0163] Furthermore, the aforementioned second estimation module 304 is specifically used for:

[0164] Construct a road surface point cloud feature library;

[0165] Calculate the similarity between the road surface point cloud within the region of interest and the features of each road surface point cloud in the road surface point cloud feature library;

[0166] Based on similarity, determine the second classification result for the road surface type;

[0167] Based on the second classification result of road surface type, a second estimate of the road surface adhesion coefficient is determined.

[0168] Furthermore, the second estimation module 304 described above is also used for:

[0169] Determine the first reflection intensity of the road surface point cloud within the region of interest;

[0170] Based on the second and first reflection intensities of each road point cloud feature in the road point cloud feature library, the similarity between the road point cloud in the region of interest and each road point cloud feature in the road point cloud feature library is calculated using a similarity formula.

[0171] Furthermore, the aforementioned fusion module 305 is specifically used for:

[0172] The first and second estimates of the road surface adhesion coefficient are fused using the weighted least squares method to obtain the target estimate of the road surface adhesion coefficient.

[0173] It should be noted that the foregoing explanation of the method embodiments also applies to the apparatus of this embodiment, and will not be repeated here.

[0174] The road surface adhesion coefficient estimation device based on image and point cloud fusion proposed in this embodiment of the invention estimates the road surface adhesion coefficient by fusing the acquired image with the lidar point cloud. While giving full play to the high detection accuracy and real-time performance of image detection, it utilizes the light-insensitive characteristic of point cloud to redundantly supplement image detection, thereby covering more complex scenes. Under the premise of ensuring real-time performance and accuracy, it effectively improves the reliability of the estimation method under different environmental conditions.

[0175] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0176] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0177] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for estimating road surface adhesion coefficient based on image and point cloud fusion, characterized in that, The method includes: Acquire vehicle sensor signals and images of the area in front of the vehicle, as well as road point clouds; Based on the sensor signals, the region of interest for estimating the road surface adhesion coefficient is determined using vehicle kinematics; Based on the region of interest and the image, a first estimate of the road surface adhesion coefficient is performed to obtain a first estimate result of the road surface adhesion coefficient. A second estimation of the road surface adhesion coefficient is performed based on the region of interest and the road surface point cloud, resulting in a second estimation result of the road surface adhesion coefficient. The first and second estimation results of the road surface adhesion coefficient are fused to obtain the target estimation result of the road surface adhesion coefficient. The first estimation of the road surface adhesion coefficient based on the region of interest and the image, to obtain the first estimation result of the road surface adhesion coefficient, includes: The image is then converted to obtain the corresponding bird's-eye view; The road surface in the area of ​​interest in the bird's-eye view is detected by a target deep learning network model, and the first classification result of the road surface type is obtained. Based on the first classification result of the road surface type, a first estimate of the road surface adhesion coefficient is determined; The second estimation of the road surface adhesion coefficient based on the region of interest and the road surface point cloud, to obtain the second estimation result of the road surface adhesion coefficient, includes: Construct a road surface point cloud feature library; Calculate the similarity between the road surface point cloud within the region of interest and each road surface point cloud feature in the road surface point cloud feature library; Based on the similarity, a second classification result for the road surface type is determined; Based on the second classification result of the road surface type, a second estimate of the road surface adhesion coefficient is determined.

2. The method according to claim 1, characterized in that, The sensor signals include the current vehicle speed and front wheel steering angle; the process of determining the region of interest for estimating the road adhesion coefficient based on the sensor signals using vehicle kinematics includes: Determine the current location of the vehicle; Based on the vehicle's current position, current speed, and front wheel angle, the predicted trajectory of the vehicle is obtained through the vehicle's kinematic equations. Based on the predicted trajectory and preset width, the region of interest for estimating the road surface adhesion coefficient is determined.

3. The method according to claim 1, characterized in that, Before detecting road surfaces within the region of interest in the bird's-eye view using a target deep learning network model to obtain a road surface type classification result, the method further includes: Obtain the first training data, and filter the first training data to obtain the second training data; The second training data is labeled to obtain the third training data; Based on the third training data, the corresponding training dataset and validation dataset are obtained; The initial deep learning network model is trained based on the training dataset to obtain the trained deep learning network model. The trained deep learning network model is validated using the validation dataset to obtain the target deep learning network model.

4. The method according to claim 1, characterized in that, The calculation of the similarity between the road point cloud within the region of interest and each road point cloud feature in the road point cloud feature library includes: Determine the first reflection intensity of the road surface point cloud within the region of interest; Based on the second reflection intensity and the first reflection intensity of each road point cloud feature in the road point cloud feature library, the similarity between the road point cloud in the region of interest and each road point cloud feature in the road point cloud feature library is calculated using a similarity formula.

5. The method according to claim 1, characterized in that, The step of fusing the first and second estimates of the road surface adhesion coefficient to obtain the target estimate of the road surface adhesion coefficient includes: fusing the first and second estimates of the road surface adhesion coefficient using a weighted least squares method to obtain the target estimate of the road surface adhesion coefficient.

6. A road surface adhesion coefficient estimation device based on image and point cloud fusion, characterized in that, The device includes: The acquisition module is used to acquire sensor signals from the vehicle and images and road point clouds in front of the vehicle; The determination module is used to determine the region of interest for estimating the road surface adhesion coefficient based on the sensor signals and vehicle kinematics; The first estimation module is used to make a first estimation of the road surface adhesion coefficient based on the region of interest and the image, and obtain a first estimation result of the road surface adhesion coefficient. The second estimation module is used to perform a second estimation of the road surface adhesion coefficient based on the region of interest and the road surface point cloud, and obtain a second estimation result of the road surface adhesion coefficient. The fusion module is used to fuse the first and second estimation results of the road surface adhesion coefficient to obtain the target estimation result of the road surface adhesion coefficient. The first estimation module is specifically used for: The image is then converted to obtain the corresponding bird's-eye view; The road surface in the area of ​​interest in the bird's-eye view is detected by a target deep learning network model, and the first classification result of the road surface type is obtained. Based on the first classification result of the road surface type, a first estimate of the road surface adhesion coefficient is determined; The second estimation module is specifically used for: Construct a road surface point cloud feature library; Calculate the similarity between the road surface point cloud within the region of interest and each road surface point cloud feature in the road surface point cloud feature library; Based on the similarity, a second classification result for the road surface type is determined; Based on the second classification result of the road surface type, a second estimate of the road surface adhesion coefficient is determined.

7. An electronic device, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-5.

Images