Road adhesion coefficient estimation method and device based on image and point cloud fusion

Abstract

The invention discloses a road adhesion coefficient estimation method and device based on image and point cloud fusion. The method comprises the following steps: acquiring a sensor signal of a vehicle and an image and road point cloud in front of the vehicle; determining a region of interest of road adhesion coefficient estimation through vehicle kinematics based on the sensor signal; performing first estimation of a road adhesion coefficient based on the region of interest and the image to obtain a first estimation result of the road adhesion coefficient; performing second estimation of the road adhesion coefficient based on the region of interest and the road point cloud to obtain a second estimation result of the road adhesion coefficient; and fusing the first estimation result and the second estimation result of the road adhesion coefficient to obtain a target estimation result of the road adhesion coefficient. According to the method, fusion estimation is carried out on the road adhesion coefficient through the obtained image and the laser radar point cloud, more complex scenes are covered, and the reliability of the estimation method under different environmental conditions is effectively improved on the premise that the real-time performance and accuracy are guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of vehicle technology, and in particular to a method and device for estimating a road adhesion coefficient based on image and point cloud fusion. Background Art

[0002] With the rapid increase in the number of cars, traffic safety issues caused by the uneven skills of drivers and the complexity and diversity of driving psychology have become increasingly prominent. Vehicle active safety technology has become a key link in solving this complex problem. Especially in the face of harsh road conditions, such as snow, ice or muddy and slippery road conditions, vehicles often face the risk of locking and sliding, uncontrolled rollover and overall instability, which causes the tire working condition to approach or even exceed its design limit, thus causing traffic accidents. Based on this, real-time and accurate perception of the road adhesion coefficient is an important guarantee for active safety technology of automobiles, and it is also an important support for coordinating the various functional modules of the automobile chassis to achieve coordinated control.

[0003] In the existing technology, there are two main problems in estimating the road adhesion coefficient: first, the dynamics-based estimation method requires that the tires be sufficiently stimulated to achieve good accuracy, which generally needs to exceed 70% of the tire adhesion limit, and a certain amount of measurement data needs to be accumulated for effective estimation, which is not easy to meet real-time requirements; second, the image and point cloud-based estimation is subject to the limitations of environmental factors. The road adhesion coefficient estimation of the camera under backlight and night conditions is extremely challenging, and the lidar needs to establish a priori database and cannot cope with the impact of random environmental factors such as accumulation on the estimation. Summary of the invention

[0004] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0005] To this end, an object of the present invention is to propose a road adhesion coefficient estimation method based on image and point cloud fusion, which estimates the road adhesion coefficient by fusing the acquired image with the lidar point cloud. While giving full play to the high detection accuracy and real-time performance of image detection, the point cloud is used to perform redundant supplementation for image detection by utilizing the insensitivity to illumination, thereby covering more complex scenes. While ensuring real-time performance and accuracy, the reliability of the estimation method under different environmental conditions is effectively improved.

[0006] Another object of the present invention is to provide a road adhesion coefficient estimation device based on image and point cloud fusion.

[0007] To achieve the above object, an embodiment of the present invention proposes a method for estimating road adhesion coefficient based on image and point cloud fusion, comprising:

[0008] Acquire the sensor signals of the vehicle and the image and road point cloud in front of the vehicle;

[0009] determining a region of interest for estimating road adhesion coefficients through vehicle kinematics based on the sensor signals;

[0010] Performing a first estimation of a road surface adhesion coefficient based on the region of interest and the image to obtain a first estimation result of the road surface adhesion coefficient;

[0011] Performing a second estimation of the road surface adhesion coefficient based on the region of interest and the road surface point cloud to obtain a second estimation result of the road surface adhesion coefficient;

[0012] The first estimation result and the second estimation result of the road adhesion coefficient are fused to obtain a target estimation result of the road adhesion coefficient.

[0013] The road adhesion coefficient estimation method based on image and point cloud fusion in the embodiment of the present invention may also have the following additional technical features:

[0014] Further, the sensor signal includes the current vehicle speed and the front wheel steering angle; the region of interest for estimating the road adhesion coefficient based on the sensor signal and through vehicle kinematics includes:

[0015] determining a current position of the vehicle;

[0016] Based on the current position of the vehicle, the current vehicle speed and the front wheel steering angle, a predicted trajectory of the vehicle is obtained through a vehicle kinematic equation;

[0017] Based on the predicted trajectory and the preset width, a region of interest for estimating the road adhesion coefficient is determined.

[0018] Further, performing a first estimation of a road adhesion coefficient based on the region of interest and the image to obtain a first estimation result of the road adhesion coefficient includes:

[0019] Converting the image to obtain a corresponding bird's-eye view;

[0020] Detecting the road surface in the area of ​​interest in the bird's-eye view using a target deep learning network model to obtain a first classification result of the road surface type;

[0021] Based on the first classification result of the road surface type, a first estimation result of the road surface adhesion coefficient is determined.

[0022] Furthermore, before detecting the road surface in the region of interest in the bird's-eye view by using the target deep learning network model to obtain a classification result of the road surface type, the method further includes:

[0023] Acquire first training data, and filter the first training data to obtain second training data;

[0024] Labeling the second training data to obtain third training data;

[0025] Based on the third training data, obtain a corresponding training data set and a verification data set;

[0026] Training the initial deep learning network model based on the training data set to obtain a trained deep learning network model;

[0027] The trained deep learning network model is verified using the verification data set to obtain a target deep learning network model.

[0028] Further, the second estimation of the road adhesion coefficient based on the region of interest and the road point cloud to obtain a second estimation result of the road adhesion coefficient includes:

[0029] Construct a road point cloud feature library;

[0030] Calculating the similarity between the road surface point cloud in the region of interest and each road surface point cloud feature in the road surface point cloud feature library;

[0031] Based on the similarity, determining a second classification result of the road surface type;

[0032] Based on the second classification result of the road surface type, a second estimation result of the road surface adhesion coefficient is determined.

[0033] Furthermore, the calculating of the similarity between the road surface point cloud in the region of interest and each road surface point cloud feature in the road surface point cloud feature library includes:

[0034] Determine a first reflection intensity of the road surface point cloud within the region of interest;

[0035] Based on the first reflection intensity and the second reflection intensity of each road surface point cloud feature in the road surface point cloud feature library, the similarity between the road surface point cloud in the area of ​​interest and each road surface point cloud feature in the road surface point cloud feature library is calculated by a similarity formula.

[0036] Furthermore, the first estimation result and the second estimation result of the road adhesion coefficient are fused to obtain a target estimation result of the road adhesion coefficient, including: fusing the first estimation result and the second estimation result of the road adhesion coefficient by a weighted least squares method to obtain a target estimation result of the road adhesion coefficient.

[0037] To achieve the above object, another embodiment of the present invention proposes a road adhesion coefficient estimation device based on image and point cloud fusion, the device comprising:

[0038] An acquisition module, used to acquire the sensor signals of the vehicle and the image and road point cloud in front of the vehicle;

[0039] a determination module for determining an area of ​​interest for estimating a road adhesion coefficient through vehicle kinematics based on the sensor signal;

[0040] A first estimation module, configured to perform a first estimation of a road adhesion coefficient based on the region of interest and the image, and obtain a first estimation result of the road adhesion coefficient;

[0041] A second estimation module, configured to perform a second estimation of a road adhesion coefficient based on the region of interest and the road point cloud to obtain a second estimation result of the road adhesion coefficient;

[0042] The fusion module is used to fuse the first estimation result and the second estimation result of the road adhesion coefficient to obtain a target estimation result of the road adhesion coefficient.

[0043] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0045] Figure 1 A flowchart of a road adhesion coefficient estimation method based on image and point cloud fusion according to an embodiment of the present invention;

[0046] Figure 2 is a schematic diagram of a vehicle kinematics model according to an embodiment of the present invention;

[0047] Figure 3 Schematic diagram of the structure of a road adhesion coefficient estimation device based on image and point cloud fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0049] The following describes a road adhesion coefficient estimation method and device based on image and point cloud fusion according to an embodiment of the present invention with reference to the accompanying drawings.

[0050] First, a road adhesion coefficient estimation method based on image and point cloud fusion proposed according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0051] Figure 1 The present invention is a flowchart of a method for estimating a road adhesion coefficient based on image and point cloud fusion according to an embodiment of the present invention.

[0052] like Figure 1 As shown, the road adhesion coefficient estimation method based on image and point cloud fusion includes the following steps:

[0053] Step S1, obtaining the sensor signal of the vehicle and the image and road point cloud in front of the vehicle;

[0054] Wherein, in one embodiment of the present invention, the sensor signal may include the current vehicle speed and the front wheel turning angle. In one embodiment of the present invention, the sensor signal of the vehicle may be acquired through a sensor.

[0055] Furthermore, in one embodiment of the present invention, an onboard camera can be used to capture images in front of the vehicle, with an image resolution of 1920×1080 and a capture frequency of 30 frames per second. The onboard camera is arranged below the rearview mirror on the inside of the front windshield, and the camera viewing angle is adjusted so that the bottom of the field of view is located at the edge of the hood, and the road surface in the captured image occupies more than three-fifths of the entire screen, ensuring that the road surface area in the captured image exceeds 50 meters.

[0056] Furthermore, in one embodiment of the present invention, a vehicle-mounted laser radar can be used to collect point clouds in front of the vehicle, with a horizontal viewing angle of 120 degrees, a vertical viewing angle of 25 degrees, and a collection frequency of 10 Hz. The laser radar is arranged in front of the middle position of the front bumper of the vehicle and is not tilted in any direction.

[0057] Furthermore, in one embodiment of the present invention, the inertial navigation system can be used to collect clock source signals through the GPRMC protocol, with an acquisition frequency of 100 Hz. The entire system uses the clock source signal as a time reference to record time stamps.

[0058] In one embodiment of the present invention, the image of the above-mentioned vehicle can be transmitted to the on-board industrial computer via USB, the point cloud of the vehicle can be transmitted to the on-board industrial computer via Ethernet, the vehicle speed and front wheel angle signals can be transmitted to the industrial computer via the CAN interface, the inertial navigation system transmits signals to the industrial computer via the serial port, and the industrial computer uses the UTC time collected by the inertial navigation system to timestamp the image, point cloud, vehicle speed and front wheel angle records, providing a reference basis for subsequent time and space synchronization.

[0059] Step S2, determining an area of ​​interest for estimating the road adhesion coefficient through vehicle kinematics based on the sensor signal;

[0060] In one embodiment of the present invention, after obtaining the sensor signal through the above steps, the region of interest for estimating the road adhesion coefficient can be determined based on the sensor signal and through vehicle kinematics.

[0061] Specifically, in one embodiment of the present invention, the method for determining the region of interest for estimating the road adhesion coefficient based on the sensor signal through vehicle kinematics may include the following steps:

[0062] Step S21, determining the current position of the vehicle;

[0063] Step S22, based on the current position, current speed and front wheel steering angle of the vehicle, obtain the predicted trajectory of the vehicle through the vehicle kinematic equation;

[0064] Step S23, determining an area of ​​interest for estimating the road adhesion coefficient based on the predicted trajectory and the preset width.

[0065] Among them, in one embodiment of the present invention, Figure 2 FIG. 1 is a schematic diagram of a vehicle kinematic model proposed in an embodiment of the present invention. Figure 2 As shown, the vehicle kinematic equation is:

[0066]

[0067] Among them, (x, y) is the position of the vehicle's center of mass in the world coordinate system, v is the vehicle speed, δ f is the front wheel turning angle, a and b are the distances from the center of mass to the front axle and rear axle respectively.

[0068] And, the position of the left front wheel (x fL ,y fL ) and the right front wheel position (x fR ,y fR ) can be expressed as:

[0069]

[0070] Where B is the front track and ψ is the vehicle yaw angle.

[0071] In one embodiment of the present invention, with the current position as a reference, the current vehicle speed and the front wheel steering angle as fixed inputs, the trajectories of the left front wheel and the right front wheel are predicted through the above steps to obtain the predicted trajectory of the vehicle, wherein the predicted distance is set to 10m, and when the vehicle speed is less than 5km / h, the vehicle speed is taken as 5km / h.

[0072] In one embodiment of the present invention, the predicted trajectories of the left and right front wheels are used as the center line, and the preset width is twice the width of the front wheel tire, which is determined as the area of ​​interest for estimating the road adhesion coefficient.

[0073] Step S3, performing a first estimation of the road adhesion coefficient based on the region of interest and the image to obtain a first estimation result of the road adhesion coefficient;

[0074] In one embodiment of the present invention, after the region of interest is determined as described above, a first estimation of the road adhesion coefficient may be performed based on the region of interest and the image to obtain a first estimation result of the road adhesion coefficient.

[0075] Specifically, in one embodiment of the present invention, the method for performing a first estimation of the road surface adhesion coefficient based on the region of interest and the image to obtain a first estimation result of the road surface adhesion coefficient may include the following steps:

[0076] Step S31, converting the image to obtain a corresponding bird's-eye view;

[0077] Step S32, detecting the road surface in the area of ​​interest in the bird's-eye view through the target deep learning network model to obtain a first classification result of the road surface type;

[0078] Step S33: determining a first estimation result of the road adhesion coefficient based on the first classification result of the road type.

[0079] In one embodiment of the present invention, the method of converting an image to obtain a corresponding bird's-eye view may include: converting the image to a bird's-eye view through an inverse perspective change based on vehicle motion correction. Converting the image from a front view to a bird's-eye view can reduce the impact of backgrounds such as the sky on road classification. When a vehicle is driving, the driving direction is often not parallel to the road direction. In order to make the main road direction in the inverse perspective transformation result more parallel to the image coordinate system, the vehicle pitch angle and yaw angle are required to compensate. Based on this, the inverse perspective based on vehicle motion correction can be:

[0080]

[0081] Where θ represents the vehicle pitch angle, x0(r,c) and y0(r,c) represent the uncorrected horizontal and vertical coordinates mapped from the image coordinate system to the world coordinate system, and x c (r,c) and y c (r,c) is the corrected coordinate.

[0082] In one embodiment of the present invention, a corresponding bird's-eye view may be obtained based on the corrected coordinates obtained through the above steps.

[0083] Furthermore, in one embodiment of the present invention, before detecting the road surface in the region of interest in the bird's-eye view by using the target deep learning network model to obtain the first classification result of the road surface type, the method may further include the following steps:

[0084] Step 1: Obtain first training data, and filter the first training data to obtain second training data;

[0085] Step 2: label the second training data to obtain third training data;

[0086] Step 3: Based on the third training data, obtain a corresponding training data set and a verification data set;

[0087] Step 4: Train the initial deep learning network model based on the training data set to obtain a trained deep learning network model;

[0088] Step 5: Use the verification data set to verify the trained deep learning network model to obtain the target deep learning network model.

[0089] In one embodiment of the present invention, the collected road images under various weather conditions are used as the first training data, and the images in the first training data are screened for clarity, and blurred images are removed to obtain the second training data.

[0090] Furthermore, in one embodiment of the present invention, the labelImg image annotation software can be used to label the images in the second training data, and the positions and types of road features in the images are annotated to obtain the third training data, wherein the road types are divided into eight categories, namely dry asphalt road surface, wet asphalt road surface, dry cement road surface, wet cement road surface, brick paved road surface, loose snow road surface, compacted snow road surface, and ice film road surface.

[0091] In one embodiment of the present invention, the third training data can be divided into a training data set and a verification data set for training the initial deep learning network model according to a preset ratio. For example, assuming that the preset ratio is 7:3, that is, 70% of the data in the third training data is randomly divided into a training data set and 30% of the data is divided into a verification data set.

[0092] Further, in one embodiment of the present invention, the method of training the initial deep learning network model based on the training data set to obtain a trained deep learning network model may include the following steps:

[0093] Step a: input the data in the training data set into the initial deep learning network model in batches to obtain the classification results of the predicted road surface type;

[0094] Step b, inputting the classification result of the predicted road surface type and the classification result of the actual road surface type into the loss function to obtain the corresponding loss value;

[0095] Step c: Use the loss value to update the network parameters in the initial deep learning network model until the number of iterations of the network parameters reaches a preset number, thereby obtaining a trained deep learning network model.

[0096] In one embodiment of the present invention, the deep learning network model may be YOLOv10-S.

[0097] And, in one embodiment of the present invention, the images in the training set can be n A batch of 10 ...

[0098] Furthermore, in one embodiment of the present invention, the above loss function is the same as that in the prior art, and the embodiment of the present disclosure will not be described in detail here.

[0099] Furthermore, in one embodiment of the present invention, after the trained deep learning network model is obtained through the above steps, the trained deep learning network model can be verified using a verification data set. If the road feature recognition accuracy of each image in the verification data set is greater than 90%, the trained deep learning network model is determined as the target deep learning network model; otherwise, it is necessary to continue training the model based on the previous training, and repeat this process until the accuracy requirement is met.

[0100] Furthermore, in one embodiment of the present invention, after obtaining the target deep learning network model through the above steps, the target deep learning network model can be used to detect the road surface in the area of ​​interest in the bird's-eye view to obtain a first classification result of the road surface type.

[0101] And, in one embodiment of the present invention, after obtaining the first classification result of the road type through the above steps, the first estimation result of the road adhesion coefficient is determined through the mapping relationship between the first classification result and the road adhesion coefficient. Considering that the road adhesion coefficient is related to factors such as road wear, tire wear, and air temperature and humidity, and that the road adhesion coefficient range value is different at different driving speeds, based on this, the road adhesion coefficient range of each road type at different vehicle speeds is determined with reference to the existing published "GA / T643-2006 Technical Appraisal of Vehicle Driving Speed ​​for Typical Traffic Accident Forms", and the peak road adhesion coefficient is selected as the final detection result. Table 1 is a mapping relationship comparison table of road types and road peak adhesion coefficients proposed in an embodiment of the present invention.

[0102] Table 1

[0103]

[0104]

[0105] According to the first classification result of the road type, the first estimation result of the road adhesion coefficient is obtained by querying the corresponding relationship in Table 1. The first estimation result is represented by the vector [index c ,conf c ,e c ,v,UTC], where index c For road surface type, conf c represents the confidence level, e c represents the peak road adhesion coefficient, v represents the vehicle speed, and UTC represents the timestamp.

[0106] Step S4, performing a second estimation of the road adhesion coefficient based on the region of interest and the road surface point cloud to obtain a second estimation result of the road adhesion coefficient;

[0107] In one embodiment of the present invention, after the region of interest is determined through the above steps, a second estimation of the road adhesion coefficient can be performed based on the region of interest and the road surface point cloud to obtain a second estimation result of the road adhesion coefficient.

[0108] Specifically, in one embodiment of the present invention, the method for performing a second estimation of the road surface adhesion coefficient based on the region of interest and the road surface point cloud to obtain a second estimation result of the road surface adhesion coefficient may include the following steps:

[0109] Step S41, constructing a road surface point cloud feature library;

[0110] Step S42, calculating the similarity between the road surface point cloud in the region of interest and each road surface point cloud feature in the road surface point cloud feature library;

[0111] Step S43, determining a second classification result of the road surface type based on the similarity;

[0112] Step S44: determining a second estimation result of the road adhesion coefficient based on the second classification result of the road type.

[0113] In one embodiment of the present invention, laser radar point cloud data of different road surfaces under different lighting and weather conditions during the day and at night can be collected, and a road surface point cloud feature library can be constructed based on the reflection intensity of each point cloud data.

[0114] In one embodiment of the present invention, the reflection intensity of the point cloud when the laser radar scans the road surface obeys the Gaussian distribution. However, due to the presence of other material objects such as traffic signs on the road surface, the probability distribution of the reflection intensity of the point cloud contains noise. The probability density function f(I) of the reflection intensity I can be expressed as:

[0115]

[0116] Among them, α i is the mixing coefficient of the i-th Gaussian distribution, ranging from [0,1], N is the probability density function of the Gaussian distribution, μ i and σ i are the mean and standard deviation of the i-th Gaussian distribution respectively.

[0117] And, in one embodiment of the present invention, assuming that the amount of point cloud data collected is n, the set of corresponding reflection intensities is I = (I1, I2, ..., I j ,...,I n ) T , where the Gaussian model response Φ is defined as:

[0118]

[0119] in,

[0120] In one embodiment of the present invention, (α i ,μ i ,σ i ) is the iterative solution formula:

[0121]

[0122] Among them, when the iteration termination condition of the above iterative solution formula is triggered or the maximum number of iterations is reached, the corresponding (α i ,μ i ,σ i ), and based on (α i ,μ i ,σ i ) Determine the reflection intensity of the point cloud. Since the reflection intensity during the day and at night is quite different, a point cloud sub-feature library of typical road surfaces in two time periods is established based on the reflection intensity of the point cloud.

[0123] Furthermore, in one embodiment of the present invention, after constructing the pavement point cloud feature library through the above steps, it is necessary to calculate the similarity between the pavement point cloud in the area of ​​interest and each pavement point cloud feature in the pavement point cloud feature library to determine the second classification result of the pavement type.

[0124] Specifically, in one embodiment of the present invention, the method for calculating the similarity between the road surface point cloud in the region of interest and each road surface point cloud feature in the road surface point cloud feature library may include the following steps:

[0125] Step 1: determine the first reflection intensity of the road surface point cloud in the area of ​​interest;

[0126] Step 2: Based on the first reflection intensity and the second reflection intensity of each road point cloud feature in the road point cloud feature library, the similarity between the road point cloud in the area of ​​interest and each road point cloud feature in the road point cloud feature library is calculated by a similarity formula.

[0127] In one embodiment of the present invention, the first reflection intensity of the road surface point cloud in the region of interest may be determined by the probability density function of the reflection intensity, which will not be described in detail in this embodiment of the present invention.

[0128] Furthermore, in one embodiment of the present invention, the similarity between the road surface point cloud in the region of interest and each road surface point cloud feature in the road surface point cloud feature library can be calculated by a similarity formula determined based on Jensen–Shannon divergence, wherein the similarity formula is:

[0129] S i =1-JSD i (f scan (I),f i (I))

[0130] in,

[0131] And, f scan (I) is the first reflection intensity of the road point cloud in the area of ​​interest, f i (I) is the second reflection intensity of the i-th road point cloud in the road point cloud feature library, S i is the similarity between the road point cloud in the area of ​​interest and the i-th road point cloud in the road point cloud feature library.

[0132] Further, in one embodiment of the present invention, after obtaining the similarity between the road surface point cloud in the region of interest and each road surface point cloud feature in the road surface point cloud feature library through the above steps, the similarity can be sorted in descending order, and the maximum similarity S max The corresponding road surface type is used as the second classification result of the road surface type. After the road surface detection in the two sub-feature libraries of daytime and nighttime is completed through the above steps, the results obtained respectively are merged to obtain the second classification result of the road surface type. And, in one embodiment of the present invention, after the second classification result of the road surface type is obtained through the above steps, the second estimation result of the road surface adhesion coefficient can be obtained by querying Table 1, wherein the second estimation result can be a vector [index l ,conf l ,e l ,v,UTC], where index l For road surface type, conf l Indicates confidence, conf l =S max , e lrepresents the peak road adhesion coefficient, v represents the vehicle speed, and UTC represents the timestamp.

[0133] Step S5, fusing the first estimation result and the second estimation result of the road adhesion coefficient to obtain a target estimation result of the road adhesion coefficient.

[0134] In one embodiment of the present invention, after obtaining the first estimation result and the second estimation result of the road adhesion coefficient through the above steps, the first estimation result and the second estimation result of the road adhesion coefficient may be fused to obtain a target estimation result of the road adhesion coefficient.

[0135] Specifically, in one embodiment of the present invention, the method of fusing the first estimation result and the second estimation result of the road adhesion coefficient to obtain the target estimation result of the road adhesion coefficient may include: fusing the first estimation result and the second estimation result of the road adhesion coefficient by a weighted least squares method to obtain the target estimation result of the road adhesion coefficient.

[0136] In one embodiment of the present invention, the target estimation result of the road adhesion coefficient obtained through the above steps is saved as a vector [e, conf, v, UTC], where e is the fused road adhesion coefficient, conf is the fused road adhesion coefficient confidence, v is the vehicle speed, and UTC is the timestamp. The fused road adhesion coefficient e is:

[0137] e=(conf c +conf l ) -1 (conf c ·e c +conf l ·e l );

[0138] The confidence conf after fusion is:

[0139]

[0140] According to the road adhesion coefficient estimation method based on image and point cloud fusion proposed in an embodiment of the present invention, the road adhesion coefficient is estimated by fusing the acquired image with the lidar point cloud. While giving full play to the high detection accuracy and real-time performance of image detection, the point cloud's insensitivity to illumination is used to perform redundant supplementation for image detection, thereby covering more complex scenes. While ensuring real-time performance and accuracy, the reliability of the estimation method under different environmental conditions is effectively improved.

[0141] Next, a road adhesion coefficient estimation device based on image and point cloud fusion proposed in an embodiment of the present invention is described with reference to the accompanying drawings.

[0142] Figure 3 The figure is a schematic diagram of the structure of a road adhesion coefficient estimation device based on image and point cloud fusion according to an embodiment of the present invention.

[0143] like Figure 3 As shown, the road adhesion coefficient estimation device 10 based on image and point cloud fusion includes: an acquisition module 301, a determination module 302, a first estimation module 303, a second estimation module 304 and a fusion module 305, wherein

[0144] An acquisition module 301 is used to acquire sensor signals of the vehicle and images and road surface point clouds in front of the vehicle;

[0145] A determination module 302 for determining an area of ​​interest for estimating a road adhesion coefficient through vehicle kinematics based on the sensor signal;

[0146] A first estimation module 303, configured to perform a first estimation of the road adhesion coefficient based on the region of interest and the image, and obtain a first estimation result of the road adhesion coefficient;

[0147] A second estimation module 304, configured to perform a second estimation of the road adhesion coefficient based on the region of interest and the road point cloud, and obtain a second estimation result of the road adhesion coefficient;

[0148] The fusion module 305 is used to fuse the first estimation result and the second estimation result of the road adhesion coefficient to obtain a target estimation result of the road adhesion coefficient.

[0149] Furthermore, the sensor signal includes the current vehicle speed and the front wheel steering angle; the above-mentioned determination module 302 is specifically used for:

[0150] Determine the current location of the vehicle;

[0151] Based on the vehicle's current position, current speed and front wheel angle, the vehicle's predicted trajectory is obtained through the vehicle's kinematic equations;

[0152] Based on the predicted trajectory and the preset width, the region of interest for road adhesion coefficient estimation is determined.

[0153] Furthermore, the first estimation module 303 is specifically configured to:

[0154] Convert the image to obtain the corresponding bird's-eye view;

[0155] The target deep learning network model is used to detect the road surface in the area of ​​interest in the bird's-eye view to obtain the first classification result of the road surface type;

[0156] Based on the first classification result of the road surface type, a first estimation result of the road surface adhesion coefficient is determined.

[0157] Furthermore, the above device is also used for:

[0158] Acquire first training data, and filter the first training data to obtain second training data;

[0159] Labeling the second training data to obtain third training data;

[0160] Based on the third training data, a corresponding training data set and a verification data set are obtained;

[0161] The initial deep learning network model is trained based on the training data set to obtain a trained deep learning network model;

[0162] The validation data set is used to validate the trained deep learning network model to obtain the target deep learning network model.

[0163] Furthermore, the second estimation module 304 is specifically configured to:

[0164] Construct a road point cloud feature library;

[0165] Calculate the similarity between the road point cloud in the area of ​​interest and each road point cloud feature in the road point cloud feature library;

[0166] Based on the similarity, determining a second classification result of the road surface type;

[0167] Based on the second classification result of the road surface type, a second estimation result of the road surface adhesion coefficient is determined.

[0168] Furthermore, the second estimation module 304 is further configured to:

[0169] Determine the first reflection intensity of the road surface point cloud within the region of interest;

[0170] Based on the second reflection intensity and the first reflection intensity of each road surface point cloud feature in the road surface point cloud feature library, the similarity between the road surface point cloud in the area of ​​interest and each road surface point cloud feature in the road surface point cloud feature library is calculated by a similarity formula.

[0171] Furthermore, the above fusion module 305 is specifically used for:

[0172] The first estimation result and the second estimation result of the road adhesion coefficient are fused by a weighted least square method to obtain a target estimation result of the road adhesion coefficient.

[0173] It should be noted that the above explanation of the method embodiment is also applicable to the device of this embodiment, and will not be repeated here.

[0174] According to the device for estimating the road adhesion coefficient based on the fusion of image and point cloud proposed in the embodiment of the present invention, the road adhesion coefficient is estimated by fusing the acquired image with the lidar point cloud. While giving full play to the high detection accuracy and real-time performance of image detection, the point cloud is used to perform redundant supplementation for image detection by utilizing the characteristic that it is insensitive to illumination, thereby covering more complex scenes. While ensuring real-time performance and accuracy, the reliability of the estimation method under different environmental conditions is effectively improved.

[0175] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0176] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0177] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A road adhesion coefficient estimation method based on image and point cloud fusion, characterized in that: The method comprises: Acquire the sensor signals of the vehicle and the image and road point cloud in front of the vehicle; determining a region of interest for estimating road adhesion coefficients through vehicle kinematics based on the sensor signals; Performing a first estimation of a road surface adhesion coefficient based on the region of interest and the image to obtain a first estimation result of the road surface adhesion coefficient; Performing a second estimation of the road surface adhesion coefficient based on the region of interest and the road surface point cloud to obtain a second estimation result of the road surface adhesion coefficient; The first estimation result and the second estimation result of the road adhesion coefficient are fused to obtain a target estimation result of the road adhesion coefficient.

2. The method according to claim 1, characterized in that The sensor signal includes the current vehicle speed and the front wheel steering angle; the region of interest for estimating the road adhesion coefficient based on the sensor signal and determined by vehicle kinematics includes: determining a current position of the vehicle; Based on the current position of the vehicle, the current vehicle speed and the front wheel steering angle, a predicted trajectory of the vehicle is obtained through a vehicle kinematic equation; Based on the predicted trajectory and the preset width, a region of interest for estimating the road adhesion coefficient is determined.

3. The method according to claim 1, characterized in that The first estimating of the road adhesion coefficient based on the region of interest and the image to obtain a first estimation result of the road adhesion coefficient includes: Converting the image to obtain a corresponding bird's-eye view; Detecting the road surface in the area of ​​interest in the bird's-eye view using a target deep learning network model to obtain a first classification result of the road surface type; Based on the first classification result of the road surface type, a first estimation result of the road surface adhesion coefficient is determined.

4. The method according to claim 3, characterized in that Before detecting the road surface in the region of interest in the bird's-eye view by using the target deep learning network model to obtain a classification result of the road surface type, the method further includes: Acquire first training data, and filter the first training data to obtain second training data; Labeling the second training data to obtain third training data; Based on the third training data, obtain a corresponding training data set and a verification data set; Training the initial deep learning network model based on the training data set to obtain a trained deep learning network model; The trained deep learning network model is verified using the verification data set to obtain a target deep learning network model.

5. The method according to claim 1, characterized in that The second estimation of the road adhesion coefficient based on the region of interest and the road point cloud to obtain a second estimation result of the road adhesion coefficient includes: Build a road point cloud feature library; Calculating the similarity between the road surface point cloud in the region of interest and each road surface point cloud feature in the road surface point cloud feature library; Based on the similarity, determining a second classification result of the road surface type; Based on the second classification result of the road surface type, a second estimation result of the road surface adhesion coefficient is determined.

6. The method according to claim 5, characterized in that The calculating the similarity between the road surface point cloud in the region of interest and each road surface point cloud feature in the road surface point cloud feature library comprises: Determine a first reflection intensity of the road surface point cloud within the region of interest; Based on the second reflection intensity of each road surface point cloud feature in the road surface point cloud feature library and the first reflection intensity, the similarity between the road surface point cloud in the area of ​​interest and each road surface point cloud feature in the road surface point cloud feature library is calculated by a similarity formula.

7. The method according to claim 1, characterized in that The step of fusing the first estimation result and the second estimation result of the road adhesion coefficient to obtain a target estimation result of the road adhesion coefficient includes: fusing the first estimation result and the second estimation result of the road adhesion coefficient by a weighted least squares method to obtain a target estimation result of the road adhesion coefficient.

8. A road adhesion coefficient estimation device based on image and point cloud fusion, characterized in that: The device comprises: An acquisition module, used to acquire the sensor signals of the vehicle and the image and road point cloud in front of the vehicle; a determination module for determining an area of ​​interest for estimating a road adhesion coefficient through vehicle kinematics based on the sensor signal; A first estimation module, configured to perform a first estimation of a road adhesion coefficient based on the region of interest and the image, and obtain a first estimation result of the road adhesion coefficient; A second estimation module, configured to perform a second estimation of a road adhesion coefficient based on the region of interest and the road point cloud to obtain a second estimation result of the road adhesion coefficient; The fusion module is used to fuse the first estimation result and the second estimation result of the road adhesion coefficient to obtain a target estimation result of the road adhesion coefficient.

9. An electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images