Road slippery area detection and early warning method based on multimodal data fusion

Abstract

The present invention discloses a method and device for detecting and warning slippery areas of a road surface based on multimodal data fusion, which extracts image features based on an image segmentation algorithm to obtain road surface detection results; obtains point cloud road surface detection results based on a point cloud segmentation algorithm; fuses the two detection results to obtain a road surface detection area; fuses the laser point cloud reflection intensity information of the road surface area with the image features, uses a pixel point bidirectional matching method to increase the degree of correlation between similar pixel point features, and uses a road surface dry and wet state segmentation network to obtain a road surface dry and wet state detection result; constructs a vehicle motion equation based on vehicle motion state information and predicts the vehicle motion trajectory and speed based on a particle filter algorithm; and provides a warning to the driver based on the road surface dry and wet state and vehicle trajectory prediction results. In this way, the road surface state in front of the vehicle can be detected in real time, the risk of vehicle instability can be evaluated, and a warning can be provided to the driver when the vehicle is at risk of instability, thereby improving driving safety and reducing the occurrence of traffic accidents.

Description

Technical Field

[0001] The present invention relates to the field of unmanned driving technology, and in particular to a road slippery area detection and early warning method based on multimodal data fusion. Background Art

[0002] Slippery road conditions caused by bad weather will reduce the tire-road adhesion coefficient between the vehicle and the road, which will increase the difficulty of vehicle control and increase driving risks. Detecting slippery road conditions and providing drivers with early warnings before the vehicle enters a slippery area can reduce traffic accidents and is of great significance to improving driving safety.

[0003] Many domestic and foreign institutions have conducted relevant research on the problem of detecting slippery road areas. Almazan E J et al. (2016) proposed a slippery road state detection algorithm based on vehicle-mounted cameras. The algorithm estimates the road vanishing point and spatial horizontal line through geometric methods, then segments the road through prior knowledge of road morphology, and finally uses a Bayesian classifier to classify the dry and wet state of the road. Asuzu P et al. (2018) proposed a slippery road state detection algorithm based on vehicle-mounted millimeter-wave radar. This method studies the echo intensity of millimeter-wave signals within 0 to 1 meter, and completes the classification of slippery road conditions through the mean of the distribution histogram of the echo intensity. Shin J et al. (2019) proposed a slippery road state detection algorithm based on lidar. This method collects road laser point cloud data, projects the point cloud in three-dimensional space to a two-dimensional plane, and completes the detection of slippery road areas by comparing and analyzing the reflection intensity values ​​of the point cloud.

[0004] Most existing algorithms are based on a single sensor to detect the slippery state of the road, which cannot specifically detect the slippery area of ​​the road. In addition, a single sensor is easily affected by conditions such as lighting, which makes the detection results lack robustness. Existing methods do not specifically design a mechanism to provide drivers with early warnings after detecting the slippery state of the road. Therefore, it is of great theoretical and practical significance to study and design a method for detecting slippery areas on the road based on multi-sensor data fusion and design a mechanism to provide drivers with early warnings. Summary of the invention

[0005] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0006] To this end, an object of the present invention is to propose a method for detecting and warning slippery road areas based on multimodal data fusion, which can detect slippery road areas and provide warnings before vehicles are in danger.

[0007] Another object of the present invention is to provide a road slippery area detection and early warning device based on multimodal data fusion.

[0008] To achieve the above object, an embodiment of the present invention provides a method for detecting and warning slippery road areas based on multimodal data fusion, comprising the following steps:

[0009] Collect road images and laser point clouds through calibrated on-board cameras and lidar;

[0010] Detect the road surface image based on an image segmentation algorithm to obtain an image feature map and an image road surface detection result;

[0011] The laser point cloud is screened based on a point cloud segmentation algorithm to obtain a point cloud road surface detection result;

[0012] Fusing the point cloud road surface detection result with the image road surface detection result to obtain a fused road surface detection area;

[0013] The laser point cloud reflection intensity information of the road surface detection area is integrated with the image features, a pixel bidirectional matching method is used to increase the correlation degree of similar pixel features, and a road surface dry and wet state segmentation network is used to obtain a road surface dry and wet state detection result;

[0014] Collect vehicle motion state information, construct vehicle motion equations, and predict vehicle motion trajectory and speed based on particle filter algorithm;

[0015] The vehicle is warned based on the road surface wetness and dryness detection result and the vehicle movement trajectory and speed.

[0016] To achieve the above object, another embodiment of the present invention proposes a road slippery area detection and early warning device based on multimodal data fusion, comprising:

[0017] The acquisition module is used to collect road images and laser point clouds through calibrated on-board cameras and laser radars;

[0018] A first detection module is used to detect the road surface image based on an image segmentation algorithm to obtain an image feature map and an image road surface detection result;

[0019] A second detection module is used to screen the laser point cloud based on a point cloud segmentation algorithm to obtain a point cloud road surface detection result;

[0020] A fusion module, used to fuse the point cloud road surface detection result with the image road surface detection result to obtain a fused road surface detection area;

[0021] The third detection module is used to fuse the laser point cloud reflection intensity information of the road surface detection area with the image features, use the pixel bidirectional matching method to increase the correlation degree of similar pixel features, and use the road surface dry and wet state segmentation network to obtain the road surface dry and wet state detection result;

[0022] The prediction module is used to collect vehicle motion state information, construct the vehicle motion equation and predict the vehicle motion trajectory and speed based on the particle filter algorithm;

[0023] The early warning module is used to warn the vehicle based on the road surface dryness and wetness detection result and the vehicle movement trajectory and speed.

[0024] The method and device for detecting and warning slippery areas on a road surface based on multimodal data fusion according to the embodiments of the present invention have the following beneficial effects:

[0025] 1) Enhance the laser point cloud in the pixel coordinate system to make the density of the laser point cloud in the road area consistent with the pixel density of the image, and assist in the subsequent target-level fusion of the laser point cloud and the image.

[0026] 2) Data fusion methods were used in the road surface detection stage and the road slippery area detection stage, respectively, which improved the accuracy of road surface detection and road slippery area detection.

[0027] 3) Perform bidirectional pixel feature matching on the feature map after data fusion to increase the correlation between similar features and expand the distinction between different types of features, making the detection results of slippery road areas more robust.

[0028] 4) Based on the detection results of slippery road areas and the prediction results of vehicle trajectory, the risk of the vehicle falling into an unstable state is assessed, and the driver can be warned before the vehicle is in danger.

[0029] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0031] Figure 1 A flow chart of a method for detecting and warning a slippery road area based on multimodal data fusion according to an embodiment of the present invention;

[0032] Figure 2 A flowchart of a method for detecting and warning a slippery road area based on multimodal data fusion according to an embodiment of the present invention;

[0033] Figure 3 This is a structural diagram of an Encoder of a road segmentation algorithm based on a deep neural network according to an embodiment of the present invention;

[0034] Figure 4A structure diagram of a Decoder of a road segmentation algorithm based on a deep neural network according to an embodiment of the present invention;

[0035] Figure 5 A structure diagram of a road segmentation algorithm PredictionHead based on a deep neural network according to an embodiment of the present invention;

[0036] Figure 6 A schematic diagram of point cloud bidirectional matching according to an embodiment of the present invention;

[0037] Figure 7 A network structure diagram of road surface dry and wet state segmentation according to an embodiment of the present invention;

[0038] Figure 8 A schematic diagram of vehicle trajectory prediction according to an embodiment of the present invention;

[0039] Fig. 9 The figure is a schematic diagram of the structure of a device for detecting and warning slippery areas on a road surface based on multimodal data fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0041] The following describes a method and device for detecting and warning slippery areas on a road surface based on multimodal data fusion according to an embodiment of the present invention with reference to the accompanying drawings.

[0042] First, a road slippery area detection and early warning method based on multimodal data fusion proposed according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0043] like Figure 1 and Figure 2 As shown, the road slippery area detection and early warning method based on multimodal data fusion includes the following steps:

[0044] In step S1, road surface images and laser point clouds are collected through calibrated vehicle-mounted cameras and laser radars.

[0045] Before collecting road images and laser point clouds, the vehicle-mounted camera and laser radar are first jointly calibrated to obtain the intrinsic parameter matrix and extrinsic parameter matrix of the vehicle-mounted camera.

[0046] Specifically, the calibration steps are:

[0047] In the software environment inside the vehicle, configure the calibration toolbox based on Autoware. After starting the calibration toolbox, the calibrator places the checkerboard calibration plate in front of the vehicle at three positions: far, medium, and near the vehicle;

[0048] Use the calibration toolbox to collect images and laser point cloud data containing the calibration plate in the field of view;

[0049] Use the built-in checkerboard calibration method in the calibration toolbox to obtain the camera intrinsic parameter matrix P; obtain the extrinsic parameter matrix T by associating the position of the calibration plate in the image and the laser point cloud. r .

[0050] In step S2, the road surface image is detected based on the image segmentation algorithm to obtain an image feature map and an image road surface detection result.

[0051] Optionally, in one embodiment of the present invention, a road surface image is detected based on an image segmentation algorithm to obtain image features and image road surface detection results, including: constructing an image segmentation algorithm model; performing data enhancement through a training set and training the image segmentation algorithm model; and obtaining an image feature map of the road surface image and image road surface detection results through the trained image segmentation algorithm model.

[0052] In one embodiment of the present invention, the image segmentation algorithm model includes a feature extraction unit, a fusion unit and a prediction unit; the feature extraction unit includes multiple series-connected convolution layers, pooling layers and activation functions, which are used to calculate the features of the road surface image and downsample the image, and output image feature maps of different scales; the fusion unit includes multiple series-connected convolution layers, pooling layers and activation functions, which are used to fuse the image feature maps of different scales output by the feature extraction unit and restore the resolution of the feature maps; the prediction unit is used to predict the road surface area in the road surface image according to the image feature map output by the fusion unit to obtain the image road surface detection result.

[0053] Specifically, an image segmentation algorithm model based on a deep neural network is designed, which includes an Encoder (such as Figure 3 As shown in ), Decoder (as shown in Figure 4 ) and Prediction Head (as shown Figure 5 The encoder consists of three parts: the encoder and decoder. The encoder consists of multiple convolutional layers, pooling layers, and activation functions connected in series. The encoder calculates the features of the input image and downsamples the image to output image feature maps of different scales. The decoder also consists of multiple convolutional layers, pooling layers, and activation functions connected in series. The decoder fuses the feature maps of different scales output by the encoder and gradually restores the resolution of the feature maps. The prediction head uses the feature map output by the decoder as input to predict the road surface area in the image.

[0054] Perform data augmentation on the training set and train the model constructed by the steps. The data augmentation methods include: random scaling of images, random flipping of images, random cropping of images, random adjustment of image brightness, random adjustment of image contrast, and random adjustment of image saturation.

[0055] During the detection process, for each input image, the algorithm simultaneously outputs the image feature map calculated by the Decoder and the road surface prediction result obtained by the Prediction Head.

[0056] In step S3, the laser point cloud is screened based on the point cloud segmentation algorithm to obtain the point cloud road surface detection result.

[0057] Use the RANSAC-based point cloud segmentation algorithm to filter the points belonging to the road surface in the laser point cloud captured in step S1.

[0058] Optionally, in one embodiment of the present invention, the laser point cloud is screened based on a point cloud segmentation algorithm to obtain a point cloud road surface detection result, including: for each frame of laser point cloud, determining a plane model through multiple laser points; determining the number of laser points in the plane model according to the distance between the laser point and the plane model; determining the optimal model according to the number of laser points, and determining the point cloud road surface detection result using the optimal model.

[0059] Specifically, as a specific implementation method, for each frame of laser point cloud, a plurality of laser points, such as 3 laser points, are randomly selected, and a plane model determined by the 3 points is calculated.

[0060] Calculate whether other laser points satisfy the plane model according to the set threshold. If they do, record them as the interior points of the model, and record the total number of interior points of the model. For example, calculate the distance from other laser points to the plane model. If the distance is less than 0.2, record it as the interior point of the model, and record the total number of interior points of the model.

[0061] Repeat the iteration N times, such as 100 times. If the total number of inliers of the model generated each time is higher than that of the existing model, it will be recorded as the optimal model, and the inliers of the optimal model will be recorded. The inliers are the detected road surface.

[0062] In step S4, the point cloud road surface detection result is fused with the image road surface detection result to obtain a fused road surface detection area.

[0063] Optionally, in one embodiment of the present invention, the point cloud road surface detection results are fused with the image road surface detection results to obtain a fused road surface detection area, including: mapping the point cloud road surface detection results to the image through the intrinsic parameter matrix and extrinsic parameter matrix of the vehicle-mounted camera, and using a point cloud enhancement method in a pixel coordinate system to obtain a dense point cloud; fusing the dense point cloud with the image road surface detection results to obtain a road surface detection area after data fusion.

[0064] Specifically, the road surface laser point cloud obtained in step S3 is mapped to the image pixel coordinate system, and the mapping formula is:

[0065] y=PT r x

[0066] Among them, x is the coordinate of the laser point cloud in the world coordinate system, P is the camera internal parameter matrix, T r is the external parameter matrix, and y is the pixel coordinate of the point cloud in the image.

[0067] In the pixel coordinate system, the information of the laser point cloud except the height (x coordinate in the world coordinate system, y coordinate in the world coordinate system, and reflection intensity) is enhanced to make it a dense point cloud. The formula for point cloud enhancement is:

[0068]

[0069]

[0070] Among them, x0 and x1 are the horizontal coordinates of two adjacent points in the x-axis direction, x0<x1, x∈(x0,x1); y0 and y1 are the vertical coordinates of two adjacent points in the y-axis direction, y0<y1, y∈(y0,y1).

[0071] The enhanced point cloud is a dense road surface detection result. It is fused with the road surface detection result obtained in step 2, and the union of the detection results is taken to obtain the road surface detection result after data fusion. According to the fused road surface detection result, the reflection intensity information of the point cloud at the corresponding position is output.

[0072] In step S5, the laser point cloud reflection intensity information of the road surface detection area is fused with the image features, the pixel bidirectional matching method is used to increase the correlation between similar pixel features, and the road surface dry and wet state segmentation network is used to obtain the road surface dry and wet state detection result.

[0073] It can be understood that according to the road surface detection result obtained in step S4, the reflection intensity information of the laser point cloud in the road surface area is obtained and feature fusion is performed with the image features obtained in step S2. The pixel bidirectional matching method is used for the feature map obtained after fusion to increase the correlation degree of similar pixel features, and then the road surface dry and wet state segmentation network is used to obtain the road surface dry and wet state detection result.

[0074] Specifically, the image feature map obtained by the Decoder in step S2 and the point cloud reflection intensity information output in step S4 are feature spliced ​​to obtain a feature map after feature fusion.

[0075] Perform bidirectional matching on each pixel of the fused feature map to increase the correlation between features of similar pixels. The specific process is as follows:

[0076] (1) Assume that the feature map has a total of N pixels: {a1,…,a N}, the feature vector at each pixel is: {b1,…,b N}. The similarity between each pixel and all other pixels is calculated by multiplying the feature vector.

[0077] (2) For each pixel a k , k∈[1,N], take the n points with the highest similarity, recorded as: Among them, n is half of the total number of pixels in the feature map.

[0078] (3) For all pixels i∈[1,n], if the n points with the highest similarity include pixel a k , then a k and match, Figure 6 The matching relationship of pixel a1 is shown in the form of a schematic diagram when the feature map has a total of 4 pixels.

[0079] (4) Take all the values ​​that are equal to a k For the matched pixels, the similarity is normalized and used as the weight to perform weighted summation on the feature vectors. The new feature vector obtained is used to update the pixel a k The characteristic value of .

[0080] The feature map obtained after bidirectional matching of pixels is input into the road surface wet / dry state segmentation network to obtain the road surface wet / dry state detection result. Figure 7 As shown in the figure, the road surface wet and dry state segmentation network consists of several convolutional layers, pooling layers and activation functions.

[0081] In step S6, the vehicle motion state information is collected, the vehicle motion equation is constructed, and the vehicle motion trajectory and speed are predicted based on the particle filter algorithm.

[0082] The vehicle speed, steering wheel angle and other motion state information are collected through the vehicle bus, the vehicle motion equation is constructed, and the vehicle motion trajectory and speed are predicted based on the particle filter algorithm.

[0083] Specifically, the two-dimensional plane on the ground is set as the global coordinate system, with the current vehicle position as the origin, and the x and y axis directions are the same as the vehicle coordinate system.

[0084] Collect vehicle motion status information: x ,d y ,v,θ}, where d x ,d yare the horizontal and vertical coordinates of the vehicle in the global coordinate system, v is the speed of the vehicle, and θ is the direction of the vehicle.

[0085] Establish the vehicle motion equations:

[0086]

[0087] 6 k+1 =θ k +δ θ

[0088] v k+1 =v k +δ v

[0089] is the vehicle position change equation, where v k is the absolute value of the vehicle speed at time k, θ k θ is the orientation angle of the vehicle at time k. k+1 is the vehicle heading angle change equation, and the noise δ is superimposed θ v is the vehicle speed change equation, and the noise δ is superimposed v .

[0090] According to the vehicle motion equation and particle filter algorithm, the position and speed of the vehicle within 3-4 seconds are predicted and its probability distribution is output, such as Figure 8 shown.

[0091] In step S7, the vehicle is warned based on the road surface wetness and dryness detection result and the vehicle movement trajectory and speed.

[0092] Optionally, in one embodiment of the present invention, the vehicle is warned based on the road wet and dry state detection results and the vehicle movement trajectory and speed, including: calculating the probability of the vehicle passing through a slippery area of ​​the road and the probability that the speed of the vehicle is greater than a threshold speed when passing through the slippery area, and warning the vehicle based on the probability and the probability threshold.

[0093] Specifically, the road surface wetness detection result obtained in step S5 is mapped to the global coordinate system using the 3D coordinates of the point cloud. The threshold speed v is set according to the wetness of the wet area. The greater the wetness of the road surface, the smaller the threshold speed.

[0094] According to the vehicle trajectory prediction result obtained in step S6, the probability p1 of the vehicle passing through the slippery area and the probability p2 of the vehicle speed being greater than the threshold speed when passing through the slippery area are estimated. If p1p2≥50%, a warning is provided to the driver.

[0095] According to the method for detecting and warning slippery road areas based on multimodal data fusion proposed in the embodiment of the present invention, the road image and laser point cloud are collected by calibrated vehicle-mounted cameras and laser radars; the road image is detected based on the image segmentation algorithm to obtain the image feature map and the image road detection result; the laser point cloud is screened based on the point cloud segmentation algorithm to obtain the point cloud road detection result; the point cloud road detection result is fused with the image road detection result to obtain the fused road detection area; the laser point cloud reflection intensity information of the road detection area is fused with the image feature, the pixel bidirectional matching method is used to increase the correlation degree of the same pixel feature, and the road dry and wet state segmentation network is used to obtain the road dry and wet state detection result; the vehicle motion state information is collected, the vehicle motion equation is constructed and the vehicle motion trajectory and speed are predicted based on the particle filter algorithm; the vehicle is warned according to the road dry and wet state detection result and the vehicle motion trajectory and speed. The road state in front of the vehicle can be detected in real time, the risk of vehicle instability can be evaluated, and the driver can be warned when the vehicle is at risk of instability, which is of great significance to improving driving safety and reducing traffic accidents.

[0096] Next, a road slippery area detection and early warning device based on multimodal data fusion proposed according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0097] Fig. 9 The figure is a schematic diagram of the structure of a device for detecting and warning slippery areas on a road surface based on multimodal data fusion according to an embodiment of the present invention.

[0098] like Fig. 9 As shown, the road slippery area detection and warning device based on multimodal data fusion includes: an acquisition module 100, a first detection module 200, a second detection module 300, a fusion module 400, a third detection module 500, a prediction module 600 and a warning module 700.

[0099] Among them, the acquisition module 100 is used to collect road images and laser point clouds through calibrated on-board cameras and laser radars. The first detection module 200 is used to detect road images based on image segmentation algorithms to obtain image feature maps and image road detection results. The second detection module 300 is used to screen laser point clouds based on point cloud segmentation algorithms to obtain point cloud road detection results. The fusion module 400 is used to fuse the point cloud road detection results with the image road detection results to obtain a fused road detection area. The third detection module 500 is used to fuse the laser point cloud reflection intensity information of the road detection area with the image features, use the pixel point bidirectional matching method to increase the correlation degree of similar pixel point features, and use the road wet and dry state segmentation network to obtain the road wet and dry state detection results. The prediction module 600 is used to collect vehicle motion state information, construct the vehicle motion equation and predict the vehicle motion trajectory and speed based on the particle filter algorithm. The warning module 700 is used to warn the vehicle based on the road wet and dry state detection results and the vehicle motion trajectory and speed.

[0100] Optionally, in one embodiment of the present invention, it also includes: a calibration module, used to jointly calibrate the vehicle-mounted camera and the laser radar to obtain the intrinsic parameter matrix and the extrinsic parameter matrix of the vehicle-mounted camera.

[0101] It should be noted that the above explanation of the method embodiment is also applicable to the device of this embodiment, and will not be repeated here.

[0102] According to the multimodal data fusion-based wet road area detection and early warning device proposed in the embodiment of the present invention, the road image and laser point cloud are collected by calibrated vehicle-mounted cameras and laser radars; the road image is detected based on the image segmentation algorithm to obtain the image feature map and the image road detection result; the laser point cloud is screened based on the point cloud segmentation algorithm to obtain the point cloud road detection result; the point cloud road detection result is fused with the image road detection result to obtain the fused road detection area; the laser point cloud reflection intensity information of the road detection area is fused with the image feature, the pixel bidirectional matching method is used to increase the correlation degree of the same pixel feature, and the road dry and wet state segmentation network is used to obtain the road dry and wet state detection result; the vehicle motion state information is collected, the vehicle motion equation is constructed, and the vehicle motion trajectory and speed are predicted based on the particle filter algorithm; the vehicle is warned according to the road dry and wet state detection result and the vehicle motion trajectory and speed. The road state in front of the vehicle can be detected in real time, the risk of vehicle instability can be evaluated, and the driver can be warned when the vehicle is at risk of instability, which is of great significance to improving driving safety and reducing traffic accidents.

[0103] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0104] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0105] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for detecting and warning slippery road areas based on multimodal data fusion, characterized in that: The following steps are involved: Collect road images and laser point clouds through calibrated on-board cameras and lidar; Detect the road surface image based on an image segmentation algorithm to obtain an image feature map and an image road surface detection result; The laser point cloud is screened based on a point cloud segmentation algorithm to obtain a point cloud road surface detection result; Fusing the point cloud road surface detection result with the image road surface detection result to obtain a fused road surface detection area; The laser point cloud reflection intensity information of the road surface detection area is integrated with the image features, a pixel bidirectional matching method is used to increase the correlation degree of similar pixel features, and a road surface dry and wet state segmentation network is used to obtain a road surface dry and wet state detection result; Collect vehicle motion state information, construct vehicle motion equations, and predict vehicle motion trajectory and speed based on particle filter algorithm; The vehicle is warned based on the road surface wetness and dryness detection result and the vehicle movement trajectory and speed.

2. The method according to claim 1, characterized in that Also includes: The vehicle-mounted camera and the laser radar are jointly calibrated to obtain the intrinsic parameter matrix and the extrinsic parameter matrix of the vehicle-mounted camera.

3. The method according to claim 1, characterized in that: The detecting the road surface image based on the image segmentation algorithm to obtain image features and image road surface detection results includes: Build an image segmentation algorithm model; Performing data enhancement through a training set and training the image segmentation algorithm model; The image feature map of the road surface image and the image road surface detection result are obtained through the trained image segmentation algorithm model.

4. The method according to claim 3, characterized in that The image segmentation algorithm model includes a feature extraction unit, a fusion unit and a prediction unit; The feature extraction unit includes a plurality of serially connected convolutional layers, pooling layers and activation functions, which are used to calculate the features of the road surface image and downsample the image to output image feature maps of different scales; The fusion unit includes a plurality of convolutional layers, pooling layers and activation functions connected in series, and is used to fuse the image feature maps of different scales output by the feature extraction unit and restore the resolution of the feature maps; The prediction unit is used to predict the road surface area in the road surface image according to the image feature map output by the fusion unit to obtain the image road surface detection result.

5. The method according to claim 4, characterized in that The data enhancement methods include: random scaling of images, random flipping of images, random cropping of images, random adjustment of image brightness, random adjustment of image contrast, and random adjustment of image saturation.

6. The method according to claim 1, characterized in that The step of screening the laser point cloud based on the point cloud segmentation algorithm to obtain the point cloud road surface detection result includes: For each frame of laser point cloud, a plane model is determined by multiple laser points; Determining the number of laser points in the plane model according to the distance between the laser point and the plane model; An optimal model is determined according to the number of laser points, and the point cloud road surface detection result is determined using the optimal model.

7. The method according to claim 2, characterized in that The step of fusing the point cloud road surface detection result with the image road surface detection result to obtain a fused road surface detection area includes: Mapping the point cloud road surface detection result to an image through the intrinsic parameter matrix and the extrinsic parameter matrix of the vehicle-mounted camera, and obtaining a dense point cloud using a point cloud enhancement method in a pixel coordinate system; The dense point cloud is fused with the image road surface detection result to obtain a road surface detection area after data fusion.

8. The method according to claim 1, characterized in that The step of providing an early warning to the vehicle according to the road surface wetness and dryness detection result and the vehicle movement trajectory and speed includes: The probability that the vehicle passes through a slippery road area and the probability that the speed of the vehicle is greater than a threshold speed when passing through the slippery road area are calculated, and the vehicle is warned according to the probability and the probability threshold.

9. A road slippery area detection and early warning device based on multimodal data fusion, characterized in that: include: The acquisition module is used to collect road images and laser point clouds through calibrated on-board cameras and laser radars; A first detection module is used to detect the road surface image based on an image segmentation algorithm to obtain an image feature map and an image road surface detection result; A second detection module is used to screen the laser point cloud based on a point cloud segmentation algorithm to obtain a point cloud road surface detection result; A fusion module, used to fuse the point cloud road surface detection result with the image road surface detection result to obtain a fused road surface detection area; The third detection module is used to fuse the laser point cloud reflection intensity information of the road surface detection area with the image features, use the pixel bidirectional matching method to increase the correlation degree of similar pixel features, and use the road surface dry and wet state segmentation network to obtain the road surface dry and wet state detection result; The prediction module is used to collect vehicle motion state information, construct the vehicle motion equation and predict the vehicle motion trajectory and speed based on the particle filter algorithm; The early warning module is used to warn the vehicle based on the road surface dryness and wetness detection result and the vehicle movement trajectory and speed.

10. The device according to claim 9, characterized in that Also includes: The calibration module is used to jointly calibrate the vehicle-mounted camera and the laser radar to obtain the intrinsic parameter matrix and the extrinsic parameter matrix of the vehicle-mounted camera.

Images