A method for segmenting slippery road areas based on vehicle sensor data fusion

Abstract

This invention belongs to the field of autonomous driving technology and proposes a method for segmenting slippery road surfaces based on vehicle-mounted sensor data fusion. The method includes: extracting LiDAR data and filtering point clouds of road surfaces; extracting camera image data and calculating its color and texture features; projecting the point clouds of the road surfaces onto the image according to calibration parameters; estimating the reflection intensity of pixels in the road surface area not covered by the point cloud based on the color and texture features; fusing the estimated reflection intensity with the image data; designing a segmentation model containing a first branch and a second branch; training the segmentation model; and segmenting slippery road surfaces using the fused data as input. This invention estimates reflection intensity data, enabling pixels not covered by the point cloud to also possess reflection intensity data. By fusing image data and reflection intensity data, it enhances the segmentation model's ability to identify slippery road surfaces and improves the accuracy of the slippery road surface segmentation algorithm.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and more specifically, to a method for segmenting slippery road surfaces based on vehicle sensor data fusion. Background Technology

[0002] Slippery road conditions caused by rain and snow reduce the tire-road adhesion coefficient between vehicles and the road, increasing the difficulty of vehicle control and driving risks. Detecting slippery road areas can provide environmental information for autonomous driving decision-making algorithms, assisting them in path planning and providing safer driving strategies.

[0003] For the detection of slippery road surfaces, many institutions both domestically and internationally have conducted related research. For example, one method for detecting slippery road surfaces based on images and a VGG16 neural network model adds two fully connected layers to the VGG16 neural network model. It uses self-collected data and a low learning rate to fine-tune the parameters of the pre-trained model. After training, the model can classify input images and detect the slippery category of the road surface. Another example is a road surface slippery state detection algorithm based on the reflection intensity of laser point clouds. This method uses roadside lidar sensors to collect point clouds of the road surface area and, based on the physical principle that light reflection intensity is related to the surface state of an object, detects specific slippery areas of the road surface by measuring the differences in the reflection intensity of the point clouds.

[0004] Most existing algorithms detect the slippery state of the road surface based on a single sensor, and cannot detect the specific location of the slippery area; while some algorithms based on multi-sensor data fusion have difficulty solving the problem of inconsistent resolution between the image and the point cloud. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a method for segmenting slippery road surfaces based on vehicle-mounted sensor data fusion. The specific technical solution is as follows:

[0006] A method for segmenting slippery road areas based on vehicle-mounted sensor data fusion, characterized by the following steps:

[0007] S1, calibrate the lidar and camera mounted on the vehicle and obtain calibration parameters;

[0008] S2, during vehicle operation, extracts LiDAR data in real time and filters out point clouds of road surface areas; extracts camera image data in real time and calculates the color and texture features of the images;

[0009] S3, according to the calibration parameters, project the point cloud of the road surface area onto the image so that the pixels covered by the point cloud have reflection intensity data;

[0010] S4. Estimate the reflection intensity data of pixels in the road surface area that are not covered by the point cloud based on the color and texture features of the image;

[0011] S5, fuse the estimated reflection intensity data of the road surface area with the image data to obtain fused data;

[0012] S6. Establish a segmentation model containing a first branch and a second branch, train the segmentation model using historical fusion data, and delete the first branch after training to obtain a trained segmentation model.

[0013] S7 inputs the fused data obtained in S5 into the trained segmentation model to segment the slippery road surface area and output the data.

[0014] Preferably, S4 specifically includes:

[0015] S401, calculate the similarity of color and texture features between pixels in the road surface area that are not covered by the point cloud and the surrounding pixels that are covered by the point cloud.

[0016] S402, calculate the weight of the surrounding pixels covered by the point cloud in the reflection intensity estimation process based on color feature similarity and texture feature similarity.

[0017] S403 calculates the reflection intensity data of pixels not covered by the point cloud based on the weight of the surrounding pixels covered by the point cloud in the reflection intensity estimation process.

[0018] Preferably, the similarity of color features in S401 is calculated using the following formula:

[0019] ;

[0020] in, p These are the pixels in the road surface area that are not covered by the point cloud. Ω for p A collection of pixels surrounded by a point cloud. q for p The surrounding pixels are covered by a point cloud, and , for p and q Similarity of color features It is the minimum constant. for p Color characteristics, for q Color characteristics;

[0021] The similarity of texture features in S401 is calculated using the following formula:

[0022] ;

[0023] in, for p and q Similarity of texture features for p Texture features, for q Texture features.

[0024] Preferably, the weight of pixels surrounded by point cloud in the reflection intensity estimation process in S402 is calculated using the following formula:

[0025] ;

[0026] in, for q In estimation p The weighting of the reflection intensity data.

[0027] Preferably, the reflection intensity data of pixels not covered by the point cloud in S403 is calculated using the following formula:

[0028] ;

[0029] in, for q The intensity of reflection, For normalization coefficients, For the calculation p The intensity of reflection.

[0030] Preferably, S5 specifically includes:

[0031] S501 represents image data as three-channel data in the RGB color space and normalizes it;

[0032] S502 normalizes the estimated reflection intensity data of the road surface area and stitches it with the image data to become the fourth channel of the image data, thus obtaining fused data.

[0033] Preferably, S6 specifically includes:

[0034] S601 is a backbone network for a segmentation model designed based on a skip connection structure, with a first and second branch designed for prediction.

[0035] S602, the backbone network is connected to the first branch and the second branch respectively to form a segmentation model;

[0036] S603 inputs the collected historical fusion data into the segmentation model and trains it with a learning rate of 0.1 and 150 iterations.

[0037] S604: After training is complete, delete the parameters of the first branch and retain the parameters of the second branch used for segmenting wet and slippery road areas.

[0038] Preferably, the first branch includes a first prediction head and a first loss function for enhancing the training of backbone network parameters; the second branch includes a second prediction head and a second loss function for training the parameters of the segmentation model.

[0039] Preferably, the fused data obtained in S7 is real-time fused data.

[0040] Preferably, S1 specifically includes: installing a lidar above the vehicle and installing a camera above the windshield inside the vehicle; calibrating the camera using a checkerboard calibration method to correct distortion, and using the Autoware calibration toolkit to perform joint calibration of the camera and lidar.

[0041] Compared with existing technologies, the present invention has the following advantages:

[0042] (1) The road surface wet and slippery area segmentation method based on vehicle sensor data fusion provided by the present invention enhances the segmentation model’s ability to identify road surface wet and slippery areas by fusing the reflection intensity of images and point clouds, thereby improving the accuracy of the road surface wet and slippery area segmentation method.

[0043] (2) The present invention designs a segmentation model based on skip connections and double loss function, which makes the segmentation model easier to train; and after training, the parameters of the first branch are deleted, which can effectively reduce the computational amount of the segmentation model in actual calculation, making the segmentation model easier to solve the results, reducing the requirements and burden on hardware, and is conducive to widespread promotion.

[0044] (3) The method for segmenting wet and slippery road areas based on vehicle sensor data fusion provided by the present invention is based on the physical principle that the reflection intensity of light is related to the color and texture features of the object surface. It estimates the reflection intensity data of pixels in the road area that are not covered by point cloud based on the color and texture features of the image, so that more image pixels can have reflection intensity data during the fusion process, thereby improving the accuracy of the algorithm. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the embodiments will be briefly introduced below. The features and advantages of the present invention can be more clearly understood by referring to the accompanying drawings. The accompanying drawings are schematic and should not be construed as limiting the present invention in any way. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart of the method for segmenting slippery road areas based on vehicle sensor data fusion according to the present invention;

[0047] Figure 2 This is a schematic diagram showing the installation positions of the vehicle-mounted camera and lidar in this invention;

[0048] Figure 3 This is a schematic diagram of the network structure of the segmentation model of the present invention;

[0049] Figure 4 To filter out point clouds belonging to the road surface area using the RANSAC point cloud segmentation algorithm of this invention;

[0050] Figure 5 This is a schematic diagram of the texture features of the image in this invention;

[0051] Figure 6 This is a schematic diagram of the point cloud reflection intensity upsampling method of the present invention;

[0052] Figure 7 This is a diagram showing the upsampling results of the point cloud reflection intensity according to the present invention. Detailed Implementation

[0053] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0054] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0055] like Figure 1 As shown, this invention provides a method for segmenting slippery road surfaces based on vehicle-mounted sensor data fusion, comprising the following steps:

[0056] Step 1: As Figure 2 As shown, a camera and lidar are installed on the vehicle, and individual and joint calibrations of the sensors are performed to obtain calibration parameters.

[0057] The specific steps are as follows:

[0058] Step 101: Install a lidar on top of the vehicle and a camera on top of the windshield inside the vehicle;

[0059] Step 102: Use the checkerboard calibration method to calibrate the camera to correct distortion, and use the Autoware calibration toolbox to perform joint calibration of the camera and LiDAR to achieve the conversion of point cloud coordinates to image coordinate system.

[0060] Step 2: During vehicle operation, extract LiDAR data in real time and calculate the point cloud of the road surface area based on the point cloud segmentation algorithm.

[0061] The specific steps are as follows:

[0062] Step 201: Extract lidar point cloud data including the three-dimensional coordinates and reflection intensity of each point;

[0063] Step 202: As Figure 4 As shown, point clouds belonging to the road surface area are filtered out based on the three-dimensional coordinates of the points using the RANSAC point cloud segmentation algorithm.

[0064] Step 3: Extract camera image data in real time and calculate the color and texture features of the image.

[0065] The specific steps are as follows:

[0066] Step 301: Extract image data from the camera and convert the color space of the image data to the HSV color space;

[0067] Step 302: Use the H channel of the HSV color space as the color feature of the image data;

[0068] Step 303: As Figure 5 As shown, texture features of image data are calculated based on the gray-level co-occurrence matrix.

[0069] Step 4: Based on the calibration parameters obtained in Step 1, project the point cloud of the road surface area onto the image so that the pixels covered by the point cloud have reflection intensity data.

[0070] The specific steps are as follows:

[0071] Step 401: Extract the calibration parameters obtained in Step 1, where C For camera matrix, R For rotation matrix, T For translation vectors:

[0072] Step 402: Project the point cloud onto the image according to the formula:

[0073]

[0074] in P Let these be the coordinates of the point cloud in the world coordinate system. The coordinates of the point cloud in the image coordinate system are given.

[0075] Step 403: Obtain reflection intensity data for pixels covered by point cloud in the image.

[0076] Step 5: Based on the color and texture features obtained in Step 3 and the pixels with existing reflection intensity data obtained in Step 4, estimate the reflection intensity data of pixels without reflection intensity in the road surface area.

[0077] The specific steps are as follows:

[0078] Step 501: As Figure 6 As shown, let p For the pixels in the image whose reflection intensity needs to be estimated, Ω for p The set of pixels within a surrounding 3x3 area. . For color characteristics, For texture features, express p Color characteristics, express p Texture features, express q Color characteristics, express q Texture features. Figure 6 In the point cloud reflection intensity map, the × at the center point of the 3×3 region in the upper left corner represents the reflection intensity obtained by estimation, and the other × represent pixels with existing reflection intensity.

[0079] Step 502: Calculation p With pixel set Ω Similarity of color features of intrapixels:

[0080] ;

[0081] in, A local minimum constant is set to avoid the denominator being 0. for p and q Similarity of color features.

[0082] Step 503: Calculation p With pixel set Ω Similarity of texture features within pixels:

[0083] ;

[0084] in, represent p and q Similarity of texture features.

[0085] Step 504: Estimationp When dealing with reflection intensity data, the set of pixels Ω Weight of each inner pixel:

[0086] ;

[0087] Step 505: Calculate based on weights p Estimated value of reflection intensity:

[0088] ;

[0089] in, for q The intensity of reflection, The normalization coefficient is... For the calculation p Reflection intensity. Figure 7 The results are from the estimation of the reflection intensity data of the road surface area.

[0090] Step 6: Fuse the estimated reflection intensity data with the image data to obtain fused data.

[0091] The specific steps are as follows:

[0092] Step 601: Represent the image data as three-channel data in the RGB color space and normalize it.

[0093] Step 602: Normalize the estimated reflection intensity data of the road surface area and perform feature stitching with the image data to make it the fourth channel of the image data, thereby realizing data-level fusion of the two types of features and obtaining fused data.

[0094] Step 7: Build a segmentation model that includes a first branch and a second branch, train the segmentation model using historical fusion data, and delete the first branch after training is complete.

[0095] The specific steps are as follows:

[0096] Step 701: As Figure 3 As shown, a backbone network for the segmentation model is designed based on a skip connection structure; where ConvBnRe represents the sequential convolution operation, batch normalization, and activation operation using the ReLU function. The parameter Kernel represents the size of the convolution kernel, Stride represents the stride of the convolution operation, and Padding represents the padding applied to the input data during convolution. ASPP represents a holed spatial pyramid pooling layer, and S2D is a downsampling method that converts spatial dimensions into channel layers.

[0097] The data input to the backbone network is scaled to a resolution of 512×512, processed by two ConvBnRe calculations, and then fed into Block1. Blocks 1 through 3 are three structurally identical modules, with the output of one module serving as the input of the next. After data is input into each module, it is first copied. One copy undergoes two ConvBnRe calculations with a stride of 1 and one MaxPooling calculation, while the other copy undergoes one ConvBnRe calculation with a stride of 2. The results of the two copies are then added to obtain the output of that module. Additionally, the output of Block1 is used as a low-level feature map, which, after undergoing one ConvBnRe calculation and an S2D operation, is fused with the high-level feature map output by ASPP.

[0098] The output of Block3 is the input of Block4. Blocks 4 through 11 are eight modules with identical structures; the output of one module is the input of the next. After data is input into each module, it is copied. One copy is then processed by three ConvBnRe calculations with a step size of 1, and added to the other copy to obtain the output of that module.

[0099] The output of Block11 is the input of the ASPP structure. The ASPP structure outputs a high-level feature map with a resolution of 32×32. The low-level feature map output by Block1 is processed through a ConvBnRe calculation and an S2D operation, and its resolution will then match that of the high-level feature map. Figure 1 At this point, high- and low-level feature maps are fused by feature channel splicing, and the fused feature map is the output of the backbone network.

[0100] Step 702: As Figure 3 As shown, a segmentation model is designed, comprising a first branch and a second branch. Where PredictionHead 1 and loss 1 forms the first branch, PredictionHead 2 and loss 2 constitutes the second branch. Segmentation represents the segmentation module, consisting of convolution operations with a kernel size of 1×1, a stride of 1, and no padding. D2S represents an upsampling method that converts the number of channels into spatial dimensions. BilinearInterpolation is a bilinear interpolation upsampling module with a scaling factor of 4. loss 1 and loss Both are cross-entropy loss functions.

[0101] The first branch takes the output of the backbone network as input, and after passing through a ConvBnRe operation and a Segmentation module, it is then... loss1. Calculate the loss. The second branch also takes the output of the backbone network as input, and sequentially passes through two ConvBnRe operations, one D2S module, one ConvBnRe operation, one Segmentation module, and one BilinearInterpolation module before being processed by... loss 2. Calculate the loss. loss 1 and loss The sum of the two values ​​is taken as the average of the losses for the entire network.

[0102] Step 703: Initialize the segmentation model parameters using the Kaiming initialization method. Set the initial learning rate of the segmentation model to 0.1 and the number of training iterations to 150. Then, train the segmentation model using historical fused data. During training, the first branch will enhance the training of the backbone network parameters. The second branch will train the overall parameters of the segmentation model. After training is complete, delete the first branch and keep only the second branch.

[0103] Step 8: Input the real-time acquired fusion data into the trained segmentation model for calculation, segment the slippery road surface area and output the data.

[0104] The specific steps are as follows:

[0105] Step 801: Preprocess the real-time acquired fused data to clean up incomplete data;

[0106] Step 802: Scale the fused data to a resolution of 512×512 and input it into the segmentation model. The output of the BilinearInterpolation module of the second branch of the segmentation model is the segmentation result of the wet and slippery road surface area.

[0107] In the description of this invention, it should be understood that the terms "longitudinal", "lateral", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this invention, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0108] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for segmenting a wet and slippery road surface area based on vehicle sensor data fusion, characterized in that, The method comprises the following steps: S1, calibrating a laser radar and a camera loaded on a vehicle and obtaining calibration parameters; S2, during driving of the vehicle, extracting laser radar data in real time and screening road surface area point clouds therefrom; extracting camera image data in real time and calculating color features and texture features of the image; S3, projecting the road surface area point clouds to the image according to the calibration parameters, so that pixel points covered by the point clouds have reflection intensity data; S4, estimating reflection intensity data of pixel points of the road surface area not covered by the point clouds according to the color features and texture features of the image; S5, fusing the reflection intensity data of the road surface area after completion of estimation with image data to obtain fused data; S6, establishing a segmentation model comprising a first branch and a second branch, training the segmentation model using historical fused data, and deleting the first branch after completion of training to obtain a trained segmentation model; S7, inputting the fused data obtained in S5 into the trained segmentation model to segment and output data of a road surface wet and slippery area; The S4 specifically comprises: S401, calculating similarity of color features and texture features between pixel points of the road surface area not covered by the point clouds and pixel points around the pixel points covered by the point clouds; S402, calculating a weight of the pixel points around the pixel points covered by the point clouds in reflection intensity estimation according to the similarity of color features and the similarity of texture features; S403, calculating reflection intensity data of the pixel points not covered by the point clouds according to the weight of the pixel points around the pixel points covered by the point clouds in reflection intensity estimation; The similarity of color features in S401 is calculated by the following formula: ； wherein, p is a pixel point of the road surface region not covered by the point cloud, Ω is p a set of pixel points around which are covered by the point cloud, q is p a pixel point around which is covered by the point cloud, and , is p a similarity with q a color feature, is a minimum value constant, is p a color feature of is q a color feature of The similarity of texture features in S401 is calculated by the following formula: ； wherein, is p with q a similarity of the texture feature, is p a texture feature, is q a texture feature.

2. The method for road wet area segmentation based on vehicle sensor data fusion according to claim 1, characterized in that, The weight of the pixel points around the pixel points covered by the point clouds in reflection intensity estimation in S402 is calculated by the following formula: ； wherein is q In estimating p the weight of the reflection intensity data.

3. The method of claim 2, wherein the road surface wet area segmentation based on vehicle sensor data fusion is characterized by, The reflection intensity data of the pixel points not covered by the point clouds in S403 is calculated by the following formula: ； wherein is q the reflection intensity, is a normalization coefficient, is the calculated p reflection intensity.

4. The method for road wet area segmentation based on vehicle sensor data fusion according to claim 1, characterized in that, The S5 specifically comprises: S501, representing the image data as three-channel data of an RGB color space and normalizing the image data; S502, normalizing the reflection intensity data of the road surface area after completion of estimation and splicing the reflection intensity data with the image data to become a fourth channel of the image data, to obtain fused data.

5. The method for road wet area segmentation based on vehicle-mounted sensor data fusion according to claim 4, characterized in that, The S6 specifically comprises: S601, designing a backbone network of the segmentation model based on a skip connection structure, and designing a first branch and a second branch for prediction; S602, connecting the backbone network to the first branch and the second branch respectively to form the segmentation model; S603, inputting historical fused data that has been collected into the segmentation model to train the segmentation model at a learning rate of 0.1 and an iteration number of 150; S604, when the training is completed, deleting parameters of the first branch and retaining parameters of the second branch for segmentation of the road surface wet and slippery area.

6. The method of claim 5, wherein the road surface wet area segmentation based on vehicle sensor data fusion is characterized by, The first branch comprises a first prediction head and a first loss function, and is used for enhancing training of parameters of the backbone network; the second branch comprises a second prediction head and a second loss function, and is used for training parameters of the segmentation model.

7. The method for road wet area segmentation based on vehicle-mounted sensor data fusion according to claim 1, characterized in that, The fused data obtained in S7 is real-time fused data. 8.The method of claim 1, wherein, The S1 specifically comprises: installing a laser radar above a vehicle and installing a camera above an inner front windshield of the vehicle; calibrating the camera using a checkerboard calibration method to correct distortion and jointly calibrating the camera and the laser radar using an Autoware calibration toolbox.

Images