Passable area detection method based on multi-sensor feature level fusion

Abstract

The invention discloses a passable area detection method based on multi-sensor feature level fusion, and the method comprises the following steps: collecting radar point cloud data and RGB image data of a plurality of driving areas through employing a laser radar and a vehicle-mounted camera, and constructing an original data set; constructing a depth map based on the collected radar point cloud data; constructing a training data set based on the RGB image and the depth map; constructing a passable area detection model, and training the model by using the training data set; and 4, executing the step 1 and the step 2 for a to-be-detected driving area, then detecting the to-be-detected driving area by using the passable area detection model trained in the step 4, and identifying the passable area in the to-be-detected driving area. According to the passable area detection method, the detection performance of the model can be improved, the capturing capability of the model on data details is enhanced, and then the detection precision of the passable area is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and in particular to a drivable area detection method based on multi-sensor feature-level fusion. Background Art

[0002] With the rapid development of intelligent transportation and autonomous driving systems, traversable area detection has become one of the key technologies for intelligent vehicle driving. The main goal of traversable area detection is to identify traversable areas in a scene and assist the navigation system to accurately determine the route to ensure smooth driving of the vehicle in complex scenes. Accurately identifying traversable areas has become an indispensable requirement at present.

[0003] Nowadays, camera image data combined with deep learning technology is widely used in the task of passable area detection. As one of the important architectures of deep learning, convolutional neural network performs well in image feature extraction and can extract rich visual information, gradually improving the accuracy and robustness of algorithm detection. However, the passable area detection algorithm based on single camera image data still has poor ability to extract detailed features and analyze global information in complex road scenes.

[0004] Therefore, it is necessary to provide a traversable area detection method based on multi-sensor feature-level fusion to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide a traversable area detection method based on multi-sensor feature-level fusion, so as to solve the problems of poor detail feature extraction and global information analysis capabilities in complex road scenes in the above-mentioned traversable area detection method based on single camera image data.

[0006] The technical solution to achieve the purpose of the present invention is: a traversable area detection method based on multi-sensor feature-level fusion, the method comprising the following steps:

[0007] Step 1: Use laser radar and vehicle-mounted camera to collect radar point cloud data and RGB image data of multiple driving areas respectively to construct the original data set;

[0008] Step 2, constructing a depth map based on the radar point cloud data collected in step 1;

[0009] Step 3, construct a training dataset based on RGB images and depth maps;

[0010] Step 4: construct a traversable area detection model and train the model using the training data set;

[0011] Step 5, for the driving area to be detected, execute steps 1 and 2, and then use the passable area detection model trained in step 4 to detect the driving area to be detected and identify the passable area therein.

[0012] Furthermore, step 2 specifically includes:

[0013] Step 2-1, read the vehicle camera calibration parameter file, extract the camera to image projection matrix P2, the camera rotation matrix R0 rect and the transformation matrix T from the laser radar to the vehicle camera velo_to_cam ;

[0014] Step 2-2, load the lidar point cloud data, extract the three-dimensional coordinates (x, y, z), and remove the points behind the onboard camera in the point cloud;

[0015] Step 2-3, calculate the projection matrix from point cloud to image:

[0016] P=P2·R0 rect ·T velo_to_cam ;

[0017] Step 2-4, project the laser radar point cloud to the image plane through the projection matrix P:

[0018] px=P·(x,y,z) T ;

[0019] Where px is the matrix containing the image plane information;

[0020] Step 2-5, calculate the image pixel coordinates and remove the points that are beyond the image range after projection, that is, project the point cloud onto the image plane:

[0021] z=px[:,3];

[0022] In the formula, x pixel is the horizontal coordinate of the image pixel, y pixel is the ordinate of the image pixel, z is the depth value corresponding to the image pixel, px[:,1], px[:,2] and px[:,3] represent the data in the first, second and third columns of the matrix px respectively;

[0023] Step 2-6: Initialize a sparse depth map that matches the target image size, and then project the point cloud to the pixel coordinates (x pixel ,y pixel ) is mapped into the sparse depth map, that is:

[0024] D i,j =z

[0025] Where Di,j is the pixel value of the sparse depth map; the target image is an RGB image;

[0026] Step 2-7, interpolating the sparse depth map to fill in undefined depth values;

[0027] Step 2-8, calculate the dense depth value, that is, generate a dense depth map, and save the dense depth map as a grayscale depth map I.

[0028] Furthermore, in step 2-7, a weighted difference method is used for difference processing, and the weights are:

[0029]

[0030] Where S, Y are weight coefficients, (x', y') is the current pixel coordinate of the sparse depth map, (x' i ,y' j ) are the coordinates of the coefficient depth map pixel in the neighborhood (i, j).

[0031] Furthermore, in step 2-8, dense depth value calculation is performed, that is, a dense depth map is generated, and the calculation formula is:

[0032]

[0033] Where D dense (x,y) is the coordinate of the dense depth map at (x,y).

[0034] Furthermore, step 3 specifically includes:

[0035] Step 3-1, naming and saving the RGB image data collected in step 1 according to timestamp or number;

[0036] Step 3-2, name and save the depth map constructed in step 2 according to the timestamp or number;

[0037] Step 3-3, segment and annotate the RGB image in step 3-1, mark the passable area and obstacles, and then name and save the labeled image by timestamp or number;

[0038] Step 3-4, check the consistency of image data, including image format and resolution. If there is inconsistency, normalize it to obtain a training data set; the training data set includes input data and training labels, the input data includes RGB images and depth maps, and the training labels are label images after the RGB images are segmented and annotated in step 3-3.

[0039] Furthermore, the traversable area detection model in step 4 includes the following set in sequence:

[0040] Data processing module, used for preliminary feature extraction and standardization of input depth map and RGB image;

[0041] Multi-scale feature extraction module, used to extract multi-scale features from the features fused by the data processing module to capture global and local information;

[0042] An upsampling decoder module is used to gradually restore low-resolution features to the resolution of the input image;

[0043] The prediction output module is used to generate the final semantic prediction map and complete the detection task of the drivable area.

[0044] Furthermore, the data processing module includes two parallel convolution layers with a convolution kernel size of 7×7, two parallel ReLU activation function layers, and a feature fusion layer that adds corresponding elements of a matrix.

[0045] Furthermore, the multi-scale feature extraction module includes four parallel convolutional layers connected sequentially, a Softmax attention mechanism layer and a feature splicing layer; the four convolutional layers are respectively a convolutional layer with a convolution kernel size of 7×7, a convolutional layer with a convolution kernel size of 5×5, a convolutional layer with a convolution kernel size of 3×3 and a convolutional layer with a convolution kernel size of 9×9.

[0046] Furthermore, the upsampling decoder module includes a bicubic interpolation layer, a feature fusion layer for combining high-resolution features, and a convolution layer with a convolution kernel of 3×3, which are connected in sequence.

[0047] Furthermore, the prediction output module includes a Sigmoid activation function layer for normalizing the value of the feature map to [0, 1], and a semantic prediction layer for generating a probability map of each pixel belonging to the target area.

[0048] Compared with the prior art, the present invention has the following significant advantages:

[0049] (1) By using convolution kernels of different sizes to extract multi-scale features, the model can capture global and local information at the same time, thereby better adapting to detail changes in complex scenes and improving the detection accuracy of traversable areas.

[0050] (2) Due to the combination of multi-scale mechanism and self-attention mechanism, the model can effectively capture long-range dependencies in the image, enhance the connection between different regions, and enable the model to understand the scene structure more comprehensively.

[0051] (3) Through the self-attention feature fusion module, multi-scale features are effectively weighted and fused to highlight important area information, which helps to improve the detection accuracy of the model under conditions such as occlusion or lighting changes.

[0052] (4) The model uses layer-by-layer upsampling during the feature fusion process, so that the output feature map is gradually restored to a higher resolution, thereby improving the fineness of the final prediction map.

[0053] (5) Through the feature-level fusion of radar point cloud data and camera image data, the model has strong robustness to complex scenes (such as rugged terrain, dynamic occlusion, etc.) and severe weather (such as strong light, etc.).

[0054] The present invention is further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 The present invention is a flowchart of a traversable area detection method based on multi-sensor feature-level fusion in one embodiment.

[0056] Figure 2 The model architecture diagram of a traversable area detection method based on multi-sensor feature-level fusion in one embodiment.

[0057] Figure 3 The figure is a performance simulation diagram of a traversable area detection method based on single camera image data in an embodiment.

[0058] Figure 4 The figure is a performance simulation diagram of a traversable area detection method based on multi-sensor feature-level fusion in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] It should be noted that if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0061] In one embodiment, in combination Figure 1 and Figure 2, provides a traversable area detection method based on multi-sensor feature-level fusion, the method comprising the following steps:

[0062] S1, using LiDAR and on-board camera to collect radar point cloud data and RGB image data of multiple driving areas respectively, and construct the original data set;

[0063] S2, constructs a depth map based on the radar point cloud data collected in S1;

[0064] S3, builds a training dataset based on RGB images and depth maps;

[0065] S4, constructing a traversable area detection model and training the model using the training data set

[0066] S5, for the driving area to be detected, execute according to S1 and S2, and then use the passable area detection model trained in S4 to detect the driving area to be detected and identify the passable area therein.

[0067] Further, in one embodiment, S2 specifically includes:

[0068] S2-1, read the vehicle camera calibration parameter file, extract the camera to image projection matrix P2, the camera rotation matrix R0 rect and the transformation matrix T from the laser radar to the vehicle camera velo_to_cam ;

[0069] S2-2, load the lidar point cloud data, extract the three-dimensional coordinates (x, y, z), and remove the points behind the onboard camera in the point cloud;

[0070] S2-3, calculate the projection matrix from point cloud to image:

[0071] P=P2·R0 rect ·T velo_to_cam ;

[0072] S2-4, projecting the laser radar point cloud onto the image plane through the projection matrix P:

[0073] px=P·(x,y,z) T ;

[0074] Where px is the matrix containing the image plane information;

[0075] S2-5, calculate the image pixel coordinates and remove the points that are beyond the image range after projection, that is, project the point cloud onto the image plane:

[0076] z=px[:,3];

[0077] In the formula, xpixel is the horizontal coordinate of the image pixel, y pixel is the ordinate of the image pixel, z is the depth value corresponding to the image pixel, px[:,1], px[:,2] and px[:,3] represent the data in the first, second and third columns of the matrix px respectively;

[0078] S2-6, initialize a sparse depth map that matches the target image size, and then project the point cloud to the pixel coordinates (x pixel ,y pixel ) is mapped into the sparse depth map, that is:

[0079] D i,j =z

[0080] Where D i,j is the pixel value of the sparse depth map; the target image is an RGB image;

[0081] S2-7, performing interpolation processing on the sparse depth map to fill undefined depth values;

[0082] S2-8, performing dense depth value calculation, that is, generating a dense depth map, and saving the dense depth map as a grayscale depth map I.

[0083] Further, in one embodiment, a weighted difference method is used to perform difference processing in S2-7, and the weights are:

[0084]

[0085] Where S, Y are weight coefficients, (x', y') is the current pixel coordinate of the sparse depth map, (x' i ,y' j ) are the coordinates of the coefficient depth map pixel in the neighborhood (i, j).

[0086] Further, in one of the embodiments, in S2-8, dense depth value calculation is performed, that is, a dense depth map is generated, and the calculation formula is:

[0087]

[0088] Where D dense (x,y) is the coordinate of the dense depth map at (x,y).

[0089] Further, in one embodiment, S3 specifically includes:

[0090] S3-1, naming and saving the RGB image data collected by S1 according to timestamp or number;

[0091] S3-2, naming and saving the depth map constructed in S2 according to timestamp or number;

[0092] S3-3, segment and annotate the RGB image of S3-1, mark the passable area and obstacles, and then name and save the labeled image according to the timestamp or number;

[0093] Step 3-4, check the consistency of the image data, including image format and resolution. If there is any inconsistency, normalize it to obtain a training data set; the training data set includes input data and training labels, the input data includes RGB images and depth maps, and the training labels are label images after the RGB images are segmented and labeled in S3-3.

[0094] Further, in one embodiment, the traversable area detection model in S4 includes:

[0095] Data processing module, used for preliminary feature extraction and standardization of input depth map and RGB image;

[0096] Multi-scale feature extraction module, used to extract multi-scale features from the features fused by the data processing module to capture global and local information;

[0097] An upsampling decoder module is used to gradually restore low-resolution features to the resolution of the input image;

[0098] The prediction output module is used to generate the final semantic prediction map and complete the detection task of the drivable area.

[0099] Furthermore, in one of the embodiments, the data processing module includes two parallel convolution layers with a convolution kernel size of 7×7, two parallel ReLU activation function layers, and a feature fusion layer that adds corresponding elements of a matrix.

[0100] Furthermore, in one of the embodiments, the multi-scale feature extraction module includes four parallel convolutional layers connected sequentially, a Softmax attention mechanism layer and a feature splicing layer; the four convolutional layers are respectively a convolutional layer with a convolution kernel size of 7×7, a convolutional layer with a convolution kernel size of 5×5, a convolutional layer with a convolution kernel size of 3×3 and a convolutional layer with a convolution kernel size of 9×9.

[0101] Furthermore, in one of the embodiments, the upsampling decoder module includes a bicubic interpolation layer, a feature fusion layer for combining high-resolution features, and a convolution layer with a convolution kernel of 3×3, which are connected in sequence.

[0102] Furthermore, in one of the embodiments, the prediction output module includes a Sigmoid activation function layer for normalizing the value of the feature map to [0, 1], and a semantic prediction layer for generating a probability map of each pixel belonging to the target area.

[0103] Furthermore, in one embodiment, the method further comprises:

[0104] S6, based on S5, the range of the passable area is covered with a green image in the vehicle driving scene to indicate a passable road without obstacles.

[0105] In one embodiment, a traversable area detection system based on multi-sensor feature-level fusion is provided, the system comprising:

[0106] The first module is used to collect radar point cloud data and RGB image data of multiple driving areas using laser radar and vehicle-mounted camera to construct the original data set;

[0107] The second module is used to construct a depth map based on the radar point cloud data collected in the first module;

[0108] The third module is used to construct a training dataset based on RGB images and depth maps;

[0109] The fourth module is used to build a traversable area detection model and train the model using the training data set;

[0110] The fifth module is used to execute the first and second modules for the driving area to be detected, and then use the passable area detection model trained by the fourth module to detect the driving area to be detected and identify the passable area therein.

[0111] For the specific definition of the passable area detection system based on multi-sensor feature-level fusion, please refer to the definition of the passable area detection method based on multi-sensor feature-level fusion above, which will not be repeated here. Each module in the above-mentioned passable area detection system based on multi-sensor feature-level fusion can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0112] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following is achieved:

[0113] Step 1: Use laser radar and vehicle-mounted camera to collect radar point cloud data and RGB image data of multiple driving areas respectively to construct the original data set;

[0114] Step 2, constructing a depth map based on the radar point cloud data collected in step 1;

[0115] Step 3, construct a training dataset based on RGB images and depth maps;

[0116] Step 4: construct a traversable area detection model and train the model using the training data set;

[0117] Step 5, for the driving area to be detected, execute steps 1 and 2, and then use the passable area detection model trained in step 4 to detect the driving area to be detected and identify the passable area therein.

[0118] For the specific limitations of each step, please refer to the limitations of the passable area detection method based on multi-sensor feature-level fusion mentioned above, which will not be repeated here.

[0119] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the computer program implements:

[0120] Step 1: Use laser radar and vehicle-mounted camera to collect radar point cloud data and RGB image data of multiple driving areas respectively to construct the original data set;

[0121] Step 2, constructing a depth map based on the radar point cloud data collected in step 1;

[0122] Step 3, construct a training dataset based on RGB images and depth maps;

[0123] Step 4: construct a traversable area detection model and train the model using the training data set;

[0124] Step 5, for the driving area to be detected, execute steps 1 and 2, and then use the passable area detection model trained in step 4 to detect the driving area to be detected and identify the passable area therein.

[0125] For the specific limitations of each step, please refer to the limitations of the passable area detection method based on multi-sensor feature-level fusion mentioned above, which will not be repeated here.

[0126] In order to verify the effectiveness of the present invention, an experimental simulation was conducted, and the experimental data was based on the laser radar point cloud data and the RGB image of the vehicle camera collected based on the actual road. Figure 3 and Figure 4As shown in the figure, both figures train different passable area detection models based on the road data of the same area. By comparison, it can be seen that the passable area detection method based on multi-sensor feature-level fusion of the present invention can adapt well to complex scenes. Under the same training round, the passable area detection model based on multi-sensor feature-level fusion can achieve higher detection accuracy than the passable area detection model based on single camera image data.

[0127] Therefore, the present invention adopts the above-mentioned passable area detection method based on multi-sensor feature-level fusion, which can improve the detection performance of the model and enhance the model's global understanding ability and detail capture ability of complex scenes.

[0128] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A traversable area detection method based on multi-sensor feature-level fusion, characterized in that: The method comprises the following steps: Step 1: Use laser radar and vehicle-mounted camera to collect radar point cloud data and RGB image data of multiple driving areas respectively to construct the original data set; Step 2, constructing a depth map based on the radar point cloud data collected in step 1; Step 3, construct a training dataset based on RGB images and depth maps; Step 4: construct a traversable area detection model and train the model using the training data set; Step 5, for the driving area to be detected, execute steps 1 and 2, and then use the passable area detection model trained in step 4 to detect the driving area to be detected and identify the passable area therein.

2. The method for detecting traversable areas based on multi-sensor feature-level fusion according to claim 1, characterized in that: Step 2 specifically includes: Step 2-1, read the vehicle camera calibration parameter file, extract the camera to image projection matrix P2, the camera rotation matrix R0 rect and the transformation matrix T from the laser radar to the vehicle camera velo_to_cam ; Step 2-2, load the lidar point cloud data, extract the three-dimensional coordinates (x, y, z), and remove the points behind the onboard camera in the point cloud; Step 2-3, calculate the projection matrix from point cloud to image: P=P2·R0 rect ·T velo_to_cam ; Step 2-4, project the laser radar point cloud to the image plane through the projection matrix P: px=P·(x,y,z) T ; Where px is the matrix containing the image plane information; Step 2-5, calculate the image pixel coordinates and remove the points that are beyond the image range after projection, that is, project the point cloud onto the image plane: In the formula, x pixel is the horizontal coordinate of the image pixel, y pixel is the ordinate of the image pixel, z is the depth value corresponding to the image pixel, px[:,1], px[:,2] and px[:,3] represent the data in the first, second and third columns of the matrix px respectively; Step 2-6: Initialize a sparse depth map that matches the target image size, and then project the point cloud to the pixel coordinates (x pixel ,y pixel ) is mapped into the sparse depth map, that is: D i,j =z Where D i,j is the pixel value of the sparse depth map; the target image is an RGB image; Step 2-7, interpolating the sparse depth map to fill in undefined depth values; Step 2-8, calculate the dense depth value, that is, generate a dense depth map, and save the dense depth map as a grayscale depth map I.

3. The method for detecting traversable areas based on multi-sensor feature-level fusion according to claim 2, characterized in that: In steps 2-7, the weighted difference method is used for difference processing, and the weights are: Where S, Y are weight coefficients, (x', y') is the current pixel coordinate of the sparse depth map, (x' i ,y' j ) are the coordinates of the coefficient depth map pixel in the neighborhood (i, j).

4. The method for detecting traversable areas based on multi-sensor feature-level fusion according to claim 3, characterized in that: In step 2-8, dense depth value calculation is performed to generate a dense depth map. The calculation formula is: Where D dense (x,y) is the coordinate of the dense depth map at (x,y).

5. The method for detecting traversable areas based on multi-sensor feature-level fusion according to claim 1, characterized in that: Step 3 specifically includes: Step 3-1, naming and saving the RGB image data collected in step 1 according to timestamp or number; Step 3-2, name and save the depth map constructed in step 2 according to the timestamp or number; Step 3-3, segment and annotate the RGB image in step 3-1, mark the passable area and obstacles, and then name and save the labeled image by timestamp or number; Step 3-4, check the consistency of image data, including image format and resolution. If there is inconsistency, normalize it to obtain a training data set; the training data set includes input data and training labels, the input data includes RGB images and depth maps, and the training labels are label images after the RGB images are segmented and annotated in step 3-3.

6. The method for detecting traversable areas based on multi-sensor feature-level fusion according to claim 1, characterized in that: The traversable area detection model in step 4 includes the following settings: Data processing module, used for preliminary feature extraction and standardization of input depth map and RGB image; Multi-scale feature extraction module, used to extract multi-scale features from the features fused by the data processing module to capture global and local information; An upsampling decoder module is used to gradually restore low-resolution features to the resolution of the input image; The prediction output module is used to generate the final semantic prediction map and complete the detection task of the drivable area.

7. The method for detecting traversable areas based on multi-sensor feature-level fusion according to claim 6, characterized in that: The data processing module includes two parallel convolution layers with a convolution kernel size of 7×7, two parallel ReLU activation function layers, and a feature fusion layer for adding corresponding elements of a matrix.

8. The method for detecting traversable areas based on multi-sensor feature-level fusion according to claim 6, characterized in that: The multi-scale feature extraction module includes four parallel convolutional layers connected in sequence, a Softmax attention mechanism layer and a feature splicing layer; the four convolutional layers are respectively a convolutional layer with a convolution kernel size of 7×7, a convolutional layer with a convolution kernel size of 5×5, a convolutional layer with a convolution kernel size of 3×3 and a convolutional layer with a convolution kernel size of 9×9.

9. The method for detecting traversable areas based on multi-sensor feature-level fusion according to claim 6, characterized in that: The upsampling decoder module includes a bicubic interpolation layer, a feature fusion layer for combining high-resolution features, and a convolution layer with a convolution kernel of 3×3, which are connected in sequence.

10. The method for detecting traversable areas based on multi-sensor feature-level fusion according to claim 6, characterized in that: The prediction output module includes a Sigmoid activation function layer for normalizing the value of the feature map to [0, 1], and a semantic prediction layer for generating a probability map of each pixel belonging to the target area.

Images