A front-view ground obstacle detection method based on multi-modal fusion

Abstract

The application discloses a kind of front view surface obstacle detection methods based on multi-modal fusion, belong to laser point cloud and the feature fusion and target detection field of vision image;After obtaining point cloud data and the data of vision image, first, the annotation of data set is carried out;Then, the ground is removed to point cloud, and the point cloud is projected from three-dimensional transformation to two-dimensional using transformation matrix, the depth information and reflection information of point cloud are retained, and the sparse point cloud depth map and reflection map are obtained;Then, the depth map and reflection map are preprocessed, the target enhancement of point cloud data is completed, and the point cloud image is obtained;Finally, in the manner of deep learning, the point cloud image and vision image are input into the target detection network of double-input adaptive feature fusion, trained and optimized, and the final detection result is output;The application solves the problem that the existing track surface obstacle detection method is difficult to accurately detect obstacles and has poor detection robustness.

Description

Technical Field

[0001] This invention relates to a method for detecting obstacles on the front-view road surface based on multimodal fusion, belonging to the field of multi-level feature fusion of laser point clouds and visual images and target detection. Background Technology

[0002] With my country's economic development and rising living standards, the number of cars in the country has increased significantly. While this increase is a positive sign of economic development, the accompanying safety issues cannot be ignored. Traditional passive safety technologies are no longer sufficient to meet modern traffic safety standards. To meet consumers' increasingly higher expectations for vehicle safety, advanced driver assistance systems (ADAS) have become a crucial development direction for intelligent traffic safety. In recent years, automotive intelligence has been vigorously promoted and researched, resulting in a growing number of driver assistance systems on the market. ADAS, or Advanced Driver Assistance System, uses sensors installed in the vehicle to collect data on the vehicle itself and its surrounding environment. This data helps drivers avoid deliberate driving errors, effectively improving driving safety.

[0003] LiDAR and cameras are two important sensors in ADAS (Advanced Driver Assistance Systems), each with different characteristics and commonly used in obstacle detection tasks. For example, cameras can capture image information of a vehicle in motion, enabling obstacle identification and classification; LiDAR can sense the distance to targets, thus obtaining the obstacle's appearance, size, and motion status. However, single-sensor obstacle detection solutions have many limitations. Camera-acquired image data is two-dimensional, lacking depth information and easily affected by ambient light; LiDAR-acquired point cloud data contains the target's three-dimensional coordinates but lacks color information, and while it can capture obstacle details, it can lead to inaccurate obstacle classification in complex scenarios.

[0004] Therefore, information fusion schemes based on LiDAR and camera sensors are attracting increasing attention from scholars and manufacturers. Sensor information fusion can combine the advantages of each sensor and compensate for the shortcomings of a single sensor. By fusing data from multiple sensors or the results from multiple sensors, it is possible to more accurately describe obstacle information around the vehicle, providing data assurance for subsequent control layers and thus improving driving safety.

[0005] In numerous studies on pavement environment perception, existing algorithms mostly focus on the detection and extraction of people and vehicles, lacking the detection of other pavement obstacles. In summary, environmental perception through the synergy of lidar and visual information has significant application prospects in remote sensing, telemetry, and autonomous driving, and has become a research hotspot both domestically and internationally. Therefore, conducting fundamental theoretical and technical research closely related to this field is urgently needed. Summary of the Invention

[0006] The purpose of this invention is to provide a pavement obstacle detection method based on multimodal fusion, which solves the problems of difficulty in accurately detecting obstacles and poor detection robustness in pavement obstacle detection methods.

[0007] The technical solution adopted in this invention is as follows:

[0008] A forward-looking road surface obstacle detection method based on multimodal fusion includes the following steps:

[0009] Step 1: Obtain k visual images of size m×n and the corresponding point cloud data ;

[0010] Step 2: Based on the visual image Mark obstacles on the pavement;

[0011] Step 3: Process point cloud data Ground removal is performed to obtain point cloud data without ground. ;

[0012] Step 4: Based on the results of Step 3, use the coordinate transformation matrix to transform the point cloud data. A sparse point cloud depth image is obtained by projecting the 3D transformation onto a 2D region that matches the visual image. and sparse point cloud reflection images ;

[0013] Step 5: Based on the results of Step 4, process the depth image. and reflected images Preprocessing is performed to obtain the enhanced depth image of the target. and reflected images and according to ( , , The points are stitched together to obtain a point cloud image. ;

[0014] Step 6: Based on the results of Step 5, utilize a dual-input adaptive feature fusion obstacle detection network to process the point cloud image. and visual images The input is fed into the network for training, and the classification results and target boxes are obtained and output.

[0015] The specific steps for step 2 are as follows:

[0016] Step 2.1: After obtaining point cloud data and image data, based on the specific road conditions, road obstacles are divided into six categories: cars, bicycles, pedestrians, trams, traffic cones, and obstacles with warning strips.

[0017] The specific steps for step 3 are as follows:

[0018] Step 3.1: Utilize the random sample consistency principle to estimate the ground model of the point cloud dataset. Perform 3D fitting on the ground in the point cloud data. The ground model is a plane, and the structure of the plane in 3D space is as follows:

[0019]

[0020] Where x, y, z are the coordinate axes of the three-dimensional coordinate system, and a, b, c are the coordinate axis coefficients and constant term coefficients;

[0021] Step 3.2: The process of plane fitting is to randomly select a subset of points from the total point cloud dataset as interior points, and then fit the plane using the interior points.

[0022] Step 3.3: Use the results obtained in step 3.2 to test other point cloud data. Compare the measured vertical distance with the set threshold. If the condition is met, it is an inner point; otherwise, it is an outer point.

[0023] Step 3.4: Continue the process in Step 3.2, test the new model, and compare it with the training results of the old model. If the new model has more inliers, save the new model; otherwise, continue to save the old model.

[0024] Step 3.5: Repeat steps 3.2, 3.3, and 3.4 until the iteration condition or the number of iterations is met;

[0025] Step 3.6: Remove the model and interior points that meet the conditions from the point cloud data, leaving only the point cloud data with the road surface removed;

[0026] Step 3.7: Repeat steps 3.2, 3.3, 3.4, and 3.5 to obtain point cloud data with road surface and sidewalk removed;

[0027] The specific steps for step 4 are as follows:

[0028] Step 4.1: Extract the calib file from the KITTI dataset for calibration of point cloud data and image data. Based on the data in the calib file, transform the point cloud data from the radar 3D coordinate system to the 2D coordinate system of camera 2 (the left-side color camera). The transformation projection formula is:

[0029]

[0030] Where y represents the two-dimensional coordinates of the point cloud data transformed to camera 2, and x represents the three-dimensional radar coordinates of the point cloud data. These are the transformation parameters from the coordinate system of camera 0 to the pixel coordinate system of camera 2. Camera 0 is the left-side grayscale camera. This is the correction matrix for camera number 0. Here is the extrinsic parameter matrix for camera number 0. Used to transform data from the radar coordinate system to the coordinate system of camera 0;

[0031] Step 4.2: After matching the point cloud data, delete the point cloud data that exceeds the range of the visual image to ensure consistency with the content of the visual image;

[0032] Step 4.3: Based on the processing results of 4.2, retain the depth information of the point cloud data to obtain a sparse point cloud depth image. ;

[0033] Step 4.4: Based on the processing results of 4.2, retain the reflection intensity information of the point cloud data to obtain a sparse point cloud reflection image. .

[0034] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0035] 1. A forward-looking road obstacle detection method based on multimodal fusion. Considering the situation of real road surfaces, the road surface is mostly composed of carriageways and sidewalks, and there is a certain height difference between the sidewalks and carriageways. Using the RANSAC principle to remove the ground once can only remove the carriageway ground, while the sidewalk ground also has a certain impact on target detection. Therefore, the RANSAC principle is used twice to remove the ground, making full use of the characteristics and differences between the ground and the target, reducing the interference of ground point clouds, and improving the robustness of the algorithm.

[0036] 2. Early target detection algorithms only utilized the depth information of point clouds. In this invention, reflection intensity information is introduced in the point cloud preprocessing, which effectively enhances the detection of obstacles with strong reflection intensity, such as traffic cones and obstacles with warning strips. This fully utilizes the characteristics of point clouds and improves the accuracy of obstacle detection.

[0037] 3. A deep learning-based object detection method is adopted, modifying the traditional single-input object detection into a dual-input object detection method that combines point clouds. The point cloud image and the visual image are processed separately through the Backbone, and the multi-layer features of the two are adaptively fused, which effectively enhances the visual robustness of obstacle detection, especially in environments such as night and exposure. Combining the data features of laser point clouds can effectively improve the target detection effect of obstacles. Even under ideal visual conditions, the detection effect is improved to a certain extent, thereby improving the detection accuracy and robustness. Attached Figure Description

[0038] The present invention will be described by way of example and with reference to the accompanying drawings, wherein:

[0039] Figure 1 This is a flowchart of a multimodal fusion method for detecting obstacles on the forward-looking road surface;

[0040] Figure 2-1 , Figure 2-2 This invention employs a dual-input feature adaptive fusion network called BackBone. Figure 2-1 For Backbone; Figure 2-2 For Fusion;

[0041] Figure 3-1 This is the point cloud data used in Embodiment 1 of the present invention;

[0042] Figure 3-2 This is the visual image used in Embodiment 1 of the present invention;

[0043] Figure 4-1 This is a sparse point cloud depth map of Embodiment 1 of the present invention;

[0044] Figure 4-2 This is a sparse point cloud reflection image of Embodiment 1 of the present invention;

[0045] Figure 5 This is the point cloud preprocessing result of Embodiment 1 of the present invention;

[0046] Figure 6 This is the obstacle detection result of the forward-looking path surface in Embodiment 1 of the present invention. Detailed Implementation

[0047] All features disclosed in this specification, or all steps in all disclosed methods or processes, may be combined in any way, except for mutually exclusive features and / or steps.

[0048] The following is combined with Figures 1 to 6 The present invention will be described in detail below.

[0049] Example 1:

[0050] Step 1: Obtain k visual images of size m×n and the corresponding point cloud data ,like Figure 3-1 and 3-2 ;

[0051] Step 2: Based on the visual image Mark the six common types of pavement obstacles;

[0052] Step 2.1: After obtaining point cloud data and image data, and in combination with the specific road conditions, the detected defects (six common road surface obstacles) are divided into six categories: Car, Cyclist, Pedestrian, Tram, Warning (traffic cones), and Warning 1 (obstacles with warning strips).

[0053] Step 3: Process point cloud data Ground removal is performed to obtain point cloud data without ground. ;

[0054] Step 3.1: Utilize the random sample consistency principle to estimate the ground model of the point cloud dataset. Perform 3D fitting on the ground in the point cloud data. The ground model is a plane, and the structure of the plane in 3D space is as follows:

[0055]

[0056] Where x, y, z are the coordinate axes of the three-dimensional coordinate system, and a, b, c are the coordinate axis coefficients and constant term coefficients;

[0057] Step 3.2: The process of plane fitting is to randomly select a subset (with three points) from the total point cloud dataset as interior points, and to fit the plane using the interior points;

[0058] Step 3.3: Take the results obtained in step 3.2 and test other point cloud data. Compare the measured vertical distance with the set threshold. If the condition is met, it is an inner point; otherwise, it is an outer point.

[0059] Step 3.4: Continue the process in Step 3.2, test the new model, and compare it with the training results of the old model. If the new model has more inliers, save the new model; otherwise, continue to save the old model.

[0060] Step 3.5: Repeat steps 3.2, 3.3, and 3.4 until the iteration condition or the number of iterations is met;

[0061] Step 3.6: Remove the model and interior points that meet the conditions from the point cloud data, leaving only the point cloud data with the road surface removed;

[0062] Step 3.7: Repeat steps 3.2, 3.3, 3.4, and 3.5 to obtain point cloud data with road surface and sidewalk removed.

[0063] Step 4: Based on the results of Step 3, use the coordinate transformation matrix to transform the point cloud data. A sparse point cloud depth image is obtained by projecting the 3D transformation onto a 2D region that matches the visual image. and sparse point cloud reflection images ,like Figure 4-1 and 4-2 ;

[0064] Step 4.1: Extract the calib file from the KITTI dataset for calibration of point cloud data and image data. Based on the data in the calib file, transform the point cloud data from the radar 3D coordinate system to the 2D coordinate system of camera 2 (the left-side color camera). The transformation projection formula is:

[0065]

[0066] Where y represents the two-dimensional coordinates of the point cloud data transformed to camera 2, and x represents the three-dimensional radar coordinates of the point cloud data. These are the transformation parameters from the coordinate system of camera 0 to the pixel coordinate system of camera 2. Camera 0 is the left-side grayscale camera. This is the correction matrix for camera number 0. Here is the extrinsic parameter matrix for camera number 0. The data can be transformed from the radar coordinate system to the coordinate system of camera 0;

[0067] Step 4.2: After matching the point cloud data, delete the point cloud data that exceeds the range of the visual image to ensure consistency with the content of the visual image.

[0068] Step 4.3: Based on the processing results of 4.2, retain the depth information of the point cloud data to obtain a sparse point cloud depth image. ;

[0069] Step 4.4: Based on the processing results of 4.2, retain the reflection intensity information of the point cloud data to obtain a sparse point cloud reflection image. .

[0070] Step 5: Based on the results of Step 4, process the depth image. and reflected images Preprocessing is performed to obtain the enhanced depth image of the target. and reflected images and according to ( , , The points are stitched together to obtain a point cloud image. ,like Figure 5 ;

[0071] Step 5.1: Process the sparse point cloud depth image A linear extension is performed, and the extended formula is:

[0072]

[0073] Where y is the expanded depth value, and x is the original depth value. This represents the maximum depth value in the current point cloud data.

[0074] Step 5.2: Interpolate the results obtained in Step 5.1 using the window inverse distance interpolation method to obtain a denser depth map. ;

[0075] Step 5.2.1: Find the points to be interpolated based on the linearly expanded sparse depth image of the point cloud;

[0076] Step 5.2.2: Establish a 21×21 ROI near the point to be interpolated;

[0077] Step 5.2.3: Calculate the depth value of the point to be interpolated in the ROI region using inverse distance interpolation.

[0078] Step 5.2.4: Repeat steps 5.2.1, 5.2.2, and 5.2.3 until all interpolation points have been traversed;

[0079] Step 5.3: Reflectance image of sparse point cloud Thresholding is performed to retain the reflection image containing the target. The thresholding formula is as follows:

[0080]

[0081] in, The reflection value of each pixel before processing. The reflection value after processing for each pixel;

[0082] Step 5.4: Perform morphological transformation on the thresholded reflection image to obtain the target-enhanced reflection image. ;

[0083] Step 5.4.1: Construct the classic 3×3 cross-shaped structural element B;

[0084] Step 5.4.2: Using four expansion operations and one etching operation, the morphological processing formula is obtained:

[0085]

[0086] in The reflection image before processing. The result is the processed reflection image, and B is the structural element operator constructed in step 5.4.1;

[0087] Step 6: Based on the results of Step 5, utilize a dual-input adaptive feature fusion obstacle detection network to process the point cloud image. and visual images The input is fed into a dual-input feature adaptive fusion target detection neural network for training, yielding classification results and target boxes, and the results are output, such as... Figure 6 .

[0088] Step 6.1: View the entire visual image and point cloud images The input is fed into a CNN network for feature extraction to obtain a feature map; Figure 2-1 , Figure 2-2 As shown, this application employs a dual-input feature adaptive fusion network called BackBone;

[0089] Step 6.1.1: View the entire visual image and point cloud images Scale to a fixed size;

[0090] Step 6.1.2: Input fixed-size visual images and point cloud images into two feature extraction networks to obtain feature maps at different levels of the point cloud images and visual images, respectively;

[0091] Step 6.1.3: Perform adaptive feature fusion on feature maps of the same level in the point cloud image and the visual image;

[0092] Step 6.2: Input the feature map, use the RPN structure to generate candidate boxes, and project the candidate boxes generated by the RPN onto the feature map to obtain the corresponding feature matrix;

[0093] Step 6.3: Scale the feature matrices of different sizes to a 7*7 feature map through the ROI pooling layer, and then flatten the feature map and feed it into the fully connected layer;

[0094] Step 6.4: Use two sibling layers: a SoftMax classification layer with K+1 categories and a regression layer with K bounding boxes to complete the classification and regression of the target.

[0095] After acquiring point cloud data and visual images, this invention first labels obstacles based on the visual images. Then, it removes ground data and performs coordinate transformation and projection on the point cloud data to obtain sparse point cloud depth maps and point cloud reflection maps. Next, it preprocesses the sparse point cloud depth maps and point cloud reflection maps to obtain target-enhanced point cloud images. Finally, it inputs the point cloud images and visual images into a dual-input feature adaptive fusion target detection network to output the final pavement obstacle detection results. This invention solves the problems of existing target detection technologies, such as difficulty in accurately detecting pavement obstacles, low detection efficiency, and slow detection speed.

[0096] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be conceived by those skilled in the art within the technical scope disclosed in the present invention without creative effort should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A forward-looking obstacle detection method based on multimodal fusion, characterized in that, Includes the following steps: Step 1: Obtain k visual images of size m×n and the corresponding point cloud data ; Step 2: Based on the visual image Mark obstacles on the pavement; Step 3: Process point cloud data Ground removal is performed to obtain point cloud data without ground. ; Step 4: Based on the results of Step 3, use the coordinate transformation matrix to transform the point cloud data. A sparse point cloud depth image is obtained by projecting the 3D transformation onto a 2D region that matches the visual image. and sparse point cloud reflection images ; Step 5: Based on the results of Step 4, process the depth image. and reflected images Preprocessing is performed to obtain the enhanced depth image of the target. and reflected images and according to ( , , The points are stitched together to obtain a point cloud image. ; Step 6: Based on the results of Step 5, utilize a dual-input adaptive feature fusion obstacle detection network to process the point cloud image. and visual images The input is fed into the network for training, and the classification results and target boxes are obtained and output. The specific steps for step 2 are as follows: Step 2.1: After obtaining point cloud data and image data, based on the specific road conditions, road obstacles are divided into six categories: cars, bicycles, pedestrians, trams, traffic cones, and obstacles with warning strips. The specific steps for step 3 are as follows: Step 3.1: Utilize the random sample consistency principle to estimate the ground model of the point cloud dataset. Perform 3D fitting on the ground in the point cloud data. The ground model is a plane, and the structure of the plane in 3D space is as follows: ， Where x, y, z are the coordinate axes of the three-dimensional coordinate system, and a, b, c are the coordinate axis coefficients and constant term coefficients; Step 3.2: The process of plane fitting is to randomly select a subset of points from the total point cloud dataset as interior points, and then fit the plane using the interior points. Step 3.3: Use the results obtained in Step 3.2 to test other point cloud data. Compare the measured vertical distance with the set threshold. If the condition is met, it is an inner point; otherwise, it is an outer point. Step 3.4: Continue the plane fitting process, test the new model, and compare it with the training results of the old model. If there are more interior points, save the new model; otherwise, continue to save the old model. Step 3.5: Repeat steps 3.2, 3.3, and 3.4 until the iteration condition or the number of iterations is met; Step 3.6: Remove the model and interior points that meet the conditions from the point cloud data, leaving only the point cloud data with the road surface removed; Step 3.7: Using the point cloud data obtained in Step 3.6 after removing the road surface, repeat the process of Step 3.2, Step 3.3, Step 3.4, and Step 3.5 to obtain point cloud data after removing both the road surface and the sidewalk. The specific steps of step 4 are as follows: Step 4.1: Extract the calib file from the KITTI dataset for calibration of point cloud data and image data. Based on the data in the calib file, transform the point cloud data from the radar 3D coordinate system to the 2D coordinate system of camera 2 (the left-side color camera). The transformation projection formula is: ， Where y represents the two-dimensional coordinates of the point cloud data transformed to camera 2, and x represents the three-dimensional radar coordinates of the point cloud data. These are the transformation parameters from the coordinate system of camera 0 to the pixel coordinate system of camera 2. Camera 0 is the left-side grayscale camera. This is the correction matrix for camera number 0. Here is the extrinsic parameter matrix for camera number 0. Used to transform data from the radar coordinate system to the coordinate system of camera 0; Step 4.2: After matching the point cloud data, delete the point cloud data that exceeds the range of the visual image to ensure consistency with the content of the visual image; Step 4.3: Based on the processing results of Step 4.2, retain the depth information of the point cloud data to obtain a sparse point cloud depth image. ; Step 4.4: Based on the processing results of Step 4.2, retain the reflection intensity information of the point cloud data to obtain a sparse point cloud reflection image. .

2. The method for detecting obstacles on the forward-looking road surface based on multimodal fusion according to claim 1, characterized in that: The specific steps for step 5 are as follows: Step 5.1: Process the sparse point cloud depth image A linear extension is performed, and the extended formula is: ， Where y is the expanded depth value, and x is the original depth value. This represents the maximum depth value in the current point cloud data. Step 5.2: Interpolate the results obtained in Step 5.1 using the window inverse distance interpolation method to obtain a denser depth map. ; Step 5.3: Reflectance image of sparse point cloud Thresholding is performed to retain the reflection image containing the target. The thresholding formula is as follows: ， in, The reflection value of each pixel before processing. The reflection value after processing for each pixel; Step 5.4: Perform morphological transformation on the thresholded reflection image to obtain the target-enhanced reflection image. .

3. The method for detecting obstacles on the forward-looking road surface based on multimodal fusion according to claim 2, characterized in that: The specific steps of step 5.2 are as follows: Step 5.2.1: Find the points to be interpolated based on the linearly expanded sparse depth image of the point cloud; Step 5.2.2: Establish a 21×21 ROI near the point to be interpolated; Step 5.2.3: Calculate the depth value of the point to be interpolated in the ROI region using inverse distance interpolation. Step 5.2.4: Repeat steps 5.2.1, 5.2.2, and 5.2.3 until all interpolation points have been traversed.

4. A forward-looking path obstacle detection method based on multimodal fusion according to claim 2 or 3, characterized in that: The specific steps of step 5.4 are as follows: Step 5.4.1: Construct the classic 3×3 cross-shaped structural element B; Step 5.4.2: Using four expansion operations and one etching operation, the morphological processing formula is obtained: ， In the morphological operations of this step, This is the reflection image before morphological processing. B is the morphologically processed reflection image, and B is the structuring element operator constructed in step 5.4.

1.

5. The method for detecting obstacles on the forward-looking road surface based on multimodal fusion according to claim 1, characterized in that: The specific steps for step 6 are as follows: Step 6.1: View the entire visual image and point cloud images The input is fed into a CNN network for feature extraction to obtain a feature map; Step 6.2: Input the feature map, use the RPN structure to generate candidate boxes, and project the candidate boxes generated by the RPN onto the feature map to obtain the corresponding feature matrix; Step 6.3: Scale the feature matrices of different sizes to a 7*7 feature map through the ROI pooling layer, and then flatten the feature map and feed it into the fully connected layer; Step 6.4: Use two sibling layers: a SoftMax classification layer with K+1 categories and a regression layer with K bounding boxes to complete the classification and regression of the target.

6. The method for detecting obstacles on the forward-looking road surface based on multimodal fusion according to claim 5, characterized in that: The specific steps of step 6.1 are as follows: Step 6.1.1: View the entire visual image and point cloud images Scale to a fixed size; Step 6.1.2: Input fixed-size visual images and point cloud images into two feature extraction networks to obtain feature maps at different levels of the point cloud images and visual images, respectively; Step 6.1.3: Perform adaptive feature fusion on feature maps of the same level in the point cloud image and the visual image.

Images