A long-distance target detection method based on mask projection

By using mask projection technology in the autonomous driving perception system, visual images are combined with laser point cloud data, the problem of poor detection effect of long-distance targets is solved, accurate identification and detection of long-distance targets is achieved, and detection accuracy is improved. and system flexibility.

CN114943945BActive Publication Date: 2024-11-08CHONGQING CHANGAN TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202210582304.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-26
Publication Date
2024-11-08
Estimated Expiration
2042-05-26

AI Technical Summary

Technical Problem

In the previous technology, in the long-distance object detection, the sparse laser point clouds lead to a weakening of the detection effect, and it is impossible to effectively utilize the visual recognition advantages.

Method used

The mask-based projection method is adopted to combine visual images with laser point cloud data, and the distance attributes of the target are determined through target instance segmentation and projection from 2D to 3D space, and the performance of long-distance target detection is targeted.

Benefits of technology

It realizes accurate judgment and detection of long-distance targets, combines the advantages of visual and laser point clouds, reduces noise interference, improves detection accuracy, and has decoupling, expansion and compatibility.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114943945B_ABST
    Figure CN114943945B_ABST
Patent Text Reader

Abstract

The present invention discloses a long-distance target detection method based on mask projection, which combines the advantages of vision in target type recognition and the advantages of laser point cloud in position estimation. The mask of 2D instance segmentation is projected from visual space to point cloud space to realize the judgment and special processing of long-distance targets, and finally obtain accurate detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving perception technology, and in particular to a long-distance target detection method based on mask projection. Background Art

[0002] Object detection is an important part of the field of autonomous driving perception. Mainstream object detection schemes are usually based on forward vision, surround vision, vision and radar fusion, laser radar point cloud clustering, etc. Vision-based object detection is more accurate in identifying the target type, but its depth and position estimation are poor. Lidar point cloud-based object detection is accurate in position estimation, but its recognition of target type is poor. Therefore, the object detection method based on vision and laser radar fusion can take advantage of each other and achieve better results. For example, patent CN107167811B implements a road drivable area detection method based on monocular vision and laser radar fusion, showing the advantages of the fusion method. However, for long-distance target detection, since the number of laser point clouds becomes sparse with the increase of distance, its detection effect will weaken with the increase of target distance, or even cannot be detected. In response to this problem, patent CN110221603B proposes a long-distance obstacle detection method based on laser radar multi-frame point cloud fusion, but it mainly uses the information of laser point clouds and does not effectively use the advantages of vision. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, an object of the present invention is to provide a long-distance target detection method based on mask projection, so as to solve the problem of poor long-distance target recognition effect in the prior art.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0005] A long-distance target detection method based on mask projection comprises the following steps:

[0006] Step 1: Obtain visual images through vehicle-mounted camera equipment, obtain laser point cloud data using laser radar, and process the visual image data and laser point cloud data;

[0007] Step 2: Perform target instance segmentation on the visual image obtained in step 1 to obtain a segmentation result of the target, and the segmentation result includes a semantic segmentation mask;

[0008] Step 3: According to the semantically segmented mask in step 2, project the mask from 2D to 3D space to obtain the 3D point cloud corresponding to the target;

[0009] Step 4: Based on the 3D point cloud obtained in step 3, determine its position to determine whether the target is a long-distance target; if it is a long-distance target, proceed to step 5; if it is a non-long-distance target, proceed to step 6;

[0010] Step 5: Detect the long-distance target through the detection network and output the position parameters of the long-distance target;

[0011] Step 6: Use the target detection model to detect non-distant targets.

[0012] In order to simultaneously utilize the advantages of vision and laser point cloud, and to improve the accuracy of long-distance target detection while ensuring the performance of close-range target detection results, the present invention uses a mask projection method to first determine the distance attributes of the target, and then specifically improve the performance of long-distance target detection.

[0013] Compared with the prior art, the present invention has the following beneficial effects:

[0014] 1. The method described in the present invention combines visual image recognition and laser point cloud technology. By projecting from 2D vision to 3D laser point cloud space based on target instance segmentation, it effectively combines the advantages of visual recognition of target type and laser point cloud detection of target position, realizes the judgment and special processing of long-distance targets, and finally obtains accurate detection.

[0015] 2. The method of the present invention uses the acquired mask for projection, which can effectively remove the background point cloud after the target is projected into the 3D space, reduce noise interference, and ensure the accuracy of long-distance target recognition.

[0016] 3. The method described in the present invention has strong decoupling, strong scalability and strong compatibility. The instance segmentation network, the target position preliminary estimation algorithm and the long-distance (or non-long-distance) target detection network can all be replaced to achieve flexible configuration. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The figure is a schematic diagram of specific implementation steps of the method of the present invention.

[0018] Figure 2 This is a schematic diagram of the long-distance pedestrian detection results based on this method.

[0019] Figure 3 Schematic diagram of long-distance vehicle detection results based on this method. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present invention. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention.

[0021] The present invention provides a long-distance target detection method based on mask projection, such as Figure 1 As shown, the following steps are included:

[0022] Step 1: Obtain visual images through the vehicle-mounted camera device, use the laser radar to obtain laser point cloud data, synchronize the visual image and laser point cloud data, and convert the coordinates of the vehicle-mounted camera device and the laser radar. The conversion matrix between the vehicle-mounted camera device and the laser radar is obtained through calibration and is used to determine the geometric projection relationship from the 2D visual space to the 3D point cloud space. In actual use, such as Figure 1 As shown in the figure, the original input data comes from two channels S1 and S2, where S1 is an RGB visual image obtained by the front-view camera mounted on the autonomous vehicle, and S2 is laser point cloud data obtained by the laser radar mounted on the autonomous vehicle. The data of S1 and S2 need to be time-synchronized, and the conversion matrix T between the sensors of S1 and S2 needs to be obtained through calibration to determine the geometric projection relationship from 2D visual space to 3D point cloud space.

[0023] Step 2: Perform target instance segmentation on the visual image obtained in step 1 to obtain the segmentation result of the target, and the segmentation result includes the semantic segmentation mask. Among them, instance segmentation can use any network model to perform target instance segmentation on the visual image. For example, Figure 1 Step M1 is to perform target instance segmentation on the RGB image I, using the network model Mask R-CNN, namely:

[0024] {R_i}=f_segmentation(I),

[0025] Among them, R_i=(c_i, b_i, M_i, s_i) is the result of an object instance segmentation, c_i is the object type, b_i is the 2D detection box, M_i is the mask of semantic segmentation, s_i is the confidence, and i is the index of the object.

[0026] Step 3: Based on the semantic segmentation mask in step 2, based on the conversion results of the vehicle-mounted camera and laser radar coordinates, the mask is projected from 2D to 3D space to obtain the 3D point cloud corresponding to the target, that is, the semantic segmentation mask is projected into the point cloud space to obtain the point cloud corresponding to the target in the original point cloud. Figure 1In step M2, the semantic segmentation mask Mi is used. Based on the conversion relationship T calibrated from the visual space to the lidar point cloud space, the mask Mi obtained in the vision can be projected into the point cloud space to obtain the partial point cloud P'_i corresponding to the target i in the original point cloud P, that is:

[0027] P'_i=f_projection(M_i, T, P).

[0028] Step 4: Based on the 3D point cloud of the target, determine its position and whether the target is a long-distance target; if it is a long-distance target, proceed to step 5; if it is a non-long-distance target, proceed to step 6. According to the 3D point cloud obtained in step 3, the center of mass of the target is calculated, and the center of mass of the target can be calculated by clustering method to obtain the depth of the target. Based on the depth of the target, compare it with the threshold of the depth of the long-distance target to determine whether it is a long-distance target. Figure 1 In step M3, the centroid is estimated based on P'_i obtained in the previous step. The centroid estimation method can adopt the histogram method or other similar clustering methods. This process can be expressed as:

[0029] (x_i, y_i, z_i)=f_clustering(P'_i),

[0030] Where x_i, y_i are the horizontal and vertical positions of target i, and z_i is its depth.

[0031] Step M4 determines whether the target i is a long-distance target based on its depth z_i. Set an adjustable long-distance target depth threshold z_th, whose specific calibration value can be set according to the sensor performance and system operating conditions, or obtained by calibration according to the test results in the actual vehicle test. Then perform long-distance target judgment. If z_i>z_th, it is determined to be a long-distance target, otherwise it goes to step M5, otherwise it goes to step M6.

[0032] Step 5: Detect the long-distance target through the detection network and output the position parameters of the long-distance target. Based on the long-distance target judgment logic in step 4, extract the training samples of the long-distance target, and use the training samples to train the network model for long-distance target detection. Figure 1 In step M5, the position of the long-distance target is accurately estimated. First, based on the long-distance target judgment logic in the previous step, the training samples dedicated to the long-distance target are extracted. These samples are used to train a dedicated network model for long-distance target detection. The process can be expressed as:

[0033] B_i=f_faraway(P'_i),

[0034] Among them, f_faraway(.) represents a dedicated model for long-distance target detection, and B_i=(x_i, y_i, z_i, a_i, w_i, l_i, h_i) is the detection result in 3D space. x_i, y_i, z_i are the horizontal, vertical, and depth positions, that is, the target center of mass position, a_i is the rotation angle, and w_i, l_i, h_i are the length, width, and height of the target.

[0035] Step 6: Use the target detection model to detect non-distant targets. Figure 1 In step M6, the position of non-distant targets is accurately estimated. This step can use a general target detection model to achieve accurate target detection:

[0036] B_i=f_sota(P'_i).

[0037] The present invention is applicable to different types of target detection in autonomous driving scenarios. Figure 2 With attached Figure 3 The effect diagrams of long-distance pedestrian detection and long-distance vehicle detection are provided respectively. Figure 2 c and 2d show the visual effects of long-distance pedestrian detection. Figure 2 a is a long-distance pedestrian target in the point cloud space. Due to the long distance, the number of point clouds belonging to this pedestrian target is small. After using the method of the present invention, the accuracy of target position detection is improved. Figure 2 b. Similarly, in the attached Figure 3 In, attached Figure 3 c is the visual effect of the corresponding vehicle target, Figure 3 a is a long-distance vehicle target in the point cloud space. Also due to the long distance, the number of point clouds belonging to this vehicle target is small. After using the method of the present invention, the accuracy of target position detection is improved. Figure 3 b.

[0038] As described above, the reminder system of the present invention is not limited to the configuration described above, and other systems that can implement the embodiments of the present invention may fall within the scope of protection of the present invention.

[0039] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the technical solution. Those skilled in the art should understand that those modifications or equivalent substitutions of the technical solution of the present invention that do not depart from the purpose and scope of the technical solution should be included in the scope of the claims of the present invention.

Claims

1. A long-distance target detection method based on mask projection, characterized in that: The steps include: Step 1: Obtain visual images through vehicle-mounted camera equipment, obtain laser point cloud data using laser radar, and process the visual image data and laser point cloud data; Step 2: Perform target instance segmentation on the visual image obtained in step 1 to obtain the segmentation result of the target, and the segmentation result includes a semantic segmentation mask; Step 3: According to the semantically segmented mask in step 2, project the mask from 2D to 3D space to obtain the 3D point cloud corresponding to the target; Step 4: Based on the 3D point cloud obtained in step 3, determine its position to determine whether the target is a long-distance target; if it is a long-distance target, proceed to step 5; if it is a non-long-distance target, proceed to step 6; Step 5: Detect the long-distance target through the detection network and output the position parameters of the long-distance target; Step 6: Use the target detection model to detect non-distant targets.

2. The long-distance target detection method based on mask projection according to claim 1, characterized in that: In step 1, the visual image and laser point cloud data are time-synchronized and the coordinates of the vehicle-mounted camera and lidar are transformed.

3. The long-distance target detection method based on mask projection according to claim 2, characterized in that: In step 1, the conversion matrix between the vehicle-mounted camera device and the lidar is obtained through calibration to determine the geometric projection relationship from the 2D visual space to the 3D point cloud space.

4. The long-distance target detection method based on mask projection according to claim 1, characterized in that: In step 2, a network model is used to perform object instance segmentation on the visual image.

5. The long-distance target detection method based on mask projection according to claim 1, characterized in that: In step 3, based on the conversion results of the vehicle-mounted camera device and the laser radar coordinates, the mask is projected from 2D to 3D space.

6. The long-distance target detection method based on mask projection according to claim 1, characterized in that: In step 3, the semantic segmentation mask is projected into the point cloud space, and the point cloud corresponding to the target is obtained in the original point cloud.

7. The long-distance target detection method based on mask projection according to claim 1, characterized in that: In step 4, the center of mass of the target is calculated based on the 3D point cloud obtained in step 3.

8. The long-distance target detection method based on mask projection according to claim 7, characterized in that: The centroid of the target is calculated by clustering method to obtain the depth of the target.

9. The long-distance target detection method based on mask projection according to claim 8, characterized in that: In step 4, based on the depth of the target, it is compared with the threshold of the depth of the distant target to determine whether it is a distant target.

10. The long-distance target detection method based on mask projection according to claim 9, characterized in that: The threshold is set according to the sensor performance and the operating conditions of the vehicle system, or is obtained through actual vehicle experiments.

11. The long-distance target detection method based on mask projection according to claim 1, characterized in that: In step 5, based on the long-distance target determination logic in step 4, training samples of long-distance targets are extracted, and the network model is trained using the training samples for long-distance target detection.

Citation Information

Patent Citations

  • A Road Driving Area Detection Method Based on Monocular Vision and LiDAR Fusion

    CN107167811B

  • A method for long-range obstacle detection based on multi-frame point cloud fusion from lidar

    CN110221603B

  • Plane detection method, computing device and circuit system

    CN110458805A

  • Laser radar target detection method and system fusing semantic information

    CN113128348A

  • Three-dimensional target detection method

    CN113256574A