4D millimeter wave radar dynamic and static point cloud detection method based on deep learning

Abstract

The invention relates to the technical field of intelligent driving, and particularly discloses a 4D millimeter-wave radar dynamic and static point cloud detection method based on deep learning, and the method comprises the steps: obtaining the multi-dimensional features of a target point cloud of the surrounding environment of a vehicle; processing to obtain a two-dimensional feature aerial view corresponding to a single frame of target point cloud and a two-dimensional feature aerial view corresponding to multiple frames of target point clouds; when the sizes of the two two-dimensional feature aerial view views are consistent, the features in the two-dimensional feature aerial view view corresponding to the single-frame target point cloud are used to associate the features in the two-dimensional feature aerial view views corresponding to the multi-frame target point cloud; after feature association, fusing the two two-dimensional feature aerial view views to generate a fused feature map, and performing convolution operation on the fused feature map; and detecting the fused feature map after the convolution operation, and outputting a feature detection result. Single-frame and multi-frame information is fused based on deep learning, and the detection precision and the detection efficiency of the 4D millimeter wave radar are optimized.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and more specifically, to a 4D millimeter-wave radar dynamic and static point cloud detection method based on deep learning. Background Art

[0002] When using 4D imaging millimeter-wave radar for perception, if it is based only on single-frame information, the sparsity of distant point clouds often leads to missed detections; while using information from several adjacent frames for compensation, the accumulation of multiple frames will cause motion blur when detecting moving targets, making positioning difficult. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a 4D millimeter-wave radar dynamic and static point cloud detection method based on deep learning. Based on deep learning, the information of single frame and multiple frames is integrated to optimize the detection accuracy and detection efficiency of 4D imaging millimeter-wave radar.

[0004] As a first aspect of the present invention, a 4D millimeter wave radar dynamic and static point cloud detection method based on deep learning is provided, and the 4D millimeter wave radar dynamic and static point cloud detection method based on deep learning comprises the following steps: Step S11: acquiring multi-dimensional features of the target point cloud of the vehicle surrounding environment collected by the 4D millimeter wave radar; wherein, processing is performed separately according to the single-frame target point cloud and the multi-frame target point cloud; Step S12: Divide the vehicle surrounding space into multiple element grids of equal volume, perform preliminary clustering on the single-frame target point cloud to obtain the average value of each dimension feature of all target point clouds in each element grid corresponding to the single-frame target point cloud; and perform preliminary clustering on the multi-frame target point clouds to obtain the average value of each dimension feature of all target point clouds in each element grid corresponding to the multi-frame target point clouds; Step S13: performing sparse convolution processing on the average values ​​of all dimensional features of all target point clouds in each element grid corresponding to the single-frame target point cloud to process them into the average values ​​of three dimensional features of all target point clouds in each element grid corresponding to the single-frame target point cloud, and compressing the average values ​​of three dimensional features of all target point clouds in each element grid corresponding to the single-frame target point cloud into a two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud; at the same time, performing sparse convolution processing on the average values ​​of all dimensional features of all target point clouds in each element grid corresponding to the multi-frame target point clouds to process them into the average values ​​of three dimensional features of all target point clouds in each element grid corresponding to the multi-frame target point clouds, and compressing the average values ​​of three dimensional features of all target point clouds in each element grid corresponding to the multi-frame target point clouds into a two-dimensional feature bird's-eye view corresponding to the multi-frame target point clouds; Step S14: when the size of the two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud is consistent with the size of the two-dimensional feature bird's-eye view corresponding to the multi-frame target point cloud, using the features in the two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud as keys and values, the features in the two-dimensional feature bird's-eye view corresponding to the multi-frame target point cloud are associated; Step S15: After the feature association, the two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud and the two-dimensional feature bird's-eye view corresponding to the multi-frame target point cloud are fused to generate a fused feature map spliced ​​based on the time dimension, and a convolution operation is performed on the fused feature map; Step S16: Detect the fused feature map after the convolution operation and output the feature detection result.

[0005] Furthermore, the method of dividing the vehicle surrounding space into a plurality of element grids of equal volume and performing preliminary clustering on the single-frame target point cloud to obtain the average value of each dimensional feature of all target point clouds in each element grid corresponding to the single-frame target point cloud also includes: The single-frame target point cloud is divided into corresponding element grids according to its multi-dimensional features, the target point clouds falling into each element grid are collected, and then the average value of each dimensional feature of all target point clouds in each element grid is calculated.

[0006] Furthermore, the method of dividing the vehicle surrounding space into a plurality of element grids of equal volume and performing preliminary clustering on the multi-frame target point clouds to obtain the average value of each dimensional feature of all target point clouds in each element grid corresponding to the multi-frame target point clouds also includes: The multi-frame target point clouds are divided into corresponding element grids according to their multi-dimensional features, the target point clouds falling into each element grid are collected, and then the average value of each dimensional feature of all target point clouds in each element grid is calculated.

[0007] Furthermore, the step S14 further includes: When the size of the two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud is inconsistent with the size of the two-dimensional feature bird's-eye view corresponding to the multi-frame target point cloud, return to step S13 to continue sparse convolution processing until the sizes of the two two-dimensional feature bird's-eye views are consistent.

[0008] Furthermore, the step S16 further includes: When the feature detection results meet the expected results, this round of processing ends.

[0009] The 4D millimeter wave radar dynamic and static point cloud detection method based on deep learning provided by the present invention has the following advantages: (1) The present invention combines the information of a single frame and multiple frames, which reduces the probability of false detection and missed detection compared with the single-frame method; (2) The present invention combines the information of a single frame and multiple frames. Compared with the multi-frame method, the positioning is more accurate and there will be no ambiguity; (3) The present invention uses the characteristics of a single-frame branch as an index to obtain a key threshold in the multi-frame features, thereby reducing the amount of calculation compared to using multiple frames for judgment and optimizing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention, but do not constitute a limitation of the present invention.

[0011] Figure 1 A flowchart of the 4D millimeter-wave radar dynamic and static point cloud detection method based on deep learning provided by the present invention.

[0012] Figure 2 A flowchart of a specific implementation of the 4D millimeter-wave radar dynamic and static point cloud detection method based on deep learning provided by the present invention. DETAILED DESCRIPTION

[0013] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation method, structure, characteristics and effects of a 4D millimeter-wave radar dynamic and static point cloud detection method based on deep learning proposed by the present invention are described in detail below in combination with the accompanying drawings and preferred embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0014] In this embodiment, a 4D millimeter wave radar dynamic and static point cloud detection method based on deep learning is provided. Figure 1 As shown, the 4D millimeter wave radar dynamic and static point cloud detection method based on deep learning includes the following steps: Step S11: Acquire the multi-dimensional features of the target point cloud of the vehicle surrounding environment collected by the 4D millimeter wave radar; Figure 2 As shown, all target point cloud data are input into two branches: single-frame target point cloud and multi-frame target point cloud, and processed according to single-frame target point cloud and multi-frame target point cloud respectively; It should be noted that the 4D millimeter-wave radar is installed on the front bumper of the vehicle.

[0015] Step S12: Divide the vehicle surrounding space into multiple element grids of equal volume, and perform preliminary clustering on the single-frame target point cloud to obtain the average value of each dimensional feature of all target point clouds in each element grid corresponding to the single-frame target point cloud; at the same time, perform preliminary clustering on the multi-frame target point clouds to obtain the average value of each dimensional feature of all target point clouds in each element grid corresponding to the multi-frame target point clouds; this step helps to reduce the complexity of the data and provide a more concise data representation for subsequent processing.

[0016] Preferably, the method of dividing the vehicle surrounding space into a plurality of element grids of equal volume and performing preliminary clustering on the single-frame target point cloud to obtain the average value of each dimensional feature of all target point clouds in each element grid corresponding to the single-frame target point cloud also includes: The single-frame target point cloud is divided into corresponding element grids according to its multi-dimensional features, the target point clouds falling into each element grid are collected, and then the average value of each dimensional feature of all target point clouds in each element grid is calculated.

[0017] Preferably, the method of dividing the vehicle surrounding space into a plurality of element grids of equal volume and performing preliminary clustering on the multi-frame target point clouds to obtain the average value of each dimensional feature of all target point clouds in each element grid corresponding to the multi-frame target point clouds also includes: The multi-frame target point clouds are divided into corresponding element grids according to their multi-dimensional features, the target point clouds falling into each element grid are collected, and then the average value of each dimensional feature of all target point clouds in each element grid is calculated.

[0018] Step S13: performing sparse convolution processing on the average values ​​of all dimensional features of all target point clouds in each element grid corresponding to the single-frame target point cloud to process them into the average values ​​of three dimensional features of all target point clouds in each element grid corresponding to the single-frame target point cloud, and compressing the average values ​​of three dimensional features of all target point clouds in each element grid corresponding to the single-frame target point cloud into a two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud; at the same time, performing sparse convolution processing on the average values ​​of all dimensional features of all target point clouds in each element grid corresponding to the multi-frame target point clouds to process them into the average values ​​of three dimensional features of all target point clouds in each element grid corresponding to the multi-frame target point clouds, and compressing the average values ​​of three dimensional features of all target point clouds in each element grid corresponding to the multi-frame target point clouds into a two-dimensional feature bird's-eye view corresponding to the multi-frame target point clouds; this step not only reduces the amount of data, but also reduces the complexity of subsequent calculations.

[0019] Step S14: When the size of the two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud is consistent with the size of the two-dimensional feature bird's-eye view corresponding to the multi-frame target point cloud, use the features in the two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud as keys and values ​​to associate the features in the two-dimensional feature bird's-eye view corresponding to the multi-frame target point cloud; since the position of the same object in the two two-dimensional feature bird's-eye views should be similar, the cross-correlation of local information can improve the accuracy and effectiveness of the association.

[0020] Preferably, the step S14 further includes: When the size of the two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud is inconsistent with the size of the two-dimensional feature bird's-eye view corresponding to the multi-frame target point cloud, return to step S13 to continue sparse convolution processing until the sizes of the two two-dimensional feature bird's-eye views are consistent. This step ensures the alignment of the feature maps and lays the foundation for subsequent feature association.

[0021] Step S15: After feature association, the two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud and the two-dimensional feature bird's-eye view corresponding to the multi-frame target point cloud are fused to generate a fused feature map spliced ​​based on the time dimension, and a convolution operation is performed on the fused feature map; this fused feature map contains more spatiotemporal information and improves the detection accuracy.

[0022] Step S16: Detect the fused feature map after the convolution operation and output the feature detection result.

[0023] Preferably, the step S16 further includes: When the feature detection results meet the expected results, this round of processing ends.

[0024] The present invention provides a 4D millimeter-wave radar dynamic and static point cloud detection method based on deep learning. By fusing the rich semantic information of multi-frame point clouds and the precise geometric information of single-frame point clouds, the probability of false detection can be greatly reduced, and the target can be located more accurately. At the same time, the information of a single frame is used as an index to search for the effective threshold in the accumulated information of multiple frames, which can reduce the amount of calculation and optimize efficiency.

[0025] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of the technical solution of the present invention.

Claims

1. A 4D millimeter wave radar dynamic and static point cloud detection method based on deep learning, characterized in that: The 4D millimeter wave radar dynamic and static point cloud detection method based on deep learning includes the following steps: Step S11: acquiring multi-dimensional features of the target point cloud of the vehicle surrounding environment collected by the 4D millimeter wave radar; wherein, processing is performed separately according to the single-frame target point cloud and the multi-frame target point cloud; Step S12: Divide the vehicle surrounding space into multiple element grids of equal volume, perform preliminary clustering on the single-frame target point cloud to obtain the average value of each dimension feature of all target point clouds in each element grid corresponding to the single-frame target point cloud; and perform preliminary clustering on the multi-frame target point clouds to obtain the average value of each dimension feature of all target point clouds in each element grid corresponding to the multi-frame target point clouds; Step S13: performing sparse convolution processing on the average values ​​of all dimensional features of all target point clouds in each element grid corresponding to the single-frame target point cloud to process them into the average values ​​of three dimensional features of all target point clouds in each element grid corresponding to the single-frame target point cloud, and compressing the average values ​​of three dimensional features of all target point clouds in each element grid corresponding to the single-frame target point cloud into a two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud; at the same time, performing sparse convolution processing on the average values ​​of all dimensional features of all target point clouds in each element grid corresponding to the multi-frame target point clouds to process them into the average values ​​of three dimensional features of all target point clouds in each element grid corresponding to the multi-frame target point clouds, and compressing the average values ​​of three dimensional features of all target point clouds in each element grid corresponding to the multi-frame target point clouds into a two-dimensional feature bird's-eye view corresponding to the multi-frame target point clouds; Step S14: when the size of the two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud is consistent with the size of the two-dimensional feature bird's-eye view corresponding to the multi-frame target point cloud, using the features in the two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud as keys and values, the features in the two-dimensional feature bird's-eye view corresponding to the multi-frame target point cloud are associated; Step S15: After the feature association, the two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud and the two-dimensional feature bird's-eye view corresponding to the multi-frame target point cloud are fused to generate a fused feature map spliced ​​based on the time dimension, and a convolution operation is performed on the fused feature map; Step S16: Detect the fused feature map after the convolution operation and output the feature detection result.

2. According to a 4D millimeter wave radar dynamic and static point cloud detection method based on deep learning according to claim 1, it is characterized in that: The process of dividing the vehicle surrounding space into a plurality of element grids of equal volume and performing preliminary clustering on the single-frame target point cloud to obtain the average value of each dimension feature of all target point clouds in each element grid corresponding to the single-frame target point cloud also includes: According to the multi-dimensional features of the single-frame target point cloud, the single-frame target point cloud is divided into corresponding element grids, the target point clouds falling into each element grid are collected, and then the average value of each dimensional feature of all target point clouds in each element grid is calculated.

3. The 4D millimeter wave radar dynamic and static point cloud detection method based on deep learning according to claim 1 is characterized in that: The method of dividing the vehicle surrounding space into a plurality of element grids of equal volume and performing preliminary clustering on the multi-frame target point clouds to obtain the average value of each dimension feature of all target point clouds in each element grid corresponding to the multi-frame target point clouds also includes: The multi-frame target point clouds are divided into corresponding element grids according to their multi-dimensional features, the target point clouds falling into each element grid are collected, and then the average value of each dimensional feature of all target point clouds in each element grid is calculated.

4. The 4D millimeter wave radar dynamic and static point cloud detection method based on deep learning according to claim 1 is characterized in that: The step S14 further includes: When the size of the two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud is inconsistent with the size of the two-dimensional feature bird's-eye view corresponding to the multi-frame target point cloud, return to step S13 to continue sparse convolution processing until the sizes of the two two-dimensional feature bird's-eye views are consistent.

5. The 4D millimeter wave radar dynamic and static point cloud detection method based on deep learning according to claim 1 is characterized in that: The step S16 further includes: When the feature detection results meet the expected results, this round of processing ends.

Images