A deep learning-based 4D millimeter wave radar dynamic and static point cloud detection method

Abstract

This invention relates to the field of intelligent driving technology, specifically disclosing a deep learning-based method for detecting dynamic and static point clouds in 4D millimeter-wave radar. The method includes: acquiring multi-dimensional features of target point clouds in the vehicle's surrounding environment; processing these features to obtain a two-dimensional feature bird's-eye view corresponding to a single frame of the target point cloud and two-dimensional feature bird's-eye views corresponding to multiple frames of the target point cloud; when the two two-dimensional feature bird's-eye views are of equal size, associating the features in the two-dimensional feature bird's-eye views corresponding to the target point cloud in the single frame with the features in the two-dimensional feature bird's-eye views corresponding to the target point cloud in the multiple frames; after feature association, fusing the two two-dimensional feature bird's-eye views to generate a fused feature map, and performing a convolution operation on the fused feature map; detecting the fused feature map after the convolution operation, and outputting the feature detection result. This invention optimizes the detection accuracy and efficiency of 4D millimeter-wave radar by fusing information from single and multiple frames using deep learning.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, and more specifically, to a method for detecting dynamic and static point clouds using 4D millimeter-wave radar based on deep learning. Background Technology

[0002] When using 4D imaging millimeter-wave radar for perception, relying solely on single-frame information often leads to missed detections due to the sparsity of distant point clouds. However, using information from several adjacent frames for compensation can cause motion blur when detecting moving targets due to the accumulation of multiple frames, making localization difficult. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a method for detecting dynamic and static point clouds of 4D millimeter-wave radar based on deep learning. This method optimizes the detection accuracy and efficiency of 4D imaging millimeter-wave radar by fusing information from single frames and multiple frames based on deep learning.

[0004] As a first aspect of the present invention, a method for detecting dynamic and static point clouds of 4D millimeter-wave radar based on deep learning is provided, the method comprising the following steps:

[0005] Step S11: Obtain multi-dimensional features of the target point cloud of the vehicle's surrounding environment collected by the 4D millimeter-wave radar; wherein, the target point cloud is processed separately according to single frame target point cloud and multi-frame target point cloud;

[0006] Step S12: Divide the space around the vehicle into multiple element grids of equal volume, perform preliminary clustering on the single-frame target point cloud to obtain the average value of each dimension feature of all target point clouds in each element grid corresponding to the single-frame target point cloud; at the same time, perform preliminary clustering on the multi-frame target point cloud to obtain the average value of each dimension feature of all target point clouds in each element grid corresponding to the multi-frame target point cloud.

[0007] Step S13: Perform sparse convolution processing on the average value of all dimensional features of all target point clouds in each element cell corresponding to the single-frame target point cloud to obtain the average value of three dimensional features of all target point clouds in each element cell corresponding to the single-frame target point cloud, and compress the average value of three dimensional features of all target point clouds in each element cell corresponding to the single-frame target point cloud into a two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud; simultaneously, perform sparse convolution processing on the average value of all dimensional features of all target point clouds in each element cell corresponding to the multi-frame target point cloud to obtain the average value of three dimensional features of all target point clouds in each element cell corresponding to the multi-frame target point cloud, and compress the average value of three dimensional features of all target point clouds in each element cell corresponding to the multi-frame target point cloud into a two-dimensional feature bird's-eye view corresponding to the multi-frame target point cloud;

[0008] Step S14: When the size of the two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud is the same as the size of the two-dimensional feature bird's-eye view corresponding to the multi-frame target point cloud, the features in the two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud are used as keys and values ​​to associate the features in the two-dimensional feature bird's-eye view corresponding to the multi-frame target point cloud.

[0009] Step S15: After feature association, the two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud and the two-dimensional feature bird's-eye view corresponding to the multi-frame target point cloud are fused to generate a fused feature map based on time dimension stitching, and a convolution operation is performed on the fused feature map.

[0010] Step S16: Detect the fused feature map after the convolution operation and output the feature detection results.

[0011] Furthermore, the process of dividing the vehicle's surrounding environment into multiple element grids of equal volume, performing preliminary clustering of the single-frame target point cloud to obtain the average value of each dimension feature of all target point clouds in each element grid corresponding to the single-frame target point cloud, also includes:

[0012] Based on the multi-dimensional features of the target point cloud in a single frame, the target point cloud in a single frame is divided into corresponding element grids. The target point clouds falling into each element grid are then collected, and the average value of each dimension feature of all target point clouds in each element grid is calculated.

[0013] Furthermore, the process of dividing the vehicle's surrounding environment into multiple element grids of equal volume, performing preliminary clustering of multi-frame target point clouds to obtain the average value of each dimension feature of all target point clouds in each element grid corresponding to the multi-frame target point clouds, also includes:

[0014] Based on the multi-dimensional features of the multi-frame target point cloud, the multi-frame target point cloud is divided into corresponding element grids. The target point clouds falling into each element grid are collected, and then the average value of each dimension feature of all target point clouds in each element grid is calculated.

[0015] Furthermore, step S14 also includes:

[0016] When the size of the two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud is inconsistent with the size of the two-dimensional feature bird's-eye view corresponding to the multi-frame target point cloud, return to step S13 to continue sparse convolution processing until the size of the two two-dimensional feature bird's-eye views is consistent.

[0017] Furthermore, step S16 also includes:

[0018] The current processing round ends when the feature detection results meet the expected results.

[0019] The 4D millimeter-wave radar dynamic and static point cloud detection method based on deep learning provided by this invention has the following advantages:

[0020] (1) This invention integrates information from a single frame and multiple frames, which reduces the probability of false detection and false negative detection compared to the single-frame method;

[0021] (2) This invention integrates information from single frames and multiple frames, which makes the positioning more accurate and avoids ambiguity compared to the multi-frame method;

[0022] (3) The present invention uses the characteristics of a single-frame branch as an index to obtain key thresholds from multi-frame features, which reduces the amount of computation compared to using multiple frames for judgment and optimizes efficiency. Attached Figure Description

[0023] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the following detailed description to explain the invention, but do not constitute a limitation thereof.

[0024] Figure 1 The flowchart shows the 4D millimeter-wave radar dynamic and static point cloud detection method based on deep learning provided by this invention.

[0025] Figure 2 The flowchart illustrates the specific implementation of the deep learning-based 4D millimeter-wave radar dynamic and static point cloud detection method provided by this invention. Detailed Implementation

[0026] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a deep learning-based 4D millimeter-wave radar dynamic and static point cloud detection method proposed according to the present invention. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0027] This embodiment provides a deep learning-based method for detecting dynamic and static point clouds using 4D millimeter-wave radar, such as... Figure 1 As shown, the deep learning-based 4D millimeter-wave radar dynamic and static point cloud detection method includes the following steps:

[0028] Step S11: Acquire multi-dimensional features of the target point cloud of the vehicle's surrounding environment collected by 4D millimeter-wave radar; among which, such as Figure 2As shown, all target point cloud data are input into two main branches: single-frame target point cloud and multi-frame target point cloud, and processed separately according to single-frame target point cloud and multi-frame target point cloud;

[0029] It should be noted that the 4D millimeter-wave radar is installed on the front bumper of the vehicle.

[0030] Step S12: Divide the space around the vehicle into multiple element grids of equal volume, perform preliminary clustering on the single-frame target point cloud to obtain the average value of each dimension feature of all target point clouds in each element grid corresponding to the single-frame target point cloud; at the same time, perform preliminary clustering on the multi-frame target point cloud to obtain the average value of each dimension feature of all target point clouds in each element grid corresponding to the multi-frame target point cloud; this step helps to reduce the complexity of the data and provide a more concise data representation for subsequent processing.

[0031] Preferably, the step of dividing the vehicle's surrounding environment into multiple element grids of equal volume, performing preliminary clustering of the single-frame target point cloud to obtain the average value of each dimension feature of all target point clouds in each element grid corresponding to the single-frame target point cloud, further includes:

[0032] Based on the multi-dimensional features of the target point cloud in a single frame, the target point cloud in a single frame is divided into corresponding element grids. The target point clouds falling into each element grid are then collected, and the average value of each dimension feature of all target point clouds in each element grid is calculated.

[0033] Preferably, the step of dividing the vehicle's surrounding environment into multiple element grids of equal volume, performing preliminary clustering of multi-frame target point clouds to obtain the average value of each dimension feature of all target point clouds in each element grid corresponding to the multi-frame target point clouds, further includes:

[0034] Based on the multi-dimensional features of the multi-frame target point cloud, the multi-frame target point cloud is divided into corresponding element grids. The target point clouds falling into each element grid are collected, and then the average value of each dimension feature of all target point clouds in each element grid is calculated.

[0035] Step S13: Perform sparse convolution processing on the average value of all dimensional features of all target point clouds in each element cell corresponding to the single-frame target point cloud to obtain the average value of three-dimensional features of all target point clouds in each element cell corresponding to the single-frame target point cloud, and compress the average value of three-dimensional features of all target point clouds in each element cell corresponding to the single-frame target point cloud into a two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud; at the same time, perform sparse convolution processing on the average value of all dimensional features of all target point clouds in each element cell corresponding to the multi-frame target point cloud to obtain the average value of three-dimensional features of all target point clouds in each element cell corresponding to the multi-frame target point cloud, and compress the average value of three-dimensional features of all target point clouds in each element cell corresponding to the multi-frame target point cloud into a two-dimensional feature bird's-eye view corresponding to the multi-frame target point cloud; this step not only reduces the amount of data, but also reduces the complexity of subsequent calculations.

[0036] Step S14: When the size of the two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud is the same as the size of the two-dimensional feature bird's-eye view corresponding to the multi-frame target point cloud, the features in the two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud are used as keys and values ​​to associate the features in the two-dimensional feature bird's-eye view corresponding to the multi-frame target point cloud. Since the positions of the same object in the two two-dimensional feature bird's-eye views should be close, the cross-association of local information can improve the accuracy and effectiveness of the association.

[0037] Preferably, step S14 further includes:

[0038] When the size of the 2D feature bird's-eye view corresponding to the single-frame target point cloud is inconsistent with the size of the 2D feature bird's-eye view corresponding to the multi-frame target point cloud, the process returns to step S13 to continue sparse convolution processing until the sizes of the two 2D feature bird's-eye views are consistent. This step ensures the alignment of the feature maps, laying the foundation for subsequent feature association.

[0039] Step S15: After feature association, the two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud and the two-dimensional feature bird's-eye view corresponding to the multi-frame target point cloud are fused to generate a fused feature map based on time dimension stitching, and a convolution operation is performed on the fused feature map; this fused feature map contains more spatiotemporal information and improves detection accuracy.

[0040] Step S16: Detect the fused feature map after the convolution operation and output the feature detection results.

[0041] Preferably, step S16 further includes:

[0042] The current processing round ends when the feature detection results meet the expected results.

[0043] This invention provides a deep learning-based method for detecting dynamic and static point clouds in 4D millimeter-wave radar. By fusing rich semantic information from multiple frame point clouds with precise geometric information from a single frame point cloud, the probability of false detection can be significantly reduced and the target can be located more accurately. At the same time, using the information from a single frame as an index to search for effective thresholds in the accumulated information of multiple frames can reduce the amount of computation and optimize efficiency.

[0044] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for detecting dynamic and static point clouds using 4D millimeter-wave radar based on deep learning, characterized in that, The deep learning-based 4D millimeter-wave radar dynamic and static point cloud detection method includes the following steps: Step S11: Obtain multi-dimensional features of the target point cloud of the vehicle's surrounding environment collected by the 4D millimeter-wave radar; wherein, the target point cloud is processed separately according to single frame target point cloud and multi-frame target point cloud; Step S12: Divide the space around the vehicle into multiple element grids of equal volume, perform preliminary clustering on the single-frame target point cloud to obtain the average value of each dimension feature of all target point clouds in each element grid corresponding to the single-frame target point cloud; at the same time, perform preliminary clustering on the multi-frame target point cloud to obtain the average value of each dimension feature of all target point clouds in each element grid corresponding to the multi-frame target point cloud. Step S13: Perform sparse convolution processing on the average value of all dimensional features of all target point clouds in each element cell corresponding to the single-frame target point cloud to obtain the average value of three dimensional features of all target point clouds in each element cell corresponding to the single-frame target point cloud, and compress the average value of three dimensional features of all target point clouds in each element cell corresponding to the single-frame target point cloud into a two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud; simultaneously, perform sparse convolution processing on the average value of all dimensional features of all target point clouds in each element cell corresponding to the multi-frame target point cloud to obtain the average value of three dimensional features of all target point clouds in each element cell corresponding to the multi-frame target point cloud, and compress the average value of three dimensional features of all target point clouds in each element cell corresponding to the multi-frame target point cloud into a two-dimensional feature bird's-eye view corresponding to the multi-frame target point cloud; Step S14: When the size of the two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud is the same as the size of the two-dimensional feature bird's-eye view corresponding to the multi-frame target point cloud, the features in the two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud are used as keys and values ​​to associate the features in the two-dimensional feature bird's-eye view corresponding to the multi-frame target point cloud. Step S15: After feature association, the two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud and the two-dimensional feature bird's-eye view corresponding to the multi-frame target point cloud are fused to generate a fused feature map based on time dimension stitching, and a convolution operation is performed on the fused feature map. Step S16: Detect the fused feature map after the convolution operation and output the feature detection results.

2. The method for detecting dynamic and static point clouds of 4D millimeter-wave radar based on deep learning according to claim 1, characterized in that, The process of dividing the vehicle's surrounding environment into multiple equal-volume element grids and performing preliminary clustering of the single-frame target point cloud to obtain the average value of each dimension feature of all target point clouds in each element grid corresponding to the single-frame target point cloud also includes: Based on the multi-dimensional features of the target point cloud in a single frame, the target point cloud in a single frame is divided into corresponding element grids. The target point clouds falling into each element grid are then collected, and the average value of each dimension feature of all target point clouds in each element grid is calculated.

3. The method for detecting dynamic and static point clouds of 4D millimeter-wave radar based on deep learning according to claim 1, characterized in that, The process of dividing the vehicle's surrounding environment into multiple equal-volume element grids and performing preliminary clustering of multi-frame target point clouds to obtain the average value of each dimension feature of all target point clouds in each element grid corresponding to the multi-frame target point clouds also includes: Based on the multi-dimensional features of the multi-frame target point cloud, the multi-frame target point cloud is divided into corresponding element grids. The target point clouds falling into each element grid are collected, and then the average value of each dimension feature of all target point clouds in each element grid is calculated.

4. The method for detecting dynamic and static point clouds of 4D millimeter-wave radar based on deep learning according to claim 1, characterized in that, Step S14 further includes: When the size of the two-dimensional feature bird's-eye view corresponding to the single-frame target point cloud is inconsistent with the size of the two-dimensional feature bird's-eye view corresponding to the multi-frame target point cloud, return to step S13 to continue sparse convolution processing until the size of the two two-dimensional feature bird's-eye views is consistent.

5. The method for detecting dynamic and static point clouds of 4D millimeter-wave radar based on deep learning according to claim 1, characterized in that, Step S16 further includes: The current processing round ends when the feature detection results meet the expected results.

Images