Stream refinement feature design method and apparatus for 4d millimeter wave radar point cloud

Abstract

This invention discloses a method and apparatus for stream refinement feature design for 4D millimeter-wave radar point clouds, belonging to the field of millimeter-wave radar 3D point cloud target detection. A dual-branch parallel feature extraction architecture is employed, performing mainstream feature encoding and temporal stream refinement encoding on the spatially discretized point column data. The mainstream feature encoding branch extracts the instantaneous geometric and physical features of the point columns; the temporal stream refinement encoding branch calculates three Delta features—density change, reflection intensity change, and velocity change—between adjacent time steps. After numerical pruning and multilayer perceptron processing, global context aggregation is performed through a Transformer encoder incorporating position encoding, and low-reflection-intensity clutter is filtered using dynamic attention masks. Finally, a smart gating mechanism fuses the mainstream features and temporal stream refinement features to generate an enhanced feature vector, which is then distributed as a bird's-eye view feature map. This vector is input into the detection network to complete target classification and 3D bounding box regression.

Description

Technical Field

[0001] This invention belongs to the field of millimeter-wave radar 3D point cloud target detection, specifically relating to a method and apparatus for designing flow refinement features for 4D millimeter-wave radar point clouds. Background Technology

[0002] Target detection is a crucial task in autonomous driving environmental perception systems. Currently, most target detection relies on LiDAR data, and LiDAR-based 3D target detection is the cornerstone of building high-precision autonomous driving perception systems. With its ability to generate high-density, high-precision 3D point clouds, LiDAR possesses unparalleled advantages in reconstructing environmental geometry and accurately delineating object contours, making it the mainstream high-performance detection solution. However, this technology also has inherent limitations: First, its performance is severely degraded in adverse weather conditions such as rain, snow, and fog due to laser beam scattering and absorption; second, LiDAR lacks the ability to directly measure target velocity, requiring indirect inference through multi-frame point cloud matching, which not only increases the computational burden but also limits the instantaneous response to highly dynamic targets. Furthermore, the high hardware costs and data annotation fees, as well as the poor generalization ability of the model across different scenarios, also pose challenges to the large-scale deployment of LiDAR.

[0003] The disadvantage of millimeter-wave radar point clouds compared to lidar lies in their lower point cloud density. Therefore, multi-frame fusion data is typically used to enrich the number of point clouds. Furthermore, the velocity measured by millimeter-wave radar only includes radial velocity information. Multi-frame data, with timestamps, can provide richer information beyond the number of point clouds, such as the flow of the point cloud at different times. The point cloud flow can provide velocity information, supplementing the measured velocity.

[0004] Most existing millimeter-wave radar point cloud feature extractor designs directly process raw information such as coordinates, velocity, and reflection intensity using a multilayer perceptron encoder for feature encoding. Few designs perform initial processing on the raw information to obtain specifically designed features before further encoding them using a multilayer perceptron. Current feature extractor designs can only obtain the target's radial velocity information or only process radial velocity information; for multi-frame point cloud data, they simply perform point cloud superposition to compensate for the sparseness of millimeter-wave point clouds. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a method and apparatus for stream refinement feature design of point clouds for 4D millimeter-wave radar. It uses the changing trends of point cloud flow, velocity, and intensity from multiple frames as new features, and aggregates these features across frames using a Transformer. The features from this new feature extractor branch and the point-column encoder are then aggregated together and input into the backbone network for processing. The specific technical solution is as follows:

[0006] A method for designing flow refinement features for 4D millimeter-wave radar point clouds includes the following steps:

[0007] Step 1, Data Preparation: Prepare a 4D millimeter-wave point cloud dataset with timestamps and multi-frame fusion. Each frame of the point cloud contains three-dimensional spatial coordinates, reflection intensity, radial velocity, radial velocity after vehicle motion compensation, and timestamp information.

[0008] Step 2, spatial discretization of point cloud: spatial discretization of the input continuous multi-frame point cloud data, dividing the entire three-dimensional space into a two-dimensional grid map, each grid extends vertically to form a point column, and all points falling into the same grid are grouped into the same point column.

[0009] Step 3, Mainstream Feature Encoding: Encode the instantaneous features through the mainstream feature encoder branch, including: for each point in the point column, calculate its coordinate offset relative to the centroid and geometric center of the point column to enhance the point-level features; process the feature vectors of all points in the point column, and aggregate the features of multiple points into a mainstream feature vector that represents the combined geometric and physical properties of the point column at the current moment through max pooling operation.

[0010] Step 4, Temporal Stream Refinement Encoding: The dynamic trend is modeled through the temporal stream refinement encoder branch, including: assigning points within a column to different time steps based on the timestamp information of each point; statistically aggregating the point cloud within each time step; calculating three macroscopic temporal features: the number of points within the column, the average reflection intensity, and the average radial velocity at that time step; calculating the difference between macroscopic features between adjacent time steps; generating three Delta features: density change feature, reflection intensity change feature, and velocity change feature; and finally outputting the temporal stream refinement feature vector.

[0011] Step 5, Feature Fusion and Object Detection: The mainstream feature vector and the refined temporal stream feature vector are concatenated along the feature channel dimension to construct a joint feature vector. This joint feature vector is then input into a gated neural network to obtain gating weight coefficients. These gating weight coefficients are then multiplied element-wise with the refined temporal stream feature vector to obtain the cleaned weighted temporal stream features. The mainstream feature vector and the weighted temporal stream features are then concatenated to form the final enhanced feature vector. The enhanced feature vector of each point column is then distributed onto a 2D feature map canvas based on its 2D coordinates to form a dense bird's-eye view feature map. This bird's-eye view feature map is then input into a 2D convolutional neural network structure and a detection head for object classification and 3D bounding box regression, completing the detection process.

[0012] A flow refinement feature design device for 4D millimeter-wave radar point clouds, comprising:

[0013] The data preparation module prepares a 4D millimeter-wave point cloud dataset with timestamps, consisting of multi-frame fusion. Each frame of the point cloud contains three-dimensional spatial coordinates, reflection intensity, radial velocity, radial velocity after vehicle motion compensation, and timestamp information.

[0014] The point cloud spatial discretization module spatially discretizes the input continuous multi-frame point cloud data, dividing the entire three-dimensional space into a two-dimensional grid map. Each grid extends vertically to form a point column, and all points falling into the same grid are grouped into the same point column.

[0015] The mainstream feature encoding module encodes instantaneous features through the mainstream feature encoder branch, including: for each point within the point column, calculating its coordinate offset relative to the centroid and geometric center of the point column to enhance point-level features; processing the feature vectors of all points within the point column, and aggregating the features of multiple points into a mainstream feature vector that represents the combined geometric and physical properties of the point column at the current moment through max pooling operation;

[0016] The temporal stream refinement coding module models dynamic trends through the temporal stream refinement encoder branch, including: assigning points within a column to different time steps based on the timestamp information of each point; statistically aggregating the point cloud within each time step to calculate three macroscopic temporal features: the number of points within the column, the average reflection intensity, and the average radial velocity; calculating the difference between macroscopic features between adjacent time steps to generate three Delta features: density change feature, reflection intensity change feature, and velocity change feature; and finally outputting the temporal stream refinement feature vector.

[0017] The feature fusion and target detection module concatenates the mainstream feature vector and the refined temporal stream feature vector along the feature channel dimension to construct a joint feature vector. This joint feature vector is then input into a gated neural network to obtain gating weight coefficients. The gating weight coefficients are then multiplied element-wise with the refined temporal stream feature vector to obtain cleaned weighted temporal stream features. Finally, the mainstream feature vector and the weighted temporal stream features are concatenated to form the final enhanced feature vector. The enhanced feature vector of each point column is then distributed onto a 2D feature map canvas according to its 2D coordinates to form a dense bird's-eye view feature map. The bird's-eye view feature map is then input into a 2D convolutional neural network structure and a detection head for target classification and 3D bounding box regression, completing the detection process.

[0018] An electronic device includes: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method.

[0019] A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to implement the method described thereon.

[0020] The present invention has the following beneficial effects:

[0021] 1. Dual-branch Parallel Feature Extraction Architecture: This invention proposes a dual-branch architecture for parallel processing of input millimeter-wave radar point clouds. One branch (the main feature encoding branch) is responsible for extracting the instantaneous geometric and physical features of the point pillars; the other innovative branch (the temporal stream refinement branch) is dedicated to in-depth mining and encoding of the dynamic change trends of the point pillars over time. This parallel design enables the model to simultaneously and efficiently perceive both the static attributes and dynamic behaviors of objects.

[0022] 2. Extraction and Fusion of Multimodal Temporal Variation Features (Delta Features): Unlike existing technologies that only utilize single point count or velocity information, this invention proposes a method for extracting and fusing multiple temporal variation features. Specifically, this method calculates and stitches together the point count changes (density flow), average radial velocity changes (velocity flow), and average reflection intensity changes (material / location flow) of point pillars within adjacent time steps. This multimodal "Delta feature" provides unprecedentedly rich information for describing the dynamic evolution of objects.

[0023] 3. Transformer-based Global Temporal Context Aggregation: This invention employs a Transformer encoder to perform deep context aggregation of multimodal temporal change features. Through its self-attention mechanism, the model can automatically learn and associate all points and pillars with similar motion patterns in the scene, even if they are not directly adjacent in space. This allows the model to understand the overall dynamics of a complete moving object from a global perspective, rather than limiting its attention to isolated changing local areas, thereby significantly improving the robustness of detecting dynamic targets (especially non-rigid or morphologically variable moving targets).

[0024] 4. Robust Temporal Feature Calculation Method: To address the computational noise problem caused by the sparsity of point clouds, this invention introduces a numerical clipping operation when calculating the Delta VR feature. This operation effectively suppresses extreme outliers caused by points "jumping in / out" between Pillars, enhancing the stability and reliability of temporal features while preserving effective dynamic signals, thereby improving the training stability and final performance of the entire model.

[0025] This invention, by calculating and fusing three multimodal temporal features—density variation, velocity variation, and reflection intensity variation—can capture more complex dynamic behaviors of objects, such as acceleration / deceleration trends, intentions to enter / leave a region, and even changes in radar reflection characteristics caused by attitude changes. This rich and three-dimensional dynamic description enables the model to more accurately predict the trajectory of moving objects and distinguish between dynamic and static targets, thereby significantly improving detection accuracy and robustness.

[0026] The Transformer encoder introduced in this invention utilizes its self-attention mechanism to establish global dependencies between all points in a scene. This allows the model to "associate" multiple points with similar motion patterns, understanding that they collectively constitute a complete moving object. This global context-aware capability effectively solves the problem of incomplete detection caused by irregular object shapes or partial occlusion, improving the overall target recognition capability.

[0027] This invention introduces a numerical clipping mechanism when calculating velocity variation features, which effectively suppresses extreme data points caused by computational artifacts. This improves the stability and reliability of input features while preserving the true motion signal. This not only makes the model training process more stable but also makes the final detection results more robust to sensor noise and data sparsity. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the network model structure of the three-dimensional target detection system of the present invention.

[0029] Figure 2 This is a schematic diagram of the process of the present invention.

[0030] Figure 3a This is a schematic diagram of the object detection results projected onto an image in the VoD dataset implemented by this invention.

[0031] Figure 3b This is a schematic diagram of the target detection results implemented in the VoD dataset under the visualization of 3D point cloud. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other. To achieve the above objectives, this invention adopts the following technical solution.

[0033] To enable those skilled in the art to better understand the technical solutions of this application, some of the technical terms and symbols involved in this application are hereby explained: In the following text, , , These represent the x-axis, y-axis, and z-axis components of the point cloud's coordinates in three-dimensional space, respectively; in this field, The radar cross section (RCS) is used to characterize the reflectivity of a target; therefore, in this invention, Represents the reflection intensity parameter; This represents the raw radial velocity obtained from radar measurements; This represents the radial velocity after compensation for the vehicle's motion. This represents the timestamp information corresponding to the point cloud acquisition. Features (or) (Feature) represents the amount of change in macroscopic physical features between adjacent time steps or frames; It refers to the columnar voxel unit formed by rasterizing the three-dimensional point cloud space on the horizontal plane and extending it in the vertical direction as needed.

[0034] To achieve the above objective—namely, to fully utilize temporal information to improve the accuracy of 3D target detection under the condition of only having radial velocity information from millimeter-wave radar and multi-frame point cloud data—the technical solution adopted in this invention is as follows:

[0035] A method for designing flowing refinement features for 4D millimeter-wave radar point clouds is presented, tested on a public dataset, and is expected to be applied in future autonomous driving or intelligent transportation systems equipped with millimeter-wave radar. The core of this method lies in designing a temporal flowing refinement module in parallel with a mainstream feature encoding network. This module is specifically designed to mine and quantify the dynamic change trends of multi-frame fused point cloud data in the temporal dimension as the original feature differences between frames. These differences are first processed by a multilayer perceptron, then positionally encoded, and finally processed by a Transformer encoder to obtain deep features. Subsequently, this "temporal flowing refined feature vector" is concatenated with the traditional instantaneous point cloud feature "mainstream feature vector," thereby providing richer input information for subsequent detection networks, including object motion history and trends, significantly improving the detection performance of dynamic targets.

[0036] like Figure 1 As shown in the diagram, this invention proposes a network model structure for a 3D target detection system. The system takes raw 4D millimeter-wave radar point clouds as input. The network structure includes a feature extractor, a backbone, a neck, and a detection head. The feature extractor includes a mainstream feature coding branch and a temporal flow refinement branch, which respectively output mainstream feature vectors and temporal flow refined feature vectors. Finally, these two feature vectors are fused to output the final detection result. This invention further proposes a flow refinement feature design method for 4D millimeter-wave radar point clouds, the specific flowchart of which is shown below. Figure 2 As shown.

[0037] Step 1: System initialization and data preparation.

[0038] System initialization includes preparing a multi-frame fused 4D millimeter-wave point cloud dataset (VoD) with timestamps and the corresponding software algorithms. Each frame of the point cloud contains a series of points, each with three-dimensional spatial coordinates (…). ), reflection intensity Radial velocity ( ), radial velocity after vehicle motion compensation ( ) and a unique timestamp ( )information.

[0039] Step 2: Point cloud spatial discretization and dual-branch feature extraction architecture.

[0040] The input multi-frame point cloud data is first spatially discretized. Specifically, the entire 3D space is divided into a 2D grid map, with each grid extending infinitely in the vertical direction to form a "pillar". All points falling within the same grid are grouped into the same pillar. After discretization, all non-empty pillars and their internal point cloud data are fed in parallel into two feature extraction branches: the main feature encoder branch and the temporal stream refinement encoder branch.

[0041] Step 3: The main feature encoder branch encodes the instantaneous features.

[0042] This branch is responsible for extracting the static and dynamic features of each point column at the current time. Its implementation process is as follows:

[0043] Step 3.1: Point-level feature enhancement: For each point within the point column, in addition to its original features (such as...) In addition to calculating the coordinate offset of the point relative to the centroid and geometric center of the cylinder, the enhanced features are also calculated. These enhanced features together constitute the input feature vector for each point.

[0044] Step 3.2: Pillar-level Feature Aggregation: Using a small neural network, such as a Pillar FeatureNet layer (PFN), the feature vectors of all points within a pillar are processed. A max-pooling operation is then used to aggregate the features of multiple points into a single, fixed-dimensional main feature vector. This vector represents the combined geometric and physical properties of the pillar at the current time step.

[0045] Step 4: The temporal flow refines the encoder branch to model the dynamic trend.

[0046] This branch runs in parallel with step three, aiming to deeply explore the patterns of point cloud evolution over time within the point pillars. Its implementation process is as follows:

[0047] Step 4.1: Temporal Feature Aggregation: Based on the timestamp information of each point, the points within a point column are assigned to different time steps (corresponding to different radar scan frames). Then, the point clouds within each time step are statistically aggregated to calculate three key macroscopic temporal features: the number of point clouds within the column at that time step, the average reflection intensity, and the average radial velocity.

[0048] Step 4.2: Calculation of temporal variation features: To capture dynamic change trends, the differences in macroscopic features between adjacent time steps are calculated, generating three types of "Delta features":

[0049] Density variation characteristic (Delta Density): The difference in the number of points, reflecting the inflow or outflow of objects.

[0050] Delta RCS: The difference in average reflectance, reflecting changes in the material or part of an object.

[0051] Velocity variation characteristic (Delta VR): The difference in average radial velocity, reflecting the acceleration or deceleration trend of an object.

[0052] In processing the velocity variation feature (Delta VR), the algorithm implementation considers the data sparsity caused by the target's cross-grid motion. If a point column is missing a single frame in two adjacent time steps (i.e., the current time step has a non-empty point cloud, while the historical time steps are empty), the system does not perform direct subtraction on the missing frame. Instead, it assigns a preset zero-padding vector to the baseline macroscopic features of the missing time step to maintain the alignment of the feature tensor dimensions. Subsequently, to suppress extreme difference noise caused by point cloud sparsity, this step performs a numerical clipping operation on the generated features (especially Delta VR), forcibly limiting them to a preset reasonable physical limit (e.g., [-5.0, 5.0]) to improve the robustness and numerical stability of the initial features before inputting into the Transformer.

[0053] Step 4.3: Global context aggregation based on the Transformer multi-head attention mechanism.

[0054] The three Delta features mentioned above are concatenated along the channel dimension to form a composite feature vector containing multiple dynamic information. This vector is then fed into an input encoder composed of a multilayer perceptron (MLP) to map it into a dense, initially fused temporal stream feature.

[0055] To achieve deep modeling of the spatiotemporal relationships between discretized units in a scene, this invention utilizes a Transformer architecture to globally enhance the initial temporal flow features output. First, a 2D position encoding based on a sine function is introduced. A position embedding vector of the corresponding dimension is generated based on the 2D spatial coordinates of each non-empty discretized unit in the bird's-eye view, and this vector is superimposed on the initial temporal flow features to endow the model with the ability to perceive and recognize the spatial distribution of units. Subsequently, the feature sequence infused with position information is fed into a multi-layer transformer encoder, where the query matrix, key matrix, and value matrix are computed in parallel using a multi-head self-attention mechanism. By calculating the dot product score between the query matrix and key matrix of different discretized units, units with similar motion patterns in the scene (even if they are not directly adjacent in space) are automatically learned and associated. When processing the Delta RCS (Delta Reflection Score), after this dot product score calculation, this scheme introduces a dynamic attention mask filtering mechanism based on radar physics priors. Specifically, the system presets an effective lower limit threshold for reflection intensity. If the average reflection intensity (Mean RCS) of a discretized unit participating in the calculation is lower than the lower limit threshold, the system determines that the unit is a false target or background noise caused by radar multipath effect. At this time, a corresponding mask signal is generated, and the dot product score corresponding to the false target in the attention score matrix is ​​forcibly replaced with an extremely negative value (e.g., ...). ), and then cover it up.

[0056] Subsequently, the value vectors are weighted and summed using attention weights that have been filtered and normalized through masking. This allows the feature representation of each unit to capture scene-level global dynamic context information and effectively avoids contamination of global features by low-reflection-intensity radar clutter. Finally, the aggregated features undergo layer normalization to output a time-series refined feature vector, thereby significantly enhancing the model's robustness in recognizing non-rigid or morphologically variable targets.

[0057] Step 5: Feature fusion and target detection.

[0058] Step 5.1: Feature Fusion Based on Intelligent Gating Mechanism. To suppress temporal stream noise caused by 4D millimeter-wave radar clutter, this step abandons the conventional direct stitching method and introduces an intelligent gating fusion module to dynamically learn the confidence weights of the temporal stream based on the joint spatial and temporal features. The specific implementation process is as follows:

[0059] First, the mainstream feature vectors representing spatial geometry generated in step three and the time-flow refined feature vectors output in step four are initially concatenated along the feature channel dimension to construct a joint feature vector.

[0060] Subsequently, the joint feature vector is input into a gated neural network consisting of a fully connected layer (Linear), a batch normalization layer (BatchNorm1D), a ReLU activation function, and a sigmoid function. The batch normalization layer is used to balance the scale differences of features from different sources and accelerate convergence; the sigmoid function is used to map the network output to gate weight coefficients with values ​​ranging from 0 to 1.

[0061] Next, a soft-gating operation is performed, which involves multiplying the gate weight coefficients element-wise with the refined feature vector of the input temporal stream. If the gate coefficient of a certain point column approaches 0, it indicates that its temporal stream is likely a multipath noise point, and the system suppresses it; if it approaches 1, it indicates that it is effective motion, and the system enhances it, thereby outputting the cleaned weighted temporal stream features.

[0062] Finally, the original mainstream feature vectors without gating decay are concatenated with the cleaned weighted temporal flow features along the feature dimension to form the final enhanced feature vectors (final_features). This step preserves the original mainstream feature vectors to prevent the gating network from destroying the basic geometric topology of the point cloud during the early stages of training. This enhanced feature vector simultaneously achieves a high-fidelity description of the instantaneous spatial state and a clean understanding of historical dynamic trends after denoising.

[0063] Step 5.2: Generate Bird’s-eye View Feature Map: Scatter the enhanced feature vector of each point column onto a two-dimensional feature map canvas according to its two-dimensional coordinates to form a dense bird’s-eye view (BEV) feature map.

[0064] Step 5.3: Target Detection: Input the information-rich BEV feature map into a standard two-dimensional convolutional neural network structure (e.g., Backbone, RPN, Neck) and a detection head to perform target classification and three-dimensional bounding box regression, thus completing the entire detection process. Figure 3a This is a schematic diagram of the object detection results projected onto an image in the VoD dataset implemented by this invention. Figure 3b This is a schematic diagram of the target detection results implemented in the VoD dataset under the visualization of 3D point cloud.

[0065] Through the above steps, the method of the present invention can effectively utilize the temporal information in multi-frame millimeter-wave radar data to generate more discriminative feature representations, thereby enabling more accurate detection, especially for moving targets, where the judgment of their state and intent will be more precise.

[0066] Another aspect of the present invention provides a flow refinement feature design apparatus for 4D millimeter-wave radar point clouds, comprising:

[0067] The data preparation module prepares a 4D millimeter-wave point cloud dataset with timestamps, consisting of multi-frame fusion. Each frame of the point cloud contains three-dimensional spatial coordinates, reflection intensity, radial velocity, radial velocity after vehicle motion compensation, and timestamp information.

[0068] The point cloud spatial discretization module spatially discretizes the input continuous multi-frame point cloud data, dividing the entire three-dimensional space into a two-dimensional grid map. Each grid extends vertically to form a point column, and all points falling into the same grid are grouped into the same point column.

[0069] The mainstream feature encoding module encodes instantaneous features through the mainstream feature encoder branch, including: for each point within the point column, calculating its coordinate offset relative to the centroid and geometric center of the point column to enhance point-level features; processing the feature vectors of all points within the point column, and aggregating the features of multiple points into a mainstream feature vector that represents the combined geometric and physical properties of the point column at the current moment through max pooling operation;

[0070] The temporal stream refinement coding module models dynamic trends through the temporal stream refinement encoder branch, including: assigning points within a column to different time steps based on the timestamp information of each point; statistically aggregating the point cloud within each time step to calculate three macroscopic temporal features: the number of points within the column, the average reflection intensity, and the average radial velocity; calculating the difference between macroscopic features between adjacent time steps to generate three Delta features: density change feature, reflection intensity change feature, and velocity change feature; and finally outputting the temporal stream refinement feature vector.

[0071] The feature fusion and target detection module concatenates the mainstream feature vector and the refined temporal stream feature vector along the feature channel dimension to construct a joint feature vector. This joint feature vector is then input into a gated neural network to obtain gating weight coefficients. The gating weight coefficients are then multiplied element-wise with the refined temporal stream feature vector to obtain cleaned weighted temporal stream features. Finally, the mainstream feature vector and the weighted temporal stream features are concatenated to form the final enhanced feature vector. The enhanced feature vector of each point column is then distributed onto a 2D feature map canvas according to its 2D coordinates to form a dense bird's-eye view feature map. The bird's-eye view feature map is then input into a 2D convolutional neural network structure and a detection head for target classification and 3D bounding box regression, completing the detection process.

[0072] Another aspect of the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method.

[0073] Another aspect of the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to implement the method described thereon.

Claims

1. A method for designing flow refinement features for 4D millimeter-wave radar point clouds, characterized in that, Includes the following steps: Step 1, Data Preparation: Prepare a 4D millimeter-wave point cloud dataset with timestamps and multi-frame fusion. Each frame of the point cloud contains three-dimensional spatial coordinates, reflection intensity, radial velocity, radial velocity after vehicle motion compensation, and timestamp information. Step 2, spatial discretization of point cloud: spatial discretization of the input continuous multi-frame point cloud data, dividing the entire three-dimensional space into a two-dimensional grid map, each grid extends vertically to form a point column, and all points falling into the same grid are grouped into the same point column. Step 3, Mainstream Feature Encoding: Encode the instantaneous features through the mainstream feature encoder branch, including: for each point in the point column, calculate its coordinate offset relative to the centroid and geometric center of the point column to enhance the point-level features; process the feature vectors of all points in the point column, and aggregate the features of multiple points into a mainstream feature vector that represents the combined geometric and physical properties of the point column at the current moment through max pooling operation. Step 4, Temporal Stream Refinement Encoding: The dynamic trend is modeled through the temporal stream refinement encoder branch, including: assigning points within a column to different time steps based on the timestamp information of each point; statistically aggregating the point cloud within each time step; calculating three macroscopic temporal features: the number of points within the column, the average reflection intensity, and the average radial velocity at that time step; calculating the difference between macroscopic temporal features between adjacent time steps; generating three Delta features: density change feature, reflection intensity change feature, and velocity change feature; and finally outputting the temporal stream refinement feature vector. Step 5, Feature Fusion and Object Detection: The mainstream feature vector and the refined temporal stream feature vector are concatenated along the feature channel dimension to construct a joint feature vector. This joint feature vector is then input into a gated neural network to obtain gating weight coefficients. These gating weight coefficients are then multiplied element-wise with the refined temporal stream feature vector to obtain the cleaned weighted temporal stream features. The mainstream feature vector and the weighted temporal stream features are then concatenated to form the final enhanced feature vector. The enhanced feature vector of each point column is then distributed onto a 2D feature map canvas based on its 2D coordinates to form a dense bird's-eye view feature map. This bird's-eye view feature map is then input into a 2D convolutional neural network structure and a detection head for object classification and 3D bounding box regression, completing the detection process.

2. The method for designing flow refinement features for 4D millimeter-wave radar point clouds according to claim 1, characterized in that, Step four, which calculates the difference in macroscopic temporal characteristics between adjacent time steps, specifically includes: Density variation is characterized by the difference in the number of points, reflecting the inflow or outflow of objects; reflection intensity variation is characterized by the difference in the average reflection intensity, reflecting changes in the material or part of the object; velocity variation is characterized by the difference in the average radial velocity, reflecting the acceleration or deceleration trend of the object.

3. The method for designing flow refinement features for 4D millimeter-wave radar point clouds according to claim 1, characterized in that, Step four includes: when calculating the velocity change features, zero-value vectors are used to fill in the missing cases of a single frame, and numerical clipping is performed on the generated features to limit them to a preset physical limit; the three Delta features are concatenated in the channel dimension and mapped to the initial temporal stream features through a multilayer perceptron, and then a two-dimensional position encoding based on a sine function is introduced and superimposed on the initial temporal stream features, and then fed into a multilayer Transformer encoder to calculate the attention score between different points through a multi-head self-attention mechanism, and a dynamic attention mask filtering mechanism based on radar physical prior is introduced to mask points with average reflection intensity below the lower limit threshold, and finally the temporal stream refined feature vector is output after layer normalization.

4. The method for designing flow refinement features for 4D millimeter-wave radar point clouds according to claim 3, characterized in that, The dynamic attention masking filtering mechanism in step four is as follows: a preset effective lower limit threshold for reflection intensity is used. If the average reflection intensity of the point pillars involved in the calculation is lower than the lower limit threshold, the point pillar is determined to be a false target or background noise generated by radar multipath effect. A corresponding mask signal is generated to forcibly replace the dot product score of the point pillar in the attention score matrix with an extremely negative value for masking, effectively avoiding the pollution of global features by low reflection intensity radar clutter.

5. The method for designing flow refinement features for 4D millimeter-wave radar point clouds according to claim 1, characterized in that, The gated neural network in step five includes a fully connected layer, a batch normalization layer, a ReLU activation function, and a Sigmoid function. The batch normalization layer is used to balance the scale differences of features from different sources and accelerate convergence. The Sigmoid function is used to map the network output to a gate weight coefficient with a value between 0 and 1, so as to dynamically learn the confidence weight of the temporal stream based on the joint features of space and time.

6. The method for designing flow refinement features for 4D millimeter-wave radar point clouds according to claim 1, characterized in that, In step five, the process of multiplying the gate weight coefficients element-wise with the refined temporal flow feature vector to obtain the cleaned weighted temporal flow feature is as follows: the gate weight coefficients are multiplied element-wise with the input refined temporal flow feature vector. If the gate weight coefficient of a certain point column approaches 0, it indicates that its temporal flow is likely to be a multipath noise point, which is suppressed. If it approaches 1, it indicates that it is an effective motion, which is enhanced, thereby outputting the cleaned weighted temporal flow feature.

7. The method for designing flow refinement features for 4D millimeter-wave radar point clouds according to claim 1, characterized in that, In step five, when concatenating the mainstream feature vector with the weighted temporal stream features, the original mainstream feature vector that has not undergone gated decay is retained.

8. A device for designing flow refinement features for 4D millimeter-wave radar point clouds, characterized in that, include: The data preparation module prepares a 4D millimeter-wave point cloud dataset with timestamps, consisting of multi-frame fusion. Each frame of the point cloud contains three-dimensional spatial coordinates, reflection intensity, radial velocity, radial velocity after vehicle motion compensation, and timestamp information. The point cloud spatial discretization module spatially discretizes the input continuous multi-frame point cloud data, dividing the entire three-dimensional space into a two-dimensional grid map. Each grid extends vertically to form a point column, and all points falling into the same grid are grouped into the same point column. The mainstream feature encoding module encodes instantaneous features through the mainstream feature encoder branch, including: for each point within the point column, calculating its coordinate offset relative to the centroid and geometric center of the point column to enhance point-level features; processing the feature vectors of all points within the point column, and aggregating the features of multiple points into a mainstream feature vector that represents the combined geometric and physical properties of the point column at the current moment through max pooling operation; The temporal stream refinement coding module models dynamic trends through the temporal stream refinement encoder branch, including: assigning points within a column to different time steps based on the timestamp information of each point; statistically aggregating the point cloud within each time step to calculate three macroscopic temporal features: the number of points within the column, the average reflection intensity, and the average radial velocity; calculating the difference between macroscopic temporal features between adjacent time steps to generate three Delta features: density change feature, reflection intensity change feature, and velocity change feature; and finally outputting the temporal stream refinement feature vector. The feature fusion and target detection module concatenates the mainstream feature vector and the refined temporal stream feature vector along the feature channel dimension to construct a joint feature vector. This joint feature vector is then input into a gated neural network to obtain gating weight coefficients. The gating weight coefficients are then multiplied element-wise with the refined temporal stream feature vector to obtain cleaned weighted temporal stream features. Finally, the mainstream feature vector and the weighted temporal stream features are concatenated to form the final enhanced feature vector. The enhanced feature vector of each point column is then distributed onto a 2D feature map canvas according to its 2D coordinates to form a dense bird's-eye view feature map. The bird's-eye view feature map is then input into a 2D convolutional neural network structure and a detection head for target classification and 3D bounding box regression, completing the detection process.

9. An electronic device, characterized in that, include: One or more processors; A memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the flow refinement feature design method for 4D millimeter-wave radar point clouds as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed by a processor, cause the processor to implement the flow refinement feature design method for 4D millimeter-wave radar point clouds as described in any one of claims 1 to 7.

Images