Semantic aided 3D object detection method based on 4D millimeter wave radar sparse point cloud

Abstract

The application discloses a kind of semantic auxiliary 3D target detection methods based on 4D millimeter wave radar sparse point cloud.The method is: given a group of 4D radar point cloud data, first respectively extract the pillar feature and voxel feature of 4D radar, then hash into the form of point feature, the center coordinates of pillar are inquired around point, the group feature of these points is extracted and is fused with pillar feature, hash back to the coordinate space of given 4D radar point cloud and obtain bird's eye view perspective feature map by sparse embedding convolution, then point feature is classified, votes and is clustered to obtain target center feature, finally target center feature is fused with bird's eye view perspective feature map and is regressed 3D bounding box.The application can obtain high-precision 3D target detection result by fusing the semantic information of 4D radar point cloud and further integrating the target center feature with dense structure information, so as to make full use of the information in 4D millimeter wave radar sparse point cloud data.

Description

Technical Field

[0001] This invention belongs to the fields of autonomous driving and deep learning, and specifically relates to a semantic-assisted 3D target detection method based on 4D millimeter-wave radar sparse point clouds. Background Technology

[0002] Millimeter-wave radar plays a crucial role in autonomous driving systems due to its all-weather performance, high-precision ranging and velocity measurement, strong penetration capability, and strong anti-interference capability. With the development of millimeter-wave radar technology, high-resolution 4D millimeter-wave radar has emerged. By combining multiple-input multiple-output antenna technology, it achieves comprehensive perception of targets in four dimensions (range, azimuth, elevation, and velocity), significantly enhancing the ability to accurately detect, track, and locate targets in complex environments.

[0003] However, measurements from 4D millimeter-wave radar are susceptible to noise caused by multipath propagation, and the point cloud data it generates is relatively sparse. To overcome these problems, many studies have divided the point cloud data of 4D millimeter-wave radar into uniform pillars and performed feature extraction on this basis. This method effectively avoids the target detection network's overemphasis on noisy interference areas, thus achieving better detection results globally and reducing the algorithm's time and resource consumption. Some methods overcome the sparsity of point clouds by accumulating continuous time steps and have demonstrated that target detection performance can be improved through vehicle motion estimation and dynamic motion correction. Other methods effectively capture long-range information and improve heading angle estimation accuracy by introducing and improving radar pillar attention.

[0004] Although these methods fully extract global contextual information from 4D millimeter-wave radar point clouds, they still fall short in representing the inherent structural details of point clouds.

[0005] To address the shortcomings of the aforementioned methods in representing the inherent structural details of point clouds, various feature extraction and fusion methods have been proposed. For example, one method utilizes complementary features from multiple views, such as the front, top, and left views, to solve the problem of insufficient point cloud information utilization. Another method employs kernel density estimation techniques to extract density-dominant semantic features and fuses them with pillar features at multiple scales. However, these methods do not fully integrate the semantic features in 4D millimeter-wave radar point clouds, particularly lacking attention to the geometric details of objects. The performance of 3D target detection based on 4D millimeter-wave radar sparse point cloud data has not yet been fully explored. Summary of the Invention

[0006] The technical problem this invention aims to solve is to address the shortcomings of existing technologies by providing a semantically assisted 3D target detection method based on 4D millimeter-wave radar sparse point clouds, achieving advanced 3D target detection results. Given a set of 4D millimeter-wave radar point cloud data, this method first extracts pillar features and voxel features using a radar pillar feature extraction network and a voxel semantic feature extractor, respectively. Then, the voxel features are hashed into point features. The center coordinates of non-empty pillars are used to query and fuse surrounding point features. A bird's-eye view feature map is obtained through sparse embedding convolution. Then, the point features are used for classification, voting, and clustering to obtain target center features. Finally, the target center features are fused with the bird's-eye view feature map to regress the 3D bounding box. This invention, by fusing the semantic information of 4D radar point clouds and further integrating structurally dense target center features, can fully extract features from 4D millimeter-wave radar sparse point clouds, thereby obtaining high-precision 3D target detection results.

[0007] This invention is achieved using the following technical solution:

[0008] A semantically assisted 3D target detection method based on 4D millimeter-wave radar sparse point clouds is proposed. This method fully extracts and fuses semantic information from the original radar point cloud, and then further integrates target center features with dense target structure information. The specific steps of the semantically assisted 3D target detection method are as follows:

[0009] S1: Given a 4D radar point cloud P∈R 5×N The radar pillar feature extraction network and the voxel semantic feature extractor are used to extract pillar features respectively. and voxel characteristics Where N is the total number of points in the radar point cloud, C1 is the number of feature channels for pillar features, M1 is the number of non-empty pillars, C2 is the number of feature channels for voxel features, and M2 is the number of non-empty voxels.

[0010] S2: Integrate pillar features and semantic features to generate a bird's-eye view feature map;

[0011] S21: Voxel characteristics Mapping to point features

[0012] S22: Using the center coordinates of each non-empty support as the center, query k points in a neighborhood with a radius of r, extract the group features of these k points, and add the group features to the corresponding support features to obtain the support features with fused semantic features.

[0013] S23: Hash the pillar features of the fused semantics back into the coordinate space of the given 4D radar point cloud to obtain a pseudo-image. Then, sparse embedding convolution is used to generate a bird's-eye view feature map F.b ∈H ×H×W Where C, H, and W represent the number of feature channels, height, and width of the feature map from the bird's-eye view, respectively.

[0014] S3: Classify, vote on, and cluster the point features to obtain the target center features;

[0015] S31: For each point feature f p The point classification network and voting network are used to predict whether a point belongs to the foreground point and its target category, and at the same time, the center coordinates of the target to which the point belongs and its offset from the target center are predicted.

[0016] S32: Cluster the point features to the target centers to generate target center features F c ∈R C′×M C′ and M are the number of feature channels and the number of corresponding center points of the central feature, respectively;

[0017] S4: Integrate target center features into a bird's-eye view feature map to generate a fused bird's-eye view feature map. To increase the number of feature channels in the bird's-eye view feature map;

[0018] S5: 3D bounding box regression is performed using fused bird's-eye view feature maps to obtain the final target detection result.

[0019] In the above technical solution, further, in step S1, a radar pillar feature extraction network is used to extract pillar features. Specifically, the 4D radar point cloud data is fully utilized, and three linear layers with non-shared weights and activation functions are used to learn the spatial, velocity, and reflection intensity information of each point in the pillar. Then, a multilayer perceptron is used to integrate the spatial, velocity, and reflection intensity information of each point in the pillar to obtain pillar features. The extracted pillar features can better express the global dependencies in the scene. A voxel semantic feature extractor is used to extract voxel features. Specifically, the 4D radar point cloud data is voxelized, and sparse convolution is used on non-empty voxels to extract voxel features, avoiding the calculation of the entire grid, thereby significantly reducing the computational complexity of voxel feature extraction.

[0020] Furthermore, the specific steps of step S21 are as follows:

[0021] First, the voxel features corresponding to each original radar point in the given 4D radar point cloud are indexed. Then, the local coordinates of the radar point relative to the voxel center are calculated. Finally, the local coordinates are concatenated with the indexed voxel features to obtain the point features. The local coordinates of the radar point relative to the voxel center are calculated using formula (1):

[0022]

[0023] Among them, c p c represents the local coordinates of the radar point. v Represents voxel coordinates, v s The size of a voxel, v r This represents the boundary of the point cloud data.

[0024] Further, the specific steps in step S22 are as follows: For each non-empty pillar, query the feature f of k points in its neighborhood with radius r. p Then, feature aggregation is performed on these k feature points to obtain group features. The group features are then fused with the corresponding pillar features to obtain the pillar features with fused semantic features. The feature fusion process is shown in formula (2):

[0025]

[0026] Where Ψ represents a multilayer perceptron, and max(·) represents a max pooling operation. Let i be the feature of the i-th point.

[0027] Furthermore, in step S31, during the training of the point classification network and the voting network, the loss function is defined as follows:

[0028] 1) A point classification network is a deep neural network used for point classification. This network predicts whether each point belongs to the foreground and its target category through a series of convolutional and fully connected layers, and then outputs a probability distribution representing the likelihood of the point belonging to each category. Specifically, for each point feature f... p The point classification network outputs a probability vector. Each element corresponds to the probability of a class. The classification loss function of the point classification network uses cross-entropy loss, defined as follows:

[0029]

[0030] in, Here, y represents the cross-entropy loss value, C is the total number of categories, and y is the cross-entrop i It is the one-hot encoding of the real label. for The probability of the i-th category;

[0031] 2) The voting network is a deep neural network used to vote on the target center, predicting the target center point for each point and its offset from the corresponding target center. The network outputs an offset vector. Each element corresponds to an offset in one spatial dimension (such as Euclidean space). The loss function of the voting network is L1 loss, as shown in Equation (4):

[0032]

[0033] in, Here, N is the L1 loss value, and t is the number of radar points. i It is the offset from the radar point to the center of the actual target. for The offset of the i-th radar point from the center of the predicted target;

[0034] 3) To address the class imbalance problem, when the number of positive samples is significantly less than the number of negative samples, FocalLoss is introduced to reduce the weight of easily classified samples and increase the weight of difficult-to-classify samples. The formula is:

[0035]

[0036] in, p is the Focal Loss value. t It is the probability that the network correctly predicts the category, α. t It is the weight that balances the positive and negative samples, and γ is the focusing parameter.

[0037] 4) By combining classification loss, voting loss, and Focal Loss, the total loss function is obtained. This jointly optimizes the classification network and the voting network, accurately predicting the feature category of each point and its offset to the target center, providing crucial information for subsequent target detection. The total loss function is shown in Equation (6):

[0038]

[0039] Further, step S32 specifically involves filtering out target center points whose predicted target center confidence is less than a set threshold, and then aggregating the features of all points that voted for the same target center through a feature aggregation operation similar to that in S22, to obtain the target center feature F. c ∈R C′×M .

[0040] Furthermore, the specific operation of step S4 is to use coordinate convolution to transform the target center feature F c Integrated into bird's-eye view feature map F b By designing two additional channels and filling them with constant values ​​to represent untrained coordinate information, the target center feature is concatenated with the bird's-eye view feature map and the two channels. Then, the concatenated features are merged using 1×1 convolution, batch normalization, and the ReLU activation function to obtain a fused bird's-eye view feature map.

[0041] Furthermore, in step S5, during the training process of the prediction network for 3D bounding box regression, the loss function is defined as follows:

[0042] 1) First, define the parameters of the 3D real bounding box and the 3D anchor box, respectively (x gt ,y gt ,z gt ,w gt ,l gt ,h gt ,θ gt ) and (x a ,y a ,z a ,w a ,l a ,h a ,θ a ), where x gt ,y gt ,z gt w represents the coordinates of the center of the 3D true bounding box. gt ,l gt ,h gt θ represents the width, length, and height of the 3D real bounding box, respectively. gt The x-axis represents the rotation angle of the 3D true bounding box. a ,y a ,z a w represents the coordinates of the center of the 3D anchor frame. a ,l a ,h a θ represents the width, length, and height of the 3D anchor frame, respectively. a This represents the rotation angle of the 3D anchor box. Then, the residual between the 3D anchor box and the ground truth bounding box is calculated. The difference between the 3D anchor box and the 3D ground truth bounding box is calculated using Smooth L1 loss. The learnable parameters of the neural network are updated using stochastic gradient descent. The localization loss is calculated as shown in formula (7):

[0043]

[0044] in, To locate the loss value, Δb is a vector consisting of Δx, Δy, Δz, Δw, Δl, Δh, and Δθ. Δθ=sin(θ gt -θ a ), gt 'a' and 'a' represent the 3D real bounding box and the 3D anchor box, respectively; 'x', 'y', and 'z' are the coordinates of the bounding box center; 'w', 'l', and 'h' are the width, length, and height of the bounding box; 'θ' is the rotation angle of the bounding box; 'd'a Indicates the length of the bottom diagonal of the 3D anchor frame:

[0045] 2) In the process of regressing the rotation angle θ of the bounding box, in order to solve the discontinuity problem caused by periodicity (e.g., 0° and 360° are equivalent), a binary classification method is used to regress the rotation angle, dividing the orientation angle into two categories, front and back, and using orientation loss to optimize it. Specifically, the orientation loss adopts cross-entropy loss, as shown in formula (8):

[0046]

[0047] in, Here, represents the directional loss value, and n represents the total number of samples. It is the predicted directional probability distribution, y′ i It is the true orientation category of the sample.

[0048] 3) Focal Loss is used as the classification loss for updating the 3D anchor box category parameters, as shown in Equation (9):

[0049]

[0050] in, For classification loss value, α′ represents the predicted class probability of the 3D anchor box, α′ is the weighting coefficient that balances positive and negative samples, used to adjust the contribution ratio of positive and negative samples in the loss function, and γ′ is the focusing parameter, used to adjust the weight of easy-to-classify and hard-to-classify samples.

[0051] 4) Finally, the localization loss, orientation loss, and classification loss are weighted and summed to obtain the total loss function, as shown in formula (10):

[0052]

[0053] Where, N pos β1, β2, and β3 represent the number of positive samples in the 3D anchor frame, respectively, and are the weights used to balance the localization loss, orientation loss, and classification loss.

[0054] Beneficial effects of the present invention

[0055] To address the issues of sparsity and inherent noise in 4D millimeter-wave radar point cloud data, this invention effectively fuses features captured by a radar strut feature extraction network and a voxel semantic feature extractor. Then, through clustering and voting operations, object-level target center feature representations are obtained. Next, from a bird's-eye view, the target center features, rich in structural information, are integrated into a semantically rich feature map, thereby utilizing a target detection network to regress and obtain a high-accuracy 3D target bounding box. This multi-feature extraction and fusion method fully utilizes the potential information in 4D millimeter-wave radar sparse point cloud data. While fusing global contextual information and local semantic information, it enhances attention to the details of the target's geometric structure, mitigating the impact of irregular noise in 4D millimeter-wave radar point clouds, and thus effectively improving the accuracy of 3D target detection results based on 4D millimeter-wave radar sparse point clouds. Attached Figure Description

[0056] Figure 1 This is an overall operation flowchart of an embodiment of the present invention;

[0057] Figure 2 This is a flowchart of the point classification-voting process according to an embodiment of the present invention;

[0058] Figure 3 These are example results of bird's-eye view feature maps and fused bird's-eye view feature maps according to embodiments of the present invention;

[0059] Figure 4 This is an example of target detection results for vehicles, pedestrians, and bicycles according to an embodiment of the present invention. Detailed Implementation

[0060] The technical solution of the present invention will be further described below with reference to specific embodiments and accompanying drawings.

[0061] Example

[0062] This invention provides a semantically assisted 3D target detection method based on 4D millimeter-wave radar sparse point clouds. By fusing semantic information from 4D radar point clouds and further integrating structurally dense target center features, the accuracy of 3D target detection is improved. The overall process of the semantically assisted 3D target detection method is as follows: Figure 1 As shown:

[0063] 1) Given a 4D radar point cloud P∈R 5×N The point cloud data is discretized on the XY plane and then divided into uniformly distributed pillars and voxels. Pillar features are extracted using a radar pillar feature extraction network and a voxel semantic feature extractor, respectively. and voxel characteristics Where N is the total number of points in the radar point cloud, C1 is the number of feature channels for pillar features, M1 is the number of non-empty pillars, C2 is the number of feature channels for voxel features, and M2 is the number of non-empty voxels. The radar pillar feature extraction network uses three linear layers with non-shared weights and activation functions to learn the spatial, velocity, and reflection intensity information of each point in the pillar, and uses a multilayer perceptron to integrate the spatial, velocity, and reflection intensity information of each point in the pillar to obtain pillar features, so that the extracted pillar features can better express the global dependencies in the scene. The voxel semantic feature extractor voxels the 4D radar point cloud data and uses sparse convolution on non-empty voxels to extract voxel features, which significantly reduces the computational complexity of voxel feature extraction.

[0064] 2) Integrate pillar features and semantic features to generate a bird's-eye view feature map;

[0065] First, the voxel features corresponding to each original radar point in the given 4D radar point cloud are indexed. Then, the local coordinates of the radar point relative to the voxel center are calculated. Finally, the local coordinates are concatenated with the indexed voxel features to obtain the point features. This facilitates subsequent processing. Then, for each non-empty pillar, query the feature f of k points in its neighborhood of radius r. p Then, feature aggregation is performed on these k feature points to obtain group features. The group features are then fused with the corresponding pillar features to obtain the pillar features with fused semantic features. Finally, the pillar features of the fused semantics are hashed back into the coordinate space of the given 4D radar point cloud to obtain a pseudo-image. Then, sparse embedding convolution is used to generate a bird's-eye view feature map F. b ∈R C×H×W Where C, H, and W represent the number of feature channels, height, and width of the feature map from the bird's-eye view, respectively. Figure 3 As shown in (a).

[0066] 3) Classify, vote on, and cluster the point features to obtain the target center features. The specific process is as follows: Figure 2 As shown;

[0067] For each point feature f pThe algorithm predicts whether a point belongs to the foreground and its target category through a point classification network and a voting network. It also predicts the center coordinates of the target to which the point belongs and its offset from the target center. Specifically, fx is input into the point classification network, which first groups and aggregates features using farthest point sampling (FPS) and ball query methods. Then, it generates virtual points close to the 3D target center using a multilayer perceptron. The voting network learns a Euclidean space offset Δx and a feature offset Δy for each virtual point, making the iteratively learned points closer to the target center. Finally, it performs classification regression on the aggregated features to predict the target center point, and generates object proposals under supervision using cross-entropy loss and focus loss, outputting the classification result and offset from the target center for each point.

[0068] Target center points whose predicted confidence level by the voting network is less than a set threshold are filtered out. Then, through a feature aggregation operation similar to that in S22, the features of all points that voted for the same target center are aggregated to obtain the target center feature F. c ∈R C′×M C′ and M represent the number of feature channels and the number of corresponding center points of the central feature, respectively.

[0069] 4) Use coordinate convolution to extract the target center feature F c Integrated into bird's-eye view feature map F b By designing two additional channels and filling them with constant values ​​to represent untrained coordinate information, the target center feature is concatenated with the bird's-eye view feature map and the two channels. Then, the concatenated features are merged using 1×1 convolution, batch normalization (BN), and ReLU activation to obtain the final fused bird's-eye view feature map. like Figure 3 (b) shows the results. 5) 3D bounding box regression is performed using the fused bird's-eye view feature maps to obtain the final target detection results.

[0070] The result will be the fusion of bird's-eye view feature maps. The data is fed into a 3D object detection network. First, a Region Proposal Network (RPN) is used to generate candidate 3D anchor boxes. Then, feature extraction and classification are performed on each 3D anchor box, and the network parameters are optimized using the loss function defined above to ensure 3D accuracy.

[0071] The accuracy of anchor box positioning, orientation, and category classification. This was achieved through 24 epochs of iterative training.

[0072] The 3D object detection network can learn to identify and locate 3D objects from a fused bird's-eye view feature map. Then, non-maximum suppression (NMS) is used to filter the optimal object detection results, obtaining the object's 3D bounding box parameters, including information such as position, size, and orientation. Figure 4 As shown, the 3D bounding box detection results for pedestrians, bicycles, and cars are represented by blue, green, and purple, respectively.

Claims

1. A semantically assisted 3D target detection method based on sparse point clouds of 4D millimeter-wave radar, characterized in that... The specific steps are as follows: S1: Given a 4D radar point cloud P∈R 5×N The radar pillar feature extraction network and the voxel semantic feature extractor are used to extract pillar features respectively. and voxel characteristics Where N is the total number of points in the radar point cloud, C1 is the number of feature channels for pillar features, M1 is the number of non-empty pillars, C2 is the number of feature channels for voxel features, and M2 is the number of non-empty voxels. S2: Integrate pillar features and semantic features to generate a bird's-eye view feature map; S21: Voxel characteristics Mapping to point features S22: Using the center coordinates of each non-empty support as the center, query k points in a neighborhood with a radius of r, extract the group features of these k points, and add the group features to the corresponding support features to obtain the support features with fused semantic features. S23: Hash the pillar features of the fused semantics back into the coordinate space of the given 4D radar point cloud to obtain a pseudo-image. Then, sparse embedding convolution is used to generate a bird's-eye view feature map F. b ∈R C×H×W Where C, H, and W represent the number of feature channels, height, and width of the feature map from the bird's-eye view, respectively. S3: Classify, vote on, and cluster the point features to obtain the target center features; S31: For each point feature f p The point classification network and voting network are used to predict whether a point belongs to the foreground point and its target category, and at the same time, the center coordinates of the target to which the point belongs and its offset from the target center are predicted. S32: Cluster the point features to the target centers to generate target center features F c ∈R C′×M C′ and M are the number of feature channels and the number of corresponding center points of the central feature, respectively; S4: Integrate target center features into a bird's-eye view feature map to generate a fused bird's-eye view feature map. To increase the number of feature channels in the bird's-eye view feature map; S5: 3D bounding box regression is performed using fused bird's-eye view feature maps to obtain the final target detection result.

2. The semantic-assisted 3D target detection method based on 4D millimeter-wave radar sparse point clouds according to claim 1, characterized in that, In step S1: the radar pillar feature extraction network is used to extract pillar features. The specific method is as follows: based on 4D radar point cloud data, three non-shared weight linear layers and activation functions are used to learn the spatial, velocity and reflection intensity information of each point in the pillar, and a multilayer perceptron is used to integrate the spatial, velocity and reflection intensity information of each point in the pillar to obtain pillar features. Voxel features are extracted using a voxel semantic feature extractor. Specifically, the 4D radar point cloud data is voxelized, and sparse convolution is used on non-empty voxels to extract voxel features.

3. The semantic-assisted 3D target detection method based on 4D millimeter-wave radar sparse point clouds according to claim 1, characterized in that, Step S21 specifically involves: indexing the voxel features of each original radar point in the given 4D radar point cloud, calculating the local coordinates of the radar point relative to the voxel center, and finally concatenating the local coordinates with the indexed voxel features to obtain the point features. The local coordinates of the radar point relative to the voxel center are calculated using formula (1): Among them, c p c represents the local coordinates of the radar point. v Represents voxel coordinates, v s The size of a voxel, v r This represents the boundary of the point cloud data.

4. The semantic-assisted 3D target detection method based on 4D millimeter-wave radar sparse point clouds according to claim 1, characterized in that, The specific steps of step S22 are as follows: For each non-empty pillar, query the feature f of k points in its neighborhood with radius r. p Then, feature aggregation is performed on these k feature points to obtain group features. The group features are then fused with the corresponding pillar features to obtain the pillar features with fused semantic features. The feature fusion process is shown in formula (2): Where Ψ represents a multilayer perceptron, and max(·) represents a max pooling operation. Let i be the feature of the i-th point.

5. The semantic-assisted 3D target detection method based on 4D millimeter-wave radar sparse point clouds according to claim 1, characterized in that, In step S31, during the training of the point classification network and the voting network, the loss function is defined as follows: 1) A point classification network is used to predict whether each point belongs to the foreground and its target category; the point classification network outputs a probability distribution representing the likelihood of the point belonging to each category; specifically, for each point feature f p The point classification network outputs a probability vector. Each element corresponds to the probability of a category; the classification loss function of the point classification network adopts cross-entropy loss, defined as follows: in, Here, y represents the cross-entropy loss value, C is the total number of categories, and y is the cross-entropy loss value. i It is the one-hot encoding of the real label. for The probability of the i-th category; 2) The voting network is used to predict the center point of the target corresponding to each point and its offset from the corresponding target center; the voting network outputs an offset vector. Each element corresponds to an offset in a spatial dimension; the voting loss function of the voting network adopts L1 loss, as shown in formula (4): in, Here, N is the L1 loss value, and t is the number of radar points. i It is the offset from the radar point to the center of the actual target. for The offset of the i-th radar point from the center of the predicted target; 3) To address the class imbalance problem, when the number of positive samples is significantly less than the number of negative samples, FocalLoss is introduced to reduce the weight of easily classified samples and increase the weight of difficult-to-classify samples; the formula is as follows: in, p is the Focal Loss value. t It is the probability that the network correctly predicts the category, α. t It is the weight that balances positive and negative samples, and γ is the focusing parameter; 4) The total loss function is obtained by combining the classification loss function, voting loss function, and Focal Loss. The point classification network and voting network are jointly optimized to accurately predict the target category and offset to the target center for each point. The total loss function is shown in Equation (6):

6. The semantic-assisted 3D target detection method based on 4D millimeter-wave radar sparse point clouds according to claim 1, characterized in that, Step S32 specifically involves: filtering out predicted target center points whose confidence level predicted by the voting network is less than a set threshold, and then aggregating the features of all points that voted for the same target center through a feature aggregation operation to obtain the target center feature F. c ∈R C′×M .

7. The semantic-assisted 3D target detection method based on 4D millimeter-wave radar sparse point clouds according to claim 1, characterized in that, The specific operation of step S4 is as follows: using coordinate convolution to extract the target center feature F c Integrated into bird's-eye view feature map F b By designing two additional channels and filling them with constant values ​​to represent untrained coordinate information, the target center feature is concatenated with the bird's-eye view feature map and the two channels. Then, the concatenated features are merged using 1×1 convolution, batch normalization, and ReLU activation function to obtain a fused bird's-eye view feature map.

8. The semantic-assisted 3D target detection method based on 4D millimeter-wave radar sparse point clouds according to claim 1, characterized in that, In step S5, during the training process of the prediction network for 3D bounding box regression, the loss function is defined as follows: 1) First, define the parameters of the 3D ground truth bounding box and the 3D anchor box; then calculate the residual between the 3D anchor box and the 3D ground truth bounding box, and calculate the difference between the 3D anchor box and the 3D ground truth bounding box using Smooth L1 loss; update the learnable parameters of the neural network using stochastic gradient descent, and calculate the localization loss as shown in formula (7): in, The location loss value; Δb is a vector composed of Δx, Δy, Δz, Δw, Δl, Δh, and Δθ. Δθ=sin(θ gt -θ a ), gt and a respectively represent Table 3D real bounding box and 3D anchor box; x, y, z are the coordinates of the bounding box center; w, l, h are the width, length and height of the bounding box; θ is the rotation angle of the bounding box; d a Indicates the length of the bottom diagonal of the 3D anchor frame: 2) In the process of regressing the rotation angle θ of the bounding box, in order to solve the discontinuity problem caused by periodicity, a binary classification method is used to regress the rotation angle, dividing the orientation angle into two categories, front and back, and using orientation loss to optimize it. Specifically, the orientation loss adopts cross-entropy loss, as shown in formula (8): in, Here, n represents the directional loss value, and n represents the total number of targets. It is the predicted directional probability distribution, y′ i It is the true orientation category of the sample; 3) Focal Loss is used as the classification loss for updating the 3D anchor box category parameters, as shown in Equation (9): in, For classification loss value, α′ represents the predicted class probability of the 3D anchor box, α′ is the weighting coefficient that balances positive and negative samples, used to adjust the contribution ratio of positive and negative samples in the loss function, and γ′ is the focusing parameter, used to adjust the weight of easy-to-classify and hard-to-classify samples. 4) The localization loss, orientation loss, and classification loss are weighted and summed to obtain the total loss function, as shown in formula (10): Where, N pos β1, β2, and β3 represent the number of positive samples in the 3D anchor frame, respectively, and are the weights used to balance the localization loss, orientation loss, and classification loss.

Images