A 4D millimeter wave radar target detection and semantic segmentation model fusing graphs and grids

Abstract

The application discloses a 4D millimeter wave radar target detection and semantic segmentation model fusing a graph and a grid, and belongs to the field of automatic driving.The model comprises a RAMGNN module, obtains specific embedding through two graph neural networks, extracts common embedding based on a shared graph neural network, learns embedding importance weight through an attention mechanism, and fuses to obtain high-dimensional representation of point cloud.A semantic segmentation module outputs prediction results of each point through a classifier.A target detection module comprises a pseudo-image encoder for projecting point cloud to a two-dimensional bird's eye view and gathering point cloud information in each cell to obtain high-dimensional features.A backbone network extracts multi-scale features of input pseudo-image, fuses multi-level and different receptive field information, and regresses and estimates a three-dimensional detection frame of a target to realize effective detection of the target.The application simultaneously pays attention to a topological space and a feature space, fuses information of the two spaces to update node embedding, and the model has high detection precision and robustness.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, specifically to a 4D millimeter-wave radar target detection and semantic segmentation method based on graph neural networks. Background Technology

[0002] Currently, autonomous driving technology, aimed at providing a safe, convenient, and comfortable transportation experience, is developing rapidly. Environmental perception technology is crucial for achieving high-level autonomous driving. Accurate and effective environmental perception technology is a vital foundation for autonomous driving systems to complete tasks such as obstacle recognition, control decisions, and path planning. Among vehicle-mounted environmental perception sensors, millimeter-wave radar is widely used in autonomous driving due to its advantages such as small size, low cost, all-weather operation, strong speed measurement capability, and high distance resolution. However, traditional millimeter-wave radar (2+1D radar) suffers from limitations in signal processing, resulting in poor angular resolution and a lack of altitude measurement capabilities, thus limiting the effectiveness of its information acquisition. The introduction of Multiple Input Multiple Output (MIMO) technology has significantly improved radar angular resolution, thereby significantly enhancing radar point cloud density. This has driven the emergence of 4D millimeter-wave radar, which can provide high-precision distance, speed, azimuth, and elevation angle measurements. Currently, the point cloud density of 4D millimeter-wave radar has reached a level comparable to low-line-count lidar, its data format is similar to lidar point clouds, and it exhibits good environmental perception performance under adverse weather conditions. Based on this, many researchers have focused on transferring depth models used in LiDAR to the 4D millimeter-wave radar domain to achieve tasks such as target detection and scene segmentation. However, due to the sparsity and low signal-to-noise ratio of millimeter-wave radar point clouds, models for processing dense LiDAR point clouds cannot be directly applied to millimeter-wave radar point clouds. Therefore, it is necessary to design targeted models based on the specific characteristics of millimeter-wave radar point clouds.

[0003] Environmental perception methods based on point cloud object detection and semantic segmentation can be mainly divided into the following three categories: point-based, grid-based, and graph-based methods.

[0004] Point-based methods directly process point clouds without requiring formal transformation. PointNet is a pioneering model in this approach, treating the point cloud as a collection and extracting structural features at both global and local levels using a Multi-Layer Perception (MLP) for subsequent classification and segmentation tasks. PointNet++ is an improved version of PointNet, introducing a grouping structure to better capture local information from the point cloud. Schumann et al. were the first to apply PointNet++ to millimeter-wave radar point clouds and conducted semantic segmentation experiments on road scenes based on traditional radar point cloud data they collected. However, the sparse, uneven, and noisy characteristics of radar point clouds prevent the PointNet++ model, which is suitable for dense point clouds, from achieving good results. For target detection tasks, Shi et al. proposed a two-stage model, PointRCNN, for 3D target detection using LiDAR point clouds. In the first stage, PointNet++ is used to extract features from the point cloud to segment foreground and background points, directly generating a small number of high-quality 3D candidate boxes on the foreground points. In the second stage, the obtained candidate boxes are transformed to their custom coordinates and combined with global semantic information of the point cloud to complete target box classification and regression. However, due to the sparsity and non-uniformity of millimeter-wave point clouds, they cannot effectively provide complete spatial information. As a result, point-based target detection models cannot effectively generate high-quality candidate boxes directly on radar point clouds, thus failing to achieve good detection results.

[0005] Mesh-based methods project point clouds onto a 2D bird's-eye view (BEV) or 3D voxels by meshing the scene space. A point cloud feature extraction module then extracts point cloud information from each cell and embeds it into the cell, thus tensorizing the point cloud data. The advantage of this method lies in its ability to effectively extract essential features from the projected point cloud data using deep learning models widely used in image and vision fields, such as Convolutional Neural Networks (CNNs). Modules like Region Proposal Networks (RPNs) are then used for subsequent bounding box classification and regression. This type of method is mainstream in millimeter-wave radar target detection. For example, Palffy et al. pioneered the application of the PointPillars target detection model based on LiDAR point clouds to 4D millimeter-wave radar point clouds for detecting 3D road targets. Experimental results showed that the mesh method is feasible for 4D millimeter-wave radar point clouds. Also based on the PointPillars framework, B. Xu et al. introduced a Pillar Feature Attention (PFA) module, effectively improving the angle estimation accuracy in bounding box regression using a self-attention mechanism. Tan et al. considered the temporal information of multi-frame radar point clouds and introduced a temporal pooling module based on the PointPillars model to aggregate continuous multi-frame radar point cloud information to improve target detection performance. However, grid-based methods typically use a simple point cloud encoder to extract point cloud information within each cell after point cloud projection, focusing only on local point cloud information within the cell and ignoring the topological relationship of the original point cloud, resulting in poor detection performance on sparse millimeter-wave point clouds.

[0006] Graph-based methods construct graphs from the original point cloud and use graph neural networks to achieve node information exchange and feature extraction. W. Shi et al. proposed a detection model for LiDAR point clouds, PointGNN. This model first constructs a graph based on the spatial coordinates of the point cloud and uses a graph neural network to propagate and update node features. Then, it constructs target candidate boxes based on the obtained foreground points and uses the Non-Maximum Suppression (NMS) method for further filtering to achieve box classification and regression. Experiments show that the model has good detection performance. Based on this model, Svenningsson et al. conducted vehicle target detection experiments on the nuScenes dataset. This was the first attempt to use a graph neural network model in the field of millimeter-wave radar, but the results were not good. Felix Fent et al. improved on the PointGNN model and proposed a new node update mechanism. This method achieved good results in semantic segmentation experiments based on the public dataset RadarScenes. It should be noted that the above works are all improvements on models applicable to LiDAR. The processing flow for millimeter-wave radar is similar to that of LiDAR, focusing on point cloud coordinate information. However, the above processing flow cannot achieve good results based on millimeter-wave radar point clouds. This is because dense lidar point clouds can provide obvious contour information, and graph neural networks can obtain excellent target features simply by using point cloud spatial information. However, sparse and non-uniform millimeter-wave radar point clouds cannot provide distinguishable spatial features. Therefore, it is necessary to combine the target dynamics and reflection characteristics (i.e., radar cross section, RCS) to obtain better distinguishable features. Summary of the Invention

[0007] The purpose of this invention is to propose a 4D millimeter-wave radar target detection and semantic segmentation model that integrates graph and mesh information. This model focuses on both topology and feature space and integrates information from both spaces to update node embeddings. The proposed model has high detection accuracy and robustness.

[0008] To achieve the above objectives, this application proposes a 4D millimeter-wave radar target detection and semantic segmentation model that fuses graphs and meshes, comprising:

[0009] The RAMGNN module, or Radar Adaptive Multi-channel Graph Neural Network (RAMGNN), constructs a graph of the original point cloud based on the radar point cloud coordinate information, using the topological space and the initial feature (dynamic and reflection characteristics) space respectively. It uses two message-passing neural networks to propagate and update the node features in the coordinate space and the feature space to obtain two specific embeddings. It also extracts the common embedding between the two spaces based on a shared graph neural network. Based on the attention mechanism, it automatically learns the embedding weights and fuses the two specific embeddings and the common embedding to obtain a high-dimensional representation of the point cloud.

[0010] The semantic segmentation module takes the output features of the RAMGNN module as input and outputs the prediction results of each point through a classifier.

[0011] The target detection module takes the output features of the RAMGNN module as input, projects the point cloud onto a two-dimensional bird's-eye view, and aggregates the point cloud information in each cell to obtain high-dimensional features, converting the point cloud data into a multi-channel pseudo-image. The backbone network extracts multi-scale features based on the pseudo-image and fuses multi-level and different receptive field information in the channel dimension to obtain a fused feature map. Based on the fused feature map, the detection head module regresses and estimates the target's three-dimensional detection box to achieve effective target detection.

[0012] Compared with the prior art, the above technical solution adopted in this invention has the following advantages: The proposed model first constructs a RAMGNN module based on topology and feature map propagation to update point cloud features. Its output node embedding can be directly used for semantic segmentation and also serves as a point cloud feature encoder for subsequent target detection. Then, the point cloud is projected onto a two-dimensional bird's-eye view and its multi-scale features are extracted using a backbone network with a channel attention mechanism. Finally, the multi-scale features are fused to obtain effective target detection. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a diagram illustrating the overall framework of the proposed model.

[0015] Figure 2 Here is the overall framework diagram of the RAMGNN module;

[0016] Figure 3 For GNN s Framework diagram;

[0017] Figure 4 This is a diagram of the backbone network and detection head framework.

[0018] Figure 5 A confusion matrix diagram of the segmentation results of the RAMGNN module on the VoD dataset;

[0019] Figure 6 A comparison chart of some detection results between the proposed model and mainstream object detection models; Detailed Implementation

[0020] The principles of this disclosure will now be described with reference to several exemplary embodiments illustrated in the accompanying drawings. While preferred embodiments of this disclosure are shown in the drawings, it should be understood that these embodiments are described merely to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way.

[0021] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment".

[0022] like Figure 1 As shown, this embodiment provides a 4D millimeter-wave radar target detection and semantic segmentation model that fuses graphs and meshes, which may specifically include:

[0023] The RAMGNN module constructs a graph of the original point cloud based on the topological space and the initial feature space using radar point cloud coordinate information. It uses two message passing neural networks to propagate and update node features in the coordinate space and feature space to obtain two specific embeddings. It also extracts the common embedding between the two spaces based on a shared graph neural network. Based on the attention mechanism, it automatically learns the embedding weights and fuses the two specific embeddings and the common embedding to obtain a high-dimensional representation of the point cloud.

[0024] In one example embodiment, such as Figure 2 As shown: The original point cloud can be represented by the set P = {p1,...,p...} n}, i = 1, ..., n represents point i as p i ={x i ,m i}, x i ∈R 3 Let m be the 3D coordinates of the point cloud. i ∈R 3 The initial features of the point cloud are as shown in equation (1), where rcs is the RCS and v rcTo compensate for the obtained absolute radial velocity, t is the time sequence number of the continuous point cloud frame;

[0025] m i =(rcs,v rc ,t) (1)

[0026] The topology graph can be constructed based on the spatial coordinates of the point cloud using the K-Nearest Neighbor (KNN) algorithm, where each point is connected to the K points with the shortest Euclidean distance. Specifically, using a ij Indicate whether there exists a directed edge from node i to node j such that if node j is one of the K points with the shortest Euclidean distance, then a ij =1; if not, then a ij =0. Therefore, the adjacency matrix A shown in equation (2) can be obtained. s In addition, all nodes retain degree c. si (Number of associated edges) is used as a new feature for this node. This node feature can be represented by h. si =(x i ,m i ,c si This is represented by ) . Simultaneously, information is encoded for each edge, and the spatial coordinate difference between the two endpoints is saved as an initial feature, then e is used. sij =(x j -x i ) represents the edge feature from node i to j. Therefore, combining n node features yields the node feature matrix H shown in equation (3). s Combining all edge features yields the edge feature matrix E shown in equation (4). s Based on the above, the topological graph G s It can be mapped to A s H s and E s The tuple G is composed of s =(A S H s E s )express.

[0027]

[0028] H s =[h s1 … h sn ] T (3)

[0029]

[0030] Where j 11 and j 1k This represents the first and kth nodes connected to node 1, j nkThis represents the Kth node connected to node n.

[0031] The feature map is constructed based on point cloud Doppler and RCS. First, the similarity matrix S∈R between n nodes is calculated based on cosine similarity. n×n Then, the KNN algorithm is used to connect each point to its K nearest neighbors with the highest similarity. Similarly, the adjacency matrix A is used. f Indicates the connection status between nodes; use h fi =(x i ,m i ,c fi ) represents the initial characteristics of node i and e represents the characteristics of node i. fij =(x j -x i ) represents the edge feature from node i to node j, where c fi Let H be the degree of that point; combining n node features yields the node feature matrix H. f Combining all edge features yields the edge feature matrix E. f Finally, use tuple G. f =(A f H f E f ) represents the feature map G f .

[0032] The two message-passing neural networks (GNNs) designed s and GNN f Used to propagate and update the topology graph G s and feature map G f Node embedding, Graph Neural Networks (GNNs) with parameter sharing strategy c This is then used to learn the common embeddings of the two graphs.

[0033] like Figure 3 As shown, the designed GNN s Its input is G s H s and E s First, 3-layer and 2-layer MLPs were used to analyze low-dimensional H... s E s Feature embedding is performed to obtain a high-dimensional representation H as shown in equation (5). s 0 And E as shown in equation (6) s ', where the features of node i are represented by h si 0 Let e ​​represent the edge characteristics from node i to node j. sij 'Use to represent. Then, H s 0 H is obtained through L-layer propagation and updating. sL Here, we will only take node i at level l = 1, ..., L as an example to describe its update process in detail. Node i first obtains the messages propagated by its neighboring nodes through the multilateral system starting from it, that is, through edge e. ij Obtain the l-1 layer features h derived from node i from node j. si l-1 The l-1 layer features h of node j sj l-1 and edge e ij Feature e sij The message z obtained after fusion by the δ function is shown in equation (7). sij l Subsequently, weighted pooling is used to calculate the importance of neighbor messages and the corresponding messages are fused based on the weights to obtain z as shown in equation (8). si l Finally, use Fusion of l-1 layer features h si l-1 and news z si l The output is the embedding h of the lth layer as shown in equation (9). si l Therefore, after the above update process, all nodes can output the feature matrix H of the l-th layer. s L .

[0034] H s 0 =[h s1 0 … h sn 0 ] T (5)

[0035]

[0036] z sij l =δ(h) si l-1 ,h sj l-1 ,e sij ') (7)

[0037]

[0038]

[0039] Where, δ and For MLP, SoftPool learns the importance of neighbors and focuses on useful information. Unlike max pooling, SoftPool is differentiable, thus helping to improve network training performance.

[0040] Similarly, Graph Neural Networks (GNNs) f Using GNN s The same design, H f The above steps can also be used to obtain the feature space-specific embedding H. f L .

[0041] Regarding the extraction of the co-embedding of the two spaces, the designed GNN c A parameter-sharing strategy is used to consider the similarity and correlation between two spaces, and its shared weight parameters can capture common features between the two spaces. GNN c First, based on the topology graph G s and feature map G f It can output two embedded H cs L and H cf L The node feature propagation and update process is similar to that of the GNN described above. s and GNN f The two are identical; then, they are fused and embedded to further extract common features between the two spaces. Common feature fusion methods include addition, multiplication, and concatenation. To obtain a richer feature representation, H is implemented based on the concatenation method. cs L and H cf L The fusion; finally, the spliced ​​embedding is input into the MLP to obtain the two-space co-embedded H c L .

[0042] Based on the above steps, a specific embedding H can be obtained. s L and H f L and co-embedded H c L To obtain highly separable feature representations of nodes, we learn the importance of each embedding based on an attention mechanism. Taking node i as an example, we first use the tanh(·) function to set its embedding h. si L ∈R 1×h A nonlinear transformation is performed to learn a shared attention vector q∈R. h×1 The attention value ω is obtained by multiplying the transformed features by the inner product, as shown in Equation (10). si Similarly, the position of node i in the embedding matrix H can be obtained. f L Attention value ω fi and embedding matrix H c L ω in ciNext, the attention values ​​are normalized using the Softmax function to obtain the final weights, such as weight α. si α can be obtained through equation (11), and similar processing methods can be used to obtain α. fi and α ci Then, combine the weights corresponding to the n nodes, i.e.: α s =[α s1 ,…,α sn ],α f =[α f1 ,…,α fn ],α c =[α c1 ,…,α cn ], and diagonalize it into the vector shown in equation (equationreference goes here); finally, sum the embedding weighted results based on the obtained attention matrix to obtain the final embedding H as shown in equation (13).

[0043] ω si =q T tanh(W(h si L ) T +b) (10)

[0044] Where W∈R h×h It is a weight matrix, b∈R h×1 It is a bias.

[0045]

[0046] a S =diag(a s ),a F =diag(a f ),a C =diag(a c (12)

[0047] H = α S H s L +α F H f L +α C H c L (13)

[0048] In summary, the RAMGNN encoder utilizes a topological space constructed based on radar point cloud coordinate information, an initial feature space (dynamic and reflection characteristics), and a combination of both to learn node embeddings, providing highly separable features for subsequent semantic segmentation and object detection. Specifically, it first constructs two structurally distinct graphs based on the radar point cloud: one based on coordinate information to represent the point cloud topological space; and the other based on the point cloud Doppler and RCS to represent the initial feature space. Then, it uses two specifically designed graph neural networks (GNNs)... s and GNN f Propagate updated node information to learn two specific embeddings H s L and H f L Considering the correlation between point clouds in two spaces (e.g., all points of a target have adjacent coordinates and similar velocity and reflection characteristics), a GNN with a parameter-sharing strategy is designed. c Learning the common embedding of point clouds in two spaces H c L Finally, an attention mechanism is used to automatically learn the importance weights of the three embeddings and adaptively fuse them based on the obtained weights to obtain a high-dimensional effective embedding H of the point cloud.

[0049] The semantic segmentation module includes a three-layer MLP that takes the high-dimensional embedding H output by the RAMGNN module as input and outputs features Y. pcls ∈R n×m The point cloud confidence score is output via the Softmax function, and the category of each point depends on the highest confidence score.

[0050] The target detection module takes the output features of the RAMGNN module as input, projects the point cloud onto a two-dimensional bird's-eye view, and aggregates the point cloud information in each cell, converting the point cloud data into a multi-channel pseudo-image. The backbone network extracts multi-scale features based on the pseudo-image and fuses multi-level and different receptive field information in the channel dimension to obtain a fused feature map. Based on the fused feature map, the target detection is achieved by regressing and estimating the target's three-dimensional detection box through the detection head module.

[0051] Specifically, considering the sparse and non-uniform characteristics of millimeter-wave radar point clouds, using 3D projection of radar point clouds will result in a large number of empty voxels, leading to poor detection performance. Therefore, this invention uses a 2D projection method to realize the transformation of point clouds into a two-dimensional bird's-eye view, that is, each cell gathers all point clouds in its airspace, and the non-empty cells containing point clouds are called pillars.

[0052] Let P be the total number of pillars, and N be the maximum number of point clouds that each pillar can hold. If the number of point clouds is less than N, it is filled with zero values; if it is greater than N, N points are randomly sampled. Thus, the sparse data obtained by point cloud projection can be represented by tensor (C,P,N), where C is the feature dimension of the point cloud, i.e., the number of columns of the high-dimensional effective embedding H obtained based on the RAMGNN module.

[0053] Then, based on the MaxPool, all point cloud information within the pillar is aggregated to obtain a tensor (C, P), while empty cells are padded with zeros to obtain a tensor (C, Num). cell -P), where Num cell Let (C, P) represent the total number of elements after point cloud projection, and record (C, Num) and (C, Num) based on indexing. cell The positions of each element in the 2D bird's-eye view (-P).

[0054] Finally, based on the above indexes, combining (C,P) and (C,Num) cell -P) yields the pseudo-image (C,H,W), where H and W are the height and width of the pseudo-image, respectively, and H×W=Num cell C represents the number of pseudo-image channels.

[0055] In summary, the pseudo-image encoder takes the high-dimensional embedding H output by RAMGNN as input, projects it onto a two-dimensional bird's-eye view, and aggregates the point clouds in each cell to obtain the high-dimensional representation of that cell. This allows for the data conversion from point cloud data to pseudo-images (three-dimensional tensors).

[0056] In another embodiment, the designed backbone network is as follows: Figure 4 As shown, it consists of two parts: a feature extraction network for extracting multi-level information from images and a multi-scale feature map upsampling and feature fusion module;

[0057] The feature extraction network includes a multi-layer squeeze-and-excitation residual network (SE-ResNet), which takes a pseudo-image (C, H, W) as input and gradually extracts feature information at different levels of the image through multiple convolutional layers, pooling layers, and residual structures, outputting multiple feature maps of different scales. The shallow SE-ResNet layer focuses on low-level image features (such as edges and textures), and its output low-level feature maps have a small receptive field, capable of capturing local information and small targets. The deep SE-ResNet layer focuses on high-level semantic features (such as overall contours), and its output high-level feature maps have a large receptive field, capable of focusing on global information and larger targets. Furthermore, SE-ResNet is used instead of common CNNs or ResNet because this module can adaptively select and focus on important feature channels, thereby improving the ability to represent features at different levels.

[0058] The upsampling and feature fusion module first upsamples multiple feature maps of different scales output by the feature extraction network to the same scale. Upsampling increases the resolution of small-scale feature maps, thereby improving target detection accuracy, while maintaining the same scale facilitates subsequent multi-feature map fusion. Next, multiple feature maps are concatenated along the channel dimension to fuse multi-level and different receptive field information to obtain a fused feature map as input to the subsequent detection head, thereby improving target detection performance.

[0059] This invention uses the detection module of SSD as the detection head. This detection head takes the fused feature map output by the backbone network as input and outputs the classification confidence of each cell and the regression parameter estimate. Each pixel cell in the fused feature map is assigned D×2 prior boxes of different sizes, where D is the number of target categories. Each category includes two heading angles, 0° and 90°, to facilitate the angle regression of the detection boxes, thereby achieving target 3D detection box classification and regression.

[0060] Design the total loss function of the proposed model; construct the function for H based on equation (14). cs L Correlation matrix C s Similarly, constructing information about H cf L Correlation matrix C f Then, the L2 norm square of the difference between the two, as shown in equation (15), is used to characterize H. cs L and H cf L Similarity, i.e.: embedding H cs L and H cf L The more similar the learned features, the better. c The smaller.

[0061]

[0062] Among them, v s ∈R 1×h For H cs L The mean vector.

[0063]

[0064] Semantic segmentation is essentially a multi-objective classification problem, therefore its loss can be represented by the cross-entropy function, i.e.:

[0065]

[0066] Where μ = [μ0,...μ] m-1 ], μ i Let y = [y0, ..., y1] represent the probability that a sample belongs to the i-th class.m-1 ] represents the one-hot encoding of the sample label. When the sample belongs to the i-th class, y i =1, otherwise 0, where m is the number of sample categories.

[0067] Object detection loss: Regarding the regression loss of the detection boxes, let the ground truth box and the prior box be defined by (x,y,z,w,l,h,θ), where (x,y,z) are the center coordinates of the box, (w,l,h) are the width, length, and height, and θ is the yaw angle. Then the regression bias between the ground truth box and the prior box can be expressed as follows:

[0068]

[0069]

[0070] Δθ=θ gt -θ a (19)

[0071] Where gt and a represent the truth box and prior box, respectively.

[0072] Therefore, the bounding box regression loss can be expressed based on the SmoothL1 function as follows:

[0073] L loc =∑ Δb∈(Δx,Δy,Δz,Δw,Δl,Δh,Δθ) SmoothL1(Δb) (20)

[0074] The SmoothL1 function is defined as shown in equation (21). The reason for using the SmoothL1 function instead of the L1 or L2 norm is that the SmoothL1 has better convergence than the L1 norm and higher stability than the L2 norm.

[0075]

[0076] Since FocalLoss can effectively overcome the model bias problem caused by the imbalance of positive and negative samples, a detection box classification loss is constructed based on FocalLoss, namely:

[0077] L cls =-(1-p t ) γ log(p t ) (twenty two)

[0078] Where p t γ is the predicted class probability value of the model shown in equation (23), and γ is a constant used to control the positive and negative imbalance of the samples.

[0079]

[0080] Based on this, the following target detection loss function can be constructed:

[0081]

[0082] Where N pos β represents the number of positive detection boxes during training. cls and β loc These are the weighting coefficients.

[0083] Based on the above, the consistency constraint L can be obtained. c and target detection loss L od Therefore, the total loss of the model can be constructed as follows:

[0084]

[0085] Where, γ c γ ss and γ od These are the weighting coefficients.

[0086] In summary, since the proposed model possesses both semantic segmentation and object detection capabilities, its loss function comprises two types of costs: the point cloud classification loss from the semantic segmentation module and the regression loss, classification loss, and angle regression loss from the detection boxes in the object detection module. Furthermore, for the shared neural network GNN in RAMGNN... c The resulting embedded H cs L and H cf L This invention constructs consistency constraints to improve the similarity between the two.

[0087] The effects of this invention can be further illustrated by the following simulations:

[0088] Simulation conditions: This experiment is based on the PyTorch deep learning framework, with an Intel Core™ i7-7700 processor (3.60GHz), 32GB of RAM, and an NVIDIA GeForce GTX 1070 graphics card. (Figure G) s And Figure G f Constructing a graph neural network (GNN) with K=15. s GNN f and GNN c The number of layers is set to 4, and the number of neurons in each layer is set to (224, 224, 128, 64); the number of categories output by the semantic segmentation head is m = 4; based on the experimental settings on the VoD dataset, the effective scene range of all point cloud data is clipped to R. x =[0,51.2],R y =[-25.6,25.6],R z= [-3,2], in meters, the same below; set the pillar's length, width, and height to [0.16, 0.16, 5], the maximum number of point clouds it can hold N = 10, and the pseudo-image size to (C, H, W) = (64, 320, 320); set three modules in the backbone network, each composed of two SE-ResNet networks, with parameters (64, 128, 256) for each module; set the prior box size (l, w, h) based on the VoD annotation truth box size as follows: Car class (3.9, 1.6, 1.56), Pedestrian class (0.8, 0.6, 1.74), and Cyclist class (1.76, 0.6, 1.73), with each cell's prior box containing two heading angles, 0° and 90°; set the target detection loss weight coefficient to γ ​​= 2.0, β cls =1.0,β loc =2.0 means the total loss weight coefficient of the model is set to γ. c =0.01,γ ss =0.5,γ od =2.0. The initial learning rate is 0.003, and the training learning rate is gradually reduced to 10%. Stochastic gradient descent with momentum of 0.9 is used as the optimization function, and the batch size is 8.

[0089] Simulation content:

[0090] Simulation 1: Based on the publicly available 4D millimeter-wave radar point cloud dataset View-of-Delft (VoD), this invention verifies the effectiveness of the proposed model by comparing it with mainstream semantic segmentation models based on millimeter-wave radar (PointNet, PointGNN, and RadarGNN models) in various aspects. Semantic segmentation quality is typically characterized by the following F1 score: Where Precision = TP / (TP+FP) is the accuracy rate, and Recall = TP / (TP+FN) is the recall rate. TP stands for True Positive, which is the number of point clouds correctly classified; FP stands for False Positive, which is the number of point clouds that were misclassified as belonging to the correct class but actually belong to another class; and FN stands for False Negative, which is the number of point clouds that were misclassified as belonging to the correct class but actually belong to the correct class.

[0091] As shown in Table 1, the sparse, uneven, and noisy characteristics of radar point clouds prevent the PointNet++ model, which is suitable for dense point clouds, from achieving good results, resulting in a low F1 score. PointGNN performs second best, as it fails to fully utilize the information of the starting node during node updates, leading to poor performance. RadarGNN improves the node update mechanism based on PointGNN, thus achieving a higher score; however, its point cloud mapping ignores the initial feature space of the radar point cloud. The proposed model focuses on both topology and feature space in its point cloud mapping and integrates information from both spaces to update node embeddings, thus achieving a higher score compared to the comparison algorithms, demonstrating the effectiveness of the proposed semantic segmentation model.

[0092] Table 1 shows the F1 scores of the RAMGNN encoder and mainstream semantic segmentation methods based on the VoD dataset.

[0093]

[0094] The target segmentation results of the proposed model are as follows: Figure 5 Confusion matrix representation. Figure 5 Each column and row in the table represents the target segmentation prediction result and corresponding true category of the proposed model. The elements on the main diagonal represent the classification accuracy of that target, and the rest represent the classification error rate. Figure 5 The proposed model performs well in segmenting Cyclist targets with an accuracy of 84.6%. This is because Cyclist targets have highly separable dynamic features, meaning they exhibit significant speed differences compared to other target classes. The accuracy for Pedestrian targets is 65.8%. Due to their reflectivity and geometric features, the point clouds of Pedestrian targets are relatively sparse and easily predicted as background. The prediction performance for Car targets is poor because Car target point clouds are usually fragmented and difficult to coalesce into a single target, resulting in poor segmentation quality.

[0095] Simulation 2: Based on the publicly available 4D millimeter-wave radar point cloud dataset View-of-Delft (VoD), this invention verifies the effectiveness of the proposed model by comparing it with mainstream millimeter-wave radar-based target detection models (PointPillars, RPFA-Net, and RadarMFNet). The evaluation metrics used are Average Precision (AP) and Average Orientation Similarity (AOS) from the KITTI dataset benchmark. AP is a commonly used evaluation metric in target detection, measuring the model's precision and recall. Specifically, the area enclosed by the PR curve (recall on the horizontal axis and precision on the vertical axis) represents AP. AOS measures the angular similarity between the detection bounding box and the ground truth box. Furthermore, this invention sets the Intersection Over Union (IOU) ratio for the detection box and the truth box, namely: 0.5 for the Car class and 0.25 for the Pedestrian and Cyclist classes.

[0096] Figure 6 The target detection results are presented for four different comparison models. Figure 6 (a) shows the ground truth images, ground truth boxes, and millimeter-wave point cloud projection images of the three types of targets. Figure 6 (b) is the target 3D truth box. Figure 6 (cf) shows the detection results of each comparison method and the proposed model. Figure 6 As shown in (c) and 6(d), PointrRCNN and PointPillars have poor detection accuracy. This is because these two methods extract point cloud features based on the PointNet++ module, while PointNet++, which focuses on the complete spatial information provided by dense point clouds, cannot achieve good performance based on sparse millimeter-wave point clouds. Furthermore, PointrRCNN suffers from severe false detections because it cannot effectively generate and merge high-quality candidate boxes directly from radar point clouds. Figure 6 (e) shows that RadarMFNet, by introducing a temporal pooling module based on PointPillars, effectively improves the overall detection box regression accuracy; however, some false negatives and missed detections still exist. Figure 6 (f) shows that the proposed model can effectively extract important information of point cloud topology and feature space and obtain highly separable embedding representations, and the backbone network can effectively fuse multi-scale information, thus enabling relatively complete target detection. Moreover, it has a larger crossover ratio than the comparison method. Therefore, the proposed model has high detection accuracy and robustness.

[0097] The AP and AOS obtained by the proposed model and the comparison models are shown in Table 2 and Table 3, respectively. Bold text in the tables indicates the maximum AP and AOS, and underlined text indicates the second-highest value. As shown in Tables 2 and 3, for the Cyclist class, both the proposed model and the comparison algorithm achieve high accuracy. This is because the Cyclist class target point cloud provides highly separable dynamic information. The proposed model achieves 3D and BEVAP of 74.16 and 77.22, respectively, which are superior to the comparison algorithm. For the Pedestrain class, the accuracy of both the proposed model and the comparison algorithm is lower. This is because the dataset labels a large number of distant Pedestrain class targets, and their few reflection points make them difficult to distinguish from background points. PointPillars performs well in detecting Pedestrain class targets, which is related to its deeper backbone network, which can capture richer multi-level information. Similar to the Pedestrain class, the proposed model and the comparison algorithm perform poorly for the Car class. This is because the dataset labels a large number of static vehicles, making it difficult to effectively distinguish them from static targets on both sides of the road. Furthermore, the proposed model exhibits high AP and AOS for various targets. The mAP (average AP for each target type) for 3D and BEV targets is 51.18 and 58.43, respectively, while the mAOS (average AOS for each target type) is 43.22, both of which are the highest among all models, indicating that the proposed model has high detection accuracy and robustness.

[0098] Table 2 shows the AP (Average Target) scores obtained by the proposed model and mainstream object detection methods based on the VoD dataset;

[0099]

[0100] Table 3 shows the AOS (Average Search Count) of the proposed model and mainstream object detection methods based on the VoD dataset.

[0101]

[0102] Simulation 3: To verify the effectiveness of the RAMGNN module, detection experiments were conducted by replacing the point cloud encoder while keeping the rest of the components unchanged, to study the impact of the module on target detection. The results are shown in Table 4. Table 4 shows that the target detection results obtained based on the RAMGNN module are 3D mAP 51.18, BEV mAP 58.43, and mAOS 43.22, which are superior to those based on the PointNet++ encoder (a point-based method) and the PointGNN encoder (a graph-based method). This indicates that the constructed RAMGNN encoder can effectively acquire highly separable features of millimeter-wave point clouds.

[0103] Table 4 shows the performance of the proposed model on the VoD dataset based on each encoder.

[0104]

[0105]

[0106] Simulation 4: To verify the effectiveness of the target detection framework designed by fusing the graph and the grid, it was compared with the point-based method PointRCNN, the grid-based method PointPillars, and the graph-based method RAMGNN+RPN. The results are shown in Table 5. Table 5 shows that the proposed model obtains highly separable embedding representations based on the graph, and projects the obtained embedding representation point cloud onto a two-dimensional bird's-eye view based on the grid. This allows the backbone network with channel attention to extract and fuse multi-level information, thereby achieving effective target detection. This demonstrates that the proposed target detection framework has good detection performance.

[0107] Table 5 shows the performance of the proposed model and the comparison method on the VoD dataset.

[0108]

[0109] Simulation results show that the proposed model achieves an F1 score of 60.2 based on the VoD dataset, which is better than mainstream millimeter-wave radar point cloud segmentation models such as PointNet++, PointGNN, and RadarGNN in terms of target segmentation performance. The proposed model also achieves 3D mAP51.18, BEV mAP58.43, and mAOS43.32, which are better than mainstream millimeter-wave radar point cloud detection models such as PointPillars, RPFA-Net, and RadarMFNet in terms of target detection performance.

[0110] The above description is merely an optional embodiment of this disclosure and is not intended to limit this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

[0111] Although the claims in this application have been formulated for specific combinations of features, it should be understood that the scope of this disclosure also includes any novel feature or any novel combination of features, whether express or implied or generalized herein, whether or not it relates to the same scheme in any of the claims currently claimed.

Claims

1. A 4D millimeter-wave radar target detection and semantic segmentation model that fuses graphs and meshes, characterized in that, include: The RAMGNN module constructs a graph of the original point cloud based on the topological space and the initial feature space using radar point cloud coordinate information. It uses two message passing neural networks to propagate and update node features in the coordinate space and feature space to obtain two specific embeddings. It also extracts the common embedding between the two spaces based on a shared graph neural network. Based on the attention mechanism, it automatically learns the embedding weights and fuses the two specific embeddings and the common embedding to obtain a high-dimensional representation of the point cloud. The semantic segmentation module takes the output features of the RAMGNN module as input and outputs the prediction results of each point through a classifier. The object detection module takes the output features of the RAMGNN module as input, projects the point cloud onto a two-dimensional bird's-eye view, and aggregates the point cloud information in each cell, converting the point cloud data into a multi-channel pseudo-image. The backbone network extracts multi-scale features based on the pseudo-image and fuses multi-level and different receptive field information in the channel dimension to obtain a fused feature map. Based on the fused feature map, the target detection is achieved by regressing and estimating the target's 3D detection box through the detection head module; The original point cloud is constructed using radar point cloud coordinate information based on both topological space and initial feature space, specifically as follows: Original point cloud using ensemble It means that among them Point for , For the three-dimensional coordinates of the point cloud, The initial features of the point cloud are as shown in equation (1). For RCS, To compensate for the obtained absolute radial velocity, The time sequence number of the continuous point cloud frames; (1) The topology graph is constructed using the K-nearest neighbor algorithm based on the spatial coordinates of the point cloud, that is, each point is located at the point with the shortest Euclidean distance to its neighbor. Connect the points; specifically, use Indicates whether there is a slave node. To the node A directed edge, if node It has the shortest Euclidean distance. One of the points, then If not, then ; Obtain the adjacency matrix as shown in equation (2). All nodes retain their degree. As a newly added feature at this point, this node feature is used This indicates that each edge is encoded, and the spatial coordinate difference between the two endpoints is saved as an initial feature. Represents a node arrive Edge features; combination The node feature matrix is ​​obtained from the node features as shown in equation (3). Combine all edge features to obtain the edge feature matrix shown in equation (4). Based on the above, the topology graph Mapped to by , and The tuples formed ; (2) (3) (4) in and This indicates that node 1 connects the first and second nodes. 1 node Represents a node The first connection One node; Feature maps are constructed based on point cloud Doppler and RCS: First, cosine similarity is calculated. Similarity matrix between nodes Using the KNN algorithm to connect each point with its corresponding data point Connect the nearest neighbors with the highest similarity; use an adjacency matrix. Indicates the connection status between nodes; using Represents a node initial features and use Represents a node To the node The edge features, where The degree of that point; combination Obtain the node feature matrix from the features of each node. Combining all edge features yields the edge feature matrix. Finally, use tuples. Representation of feature map .

2. The 4D millimeter-wave radar target detection and semantic segmentation model based on fused graphs and meshes as described in claim 1, characterized in that, Two message-passing neural networks are used to propagate and update node features in the coordinate space and feature space to obtain two specific embeddings. A shared graph neural network is then used to extract the common embedding between the two spaces. Specifically, this involves using a graph neural network... and Propagate and update the topology graph and feature map Node embeddings and graph neural networks with parameter sharing strategies This is then used to learn the common embeddings of the topological graph and the feature graph; For graph neural networks Its input is a topology graph. Node feature matrix Sum of edge feature matrices First, we use 3-layer MLP and 2-layer MLP to process the low-dimensional node feature matrix. Edge feature matrix Feature embedding is performed to obtain a high-dimensional representation as shown in equation (5). and as shown in equation (6) , where nodes Features denoted by , node To the node The edge features are then used Used to indicate; then go through Layer propagation and updates to obtain The update process is as follows: node First, the messages propagated by neighboring nodes are obtained through a multi-edge approach, i.e., via edges. From node Get from nodes of Layer features ,node of Layer features Sum of edge features go through The message obtained by function fusion is shown in equation (7). Subsequently, the importance of neighbor messages is calculated using weighted pooling, and the corresponding messages are fused based on the weights to obtain the message shown in Equation (8). Finally, use Fusion Layer features and news The output is as shown in equation (9). Layer embedding Therefore, all nodes output the first value after the above update process. Feature matrix of layer : (5) (6) (7) (8) (9) in, and For MLP, For SoftPool, learn the importance of neighbors and focus on useful information; Graph Neural Networks With graph neural networks The design is the same, and the node feature matrix is ​​identical. The above steps yield a feature space-specific embedding. ; Graph Neural Networks First, based on the topology graph and feature map Output two embeddings and The node feature propagation and update process is similar to that of the graph neural network described above. Graph Neural Networks Same; then merge the two embeddings. and To extract common features between the two spaces; the two fused embeddings are then input into an MLP to obtain the common embeddings between the two spaces. .

3. The 4D millimeter-wave radar target detection and semantic segmentation model based on fused graphs and meshes as described in claim 2, characterized in that, The system automatically learns embedding weights based on an attention mechanism and fuses two specific embeddings and a common embedding to obtain a high-dimensional representation of point clouds. Specifically, it utilizes... Function embedding Perform nonlinear transformations to learnable shared attention vectors The attention value is obtained by multiplying the transformed features by the inner product, as shown in Equation (10). ; and thus obtain the node In the embedding matrix Attention value in and embedding matrix Attention value in Next, the attention values ​​are normalized using the Softmax function to obtain the weights shown in Equation (11); then, they are combined. Each node corresponds to a weight, that is: And diagonalize it into the vector shown in Equation (12); finally, sum the embedding weighted results based on the obtained attention matrix to obtain the final embedding as shown in Equation (13). : (10) in, It is a weight matrix. It is a bias; (11) (12) (13)。 4. The 4D millimeter-wave radar target detection and semantic segmentation model based on fused graphs and meshes as described in claim 3, characterized in that, The semantic segmentation module includes a three-layer MLP, which outputs a high-dimensional embedding from a RAMGNN module. Input, output features The point cloud confidence score is output via the Softmax function, and the category of each point depends on the highest confidence score.

5. The 4D millimeter-wave radar target detection and semantic segmentation model based on fused graphs and meshes as described in claim 1, characterized in that, The point cloud is projected onto a 2D bird's-eye view and the point cloud information within each cell is aggregated. The point cloud data is then converted into a multi-channel pseudo-image, specifically: set up The total number of pillars is [number], and the maximum number of point clouds that each pillar can hold is [number]. If the number of point clouds is less than If it is greater than 0, then fill it with zero; if it is greater than 0, then fill it with zero. Then random sampling Therefore, the sparse data obtained from point cloud projection is composed of tensors. It means that, among them The point cloud feature dimension is the high-dimensional embedding obtained based on the RAMGNN module. The number of columns; Tensors are obtained by aggregating all point cloud information within the pillar using MaxPool. Simultaneously, empty cells are padded with zeros to obtain tensors. ,in This represents the total number of elements after point cloud projection, recorded using an index-based method. and The position of each element in the two-dimensional bird's-eye view; Based on the above index combination and Obtaining pseudo-images ,in and These are the height and width of the pseudo-image, respectively. , This represents the number of pseudo-image channels.

6. The 4D millimeter-wave radar target detection and semantic segmentation model based on fused graphs and meshes as described in claim 1, characterized in that, The backbone network extracts multi-scale features from pseudo-images and fuses multi-level and different receptive field information along the channel dimension to obtain a fused feature map, specifically: The backbone network is designed as a feature extraction network for extracting multi-level information from images and a multi-scale feature map upsampling and feature fusion module. The feature extraction network includes a multi-layer compressed excitation residual network SE-ResNet, using pseudo-images. As input, it gradually extracts feature information at different levels of the image through multiple convolutional layers, pooling layers and residual structures and outputs multiple feature maps at different scales; its shallow SE-ResNet focuses on low-level image features, while the deep SE-ResNet focuses on high-level semantic features. The upsampling and feature fusion module first upsamples multiple feature maps of different scales output by the feature extraction network to the same scale; then, it concatenates multiple feature maps along the channel dimension to fuse multi-level and different receptive field information to obtain a fused feature map.

7. The 4D millimeter-wave radar target detection and semantic segmentation model based on fused graphs and meshes as described in claim 1, characterized in that, Based on the fused feature map, effective target detection is achieved by regressing and estimating the 3D detection box of the target through the detection head module. Specifically, the detection module of SSD is used as the detection head, which takes the fused feature map output by the backbone network as input and outputs the classification confidence of each cell and the regression parameter estimate; each pixel cell in the fused feature map is set... Prior boxes of different sizes It represents the number of target categories, with each category containing two heading angles: 0° and 90°.

8. The 4D millimeter-wave radar target detection and semantic segmentation model based on fused graphs and meshes according to claim 2, characterized in that, Design a loss function to achieve forward propagation and gradient update: Construct a function based on the embedding of equation (14). Correlation matrix and about embedding Correlation matrix ; Using the difference between the two as shown in equation (15) Norm square representation and Similarity; (14) in, For about The mean vector; (15) The semantic segmentation loss is represented by the cross-entropy function, i.e.: (16) in , Indicates that the sample belongs to the first The probability of a class One-hot encoding for sample labels, when the sample belongs to the first... Class Time Otherwise, it is 0. The number of sample categories; Object detection loss: Regression loss for the detection boxes, where the ground truth boxes and prior boxes are calculated by... Define, where The coordinates of the center of the box The dimensions are width, length, and height. Let yaw angle be the regression angle. The regression bias between the truth box and the prior box is expressed as follows: (17) (18) (19) in, and These represent the truth box and the prior box, respectively. ; Therefore, the bounding box regression loss is based on The function is represented as follows: (20) in The function is defined as shown in equation (21): (21) The bounding box classification loss is constructed based on Focal Loss, namely: (22) in The model shown in equation (23) predicts the class probability value. It is a constant used to control the positive and negative imbalance of the samples; (23) Construct the following target detection loss function: (24) in This represents the number of positive sample detection boxes during training. and These are the weighting coefficients; Therefore, the total loss of the model is as follows: (25)。

Images