Radar Target Data Association Method Based on Graph Neural Network

Abstract

The present invention relates to a radar target data association method based on a graph neural network, which includes the following steps: performing multi-dimensional feature extraction on radar point tracks and AD data; constructing graph data: using the extracted multi-dimensional features, target tracks, and point tracks to be associated to form graph data; implementing a graph neural network: including performing graph neural network edge update, performing graph neural network node update, performing graph neural network global variable update, and graph neural network edge classification. Different from traditional expert rule-based methods, the present invention is data-driven and can learn data association criteria using labeled samples. The present invention is an end-to-end architecture that models radar target data association as an edge classification problem in a graph data structure and uses an advanced graph neural network to solve the graph model in one go.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and particularly to a radar target data association method based on a graph neural network. Background Art

[0002] Target data association is the most critical technology for radar multi-target tracking. Currently, typical data association methods include Global Nearest Neighbor (GNN), Joint Probabilistic Data Association (JPDA), and Multiple Hypothesis Tracking (MHT). GNN models data association as a minimum-cost job assignment problem, solves this problem using the Hungarian algorithm, and gives a deterministic association result of track-points. JPDA calculates the probability of association events and gives a probabilistic association result of track-points. JPDA has a higher data association accuracy rate than GNN, but also has a higher computational complexity. MHT describes the track and point association problem as a tree structure, and determines the association relationship through methods such as delayed decision-making and track pruning. Compared with GNN and JPDA, MHT has the best tracking effect and the highest computational complexity. The main characteristic of traditional radar target data association is that engineers use the method of forward deduction to strive to deduce a mathematical model that is closest to the objective world, and the quality of the model completely depends on the designer's own cognition.

[0003] Some experts and scholars have also applied deep learning and neural networks to the research of radar target data association technology. A method for tracking a vehicle in front by fusing a millimeter-wave radar and deep learning vision (Patent Publication No.: CN109459750A) combines the advantages of millimeter-wave radar and vision deep learning, effectively improving the accuracy and robustness of tracking a vehicle in front. However, when this method performs data association, it does not utilize the global information formed by the topological structure of detection points and tracks, and there is room for further improvement in the algorithm performance.

[0004] A system and method for multi-target tracking and autofocus based on deep machine learning and lidar (Patent Publication No.: CN109479088A) discloses a system and method for identifying, tracking, and focusing on moving targets. According to the disclosed embodiments, the system and method can identify and track moving targets moving relative to an imaging device, and determine the distance from the imaging device to the moving target. The tracking framework of this invention adopts a traditional architecture including track initiation, data association, and tracking filtering. How to explore and utilize an end-to-end architecture to achieve multi-target tracking requires further research. Summary of the Invention

[0005] To solve the existing technical problems, the present invention provides a radar target data association method based on a graph neural network.

[0006] The specific content of the present invention is as follows: A radar target data association method based on a graph neural network includes the following steps:

[0007] Extract multi-dimensional features from radar point tracks and AD data;

[0008] Construct graph data: Use the extracted multi-dimensional features, target tracks, and point tracks to be associated to form graph data;

[0009] Implement a graph neural network: including performing edge updates of the graph neural network, performing node updates of the graph neural network, performing global variable updates of the graph neural network, and edge classification of the graph neural network.

[0010] Furthermore, use the labeled samples to train the graph neural network model and automatically learn the radar target data association criteria;

[0011] Input the point tracks to be tested into the graph neural network model to complete radar multi-target tracking.

[0012] Furthermore, the multi-dimensional features extracted from the detected radar point tracks and AD data include target amplitude envelope features, phase features, Doppler features, signal-to-noise ratio, RCS (radar cross section), target distance dimension, azimuth dimension, and elevation dimension.

[0013] Furthermore, in constructing the graph data, the nodes of the graph include target tracks and point tracks to be associated, the edges of the graph are connected by the target tracks and the point tracks to be associated, and the connection weights of the edges represent the corresponding association relationships.

[0014] Furthermore, the input of the graph neural network edge update is the target track, target point track, edge before update, and global variable, and the output is the edge after update. This module includes two fully connected layers and a ReLU function.

[0015] Furthermore, the graph neural network node update updates the detected node features by condensing the historical features into the detected node features. The input is the target track, target point track, updated edge, and global variable, and the output is the updated target point track.

[0016] Furthermore, the graph neural network global variable update uses the connection information of the target track and target point track to mine the global information of the graph structure. The input is the global variable before update, aggregated nodes and edges, and the output is the global variable after update.

[0017] Furthermore, the input of the graph neural network edge classification is the target track, updated target point, updated edge, and updated global variable, and the output is the final edge classification result, that is, the association relationship between the track and the point track.

[0018] Further, the formation process of the training samples is as follows: Match the true value data with the radar traces, and use the matching results to extract information such as target amplitude envelope features, phase features, Doppler features, signal-to-noise ratio, RCS, target distance size, azimuth size, and pitch size from the AD data. Then, balance the samples and normalize the features to form the training samples.

[0019] Further, the use of the graph neural network model includes:

[0020] Graph feature extraction: Use the radar traces to extract target features from the echo data, including target amplitude envelope features, phase features, Doppler features, signal-to-noise ratio, RCS, target distance size, azimuth size, and pitch size information. Normalize the features to form the node features of the graph data.

[0021] Graph data construction: Use the previous moment's track as the track node, the currently detected trace as the trace node, and the connection between the track and the trace as the edge of the graph data to form the graph data to be solved. Input it into the graph neural network to classify the edges of the graph data.

[0022] Association relationship reasoning: First, the graph data flows through the graph neural network edge update module to update the graph data edges. Then, the graph data flows through the graph neural network node update module to update the graph data trace node features. Then, the graph data flows through the global variable update module to update the global variable representing the connection relationship of the graph data. Finally, the graph data flows through the graph neural network edge classification module to output the association relationship result.

[0023] Track filtering output: After determining the association relationship, perform Kalman filtering on the points belonging to the same track, estimate information such as target position, target speed, target heading, and target acceleration, mark the same batch number, and output the tracking track.

[0024] The radar target data association method based on graph neural network of the present invention is different from the traditional expert rule-based method. It is data-driven and can learn data association criteria using labeled samples. In addition, different from the traditional tracking architecture, the present invention is an end-to-end architecture, which models the radar target data association as a classification problem of edges in the graph data structure, and uses an advanced graph neural network to solve the graph model once. Brief Description of the Drawings

[0025] The following further clarifies the specific implementation manners of the present invention with reference to the drawings.

[0026] Figure 1 is the processing flow chart of the radar target data association method based on graph neural network of the present invention;

[0027] Figure 2 is the architecture diagram of the graph neural network of the present invention;

[0028] Figure 3 Schematic diagram of sample amplitude features

[0029] Figure 4 Schematic diagram of the implementation of the graph neural network structure

[0030] Figure 5 Test chart of the correct rate of graph neural network edge classification

[0031] Figure 6 Tracking effect diagram of traditional method

[0032] Figure 7 Tracking effect diagram of graph neural network

[0033] Figure 8 Implementation process diagram of the present invention Specific implementation manner

[0034] Combined with Figures 1 - 8 , the radar target data association method based on graph neural network of the present invention includes the following steps: First, extract target amplitude envelope features, phase features, Doppler features, signal-to-noise ratio, RCS, target distance size, azimuth size, pitch size and other features from AD data according to radar point data. The typical amplitude envelope features of the target are shown in Figure 3 ; Then, form graph data with the extracted multi-dimensional features, target detection point data and target tracks, and model the radar target data association as a graph data edge classification problem; Then, use the labeled samples to train the graph neural network model and automatically learn the radar target data association criterion; Finally, input the point data to be tested into the graph neural network model to complete radar multi-target tracking

[0035] The specific implementation process of the present invention is divided into three stages: model training stage, model verification stage and model application stage. The implementation time sequence relationship is as shown in Figure 8 shown

[0036] 1. Model training stage

[0037] The model training stage of a radar target data association method based on graph neural network includes the design of the network model, the generation of training sample data and the training of the model server. The specific process is as follows

[0038] 1) Network model design

[0039] The graph neural network model adopted by the present invention is as shown in Figure 4 shown. In order to prevent overfitting, the network is mainly composed of a simple fully connected network and a ReLU function, and mainly includes graph neural network edge update, graph neural network node update, graph neural network global variable update and graph neural network edge classification module. The specific situation is as follows

[0040] Graph Neural Network Edge Update:

[0041] The input is the target track, target plot, edges before update, and global variables, and the output is the updated edges. This module consists of two fully connected layers and a ReLU function. Let represent the track node, represent the plot node, and e k represent the edge connecting the nodes. The updated edge is where NN is the network model to be trained.

[0042] Graph Neural Network Node Update:

[0043] The detection node features are updated by aggregating historical features into the detection node features. The input is the target track, target plot, updated edges, and global variables, and the output is the updated target plot. The updated node is where NN is the network model to be trained.

[0044] Graph Neural Network Global Variable Update:

[0045] The input is the global variables before update, aggregated nodes and edges, and the output is the updated global variables. Among them, the aggregated node is The aggregated edge is The updated global variable is

[0046] Graph Neural Network Edge Classification:

[0047] The input is the target track, updated target points, updated edges, and updated global variables, and the output is the final edge classification result, that is

[0048] 2) Formation of Training Samples

[0049] Match the truth data with the radar plots, and use the matching results to extract information such as target amplitude envelope features, phase features, Doppler features, signal-to-noise ratio, RCS, target distance size, azimuth size, and elevation size from the AD data. Then, balance the samples and normalize the features to form training samples.

[0050] In the actual scenario, not all targets have truth values, so it is impossible to label all plots. The solution of this invention is as follows: for cooperative targets, use ADS-B, AIS, and radar secondary data for matching and labeling; for non-cooperative targets, use non-causal complex tracking methods to extract tracks and complete sample labeling under the condition of manual assistance.

[0051] 3) Server Training

[0052] After completing the annotation of the training samples, the graph neural network model was trained on Nvidia V100 for 5 days, and the training of the model was completed.

[0053] 2. Model Verification Phase

[0054] 20% of the training data was divided into a validation set to verify the model, and the verification results are as Figure 5 shown. The classification accuracy of the model for the correlation relationship between the target track and the plot is significantly higher than that of the traditional method, meeting the conditions for model application.

[0055] 3. Model Usage Phase

[0056] This phase is divided into four parts, namely graph feature extraction, graph data construction, correlation relationship reasoning, and track filtering output.

[0057] 1). Graph Feature Extraction

[0058] Target features are extracted from the echo data using radar plots, including target amplitude envelope features, phase features, Doppler features, signal-to-noise ratio, RCS, target distance size, azimuth size, elevation size, etc. The features are normalized and used as the node features of the graph data.

[0059] 2). Graph Data Construction

[0060] The previous track is used as a track node, the current detected plot is used as a plot node, and the connection between the track and the plot is used as the edge of the graph data, forming the graph data to be solved and inputting it into the graph neural network to classify the edges of the graph data.

[0061] 3). Correlation Relationship Reasoning

[0062] First, the graph data flows through the graph neural network edge update module to update the graph data edges; then, the graph data flows through the graph neural network node update module to update the plot node features of the graph data; then, the graph data flows through the global variable update module to update the global variable representing the connection relationship of the graph data; finally, the graph data flows through the graph neural network edge classification module to output the correlation relationship result.

[0063] 4). Track Filtering Output

[0064] After determining the correlation relationship, the points belonging to the same track are subjected to Kalman filtering to estimate information such as the target position, target speed, target heading, and target acceleration, and the same batch number is assigned, and the tracking track is output. The track tracked by the graph neural network is as Figure 7 shown. Figure 6 The following is the tracking effect of the traditional method. By comparison, it can be seen that the graph neural network method has a very high accuracy in classifying the correlation relationship, and the continuity and maximum tracking distance of the tracked track are significantly better than those of the traditional method.

[0065] Internationally, the research on graph neural networks is in the theoretical exploration stage. This invention uses graph neural networks to solve the radar target data association problem in an "end-to-end" manner, which is the first time internationally. Compared with traditional methods, it innovatively uses graph neural networks in an "end-to-end" manner to automatically learn the radar target data association criteria from labeled samples, and can effectively fuse information such as target motion, target amplitude envelope characteristics, phase characteristics, Doppler characteristics, signal-to-noise ratio, RCS, target distance size, azimuth size, and pitch size. In terms of tracking effect, compared with traditional tracking methods, the tracks tracked by this invention have significant improvements in core indicators such as data association correct rate, track continuity, and maximum tracking distance. At the same time, this method does not rely on expert knowledge and automatically learns the association criteria using labeled samples. It can effectively fuse information of different dimensions and different dimensions, can effectively mine the global information of the connection between tracks and dots, and greatly improves the accuracy of data association. It solves the problems of missed batches, broken batches, and mixed batches in radar target tracking in complex scenarios; solves the problem of many traditional data association parameters and great difficulty in tuning; solves the problem of mining the topological structure information of data association point tracks; solves the problem of end-to-end data association modeling and solution.

[0066] In the detection problem of small targets, due to the low detection rate of small targets, it is difficult for traditional starting criteria to effectively start for small targets. This invention uses graph neural networks to automatically learn the starting criteria through labeled samples and multi-dimensional features, and can effectively improve the starting efficiency of small targets. By using graph neural networks to mine the global topological information of target tracks and target dot tracks, the data association success rate is higher under the condition of low detection rate, and the maximum tracking distance of small targets can be effectively improved. The traditional MHT method requires >= 20 parameters to be tuned, and the performance of the algorithm depends on expert experience. The graph neural network method proposed in this invention is data-driven, and the model parameters are automatically learned through labeled samples, without the need for users to tune parameters such as association thresholds and filtering. The number of tuning parameters <= 2, and the tuning difficulty is greatly reduced, solving the problem of many target tracking parameters and great difficulty in tuning.

[0067] In the above description, many specific details are elaborated to fully understand this invention. However, the above description is only a preferred embodiment of this invention, and this invention can be implemented in many other ways different from those described here. Therefore, this invention is not limited by the specific implementation disclosed above. At the same time, any person familiar with the technical field can make many possible changes and modifications to the technical solution of this invention, or modify it into an equivalent embodiment with equivalent changes, without departing from the scope of the technical solution of this invention. Any simple modification, equivalent change, and modification made to the above embodiment based on the technical essence of this invention without departing from the content of the technical solution of this invention still fall within the scope of the protection of the technical solution of this invention.

Claims

1. A radar target data association method based on graph neural network, characterized by: The steps include: Perform multi-dimensional feature extraction on radar traces and AD data; Constructing graph data: Graph data is constructed using the extracted multi-dimensional features, target tracks, and points to be associated; Implementing graph neural networks: including graph neural network edge updates, graph neural network node updates, graph neural network global variable updates, and graph neural network edge classification; After the model is built, the labeled samples are used to train the graph neural network model to automatically learn the radar target data association criteria. The point traces to be tested are input into the graph neural network model to complete radar multi-target tracking. Uses of graph neural network models include: Graph feature extraction: Target features are extracted from echo data using radar traces, including target amplitude envelope features, phase features, Doppler features, signal-to-noise ratio, RCS, target range, azimuth, and elevation dimensions. These features are then normalized to form node features of the graph data. Graph data construction: The track at the previous moment is used as the track node, the point track detected at the current moment is used as the point track node, and the connection between the track and the point track is used as the edge of the graph data to form the graph data to be solved. The graph data is input into the graph neural network to classify the edges of the graph data; Association reasoning: First, graph data flows through the graph neural network edge update module to update the graph data edges; then, the graph data flows through the graph neural network node update module to update the graph data trace node features; then, the graph data flows through the global variable update module to update the global variables representing the graph data connection relationships; finally, the graph data flows through the graph neural network edge classification module to output the association relationship results; Track filtering output: After determining the association relationship, Kalman filtering is performed on the points belonging to the same track to estimate the target position, target speed, target heading, target acceleration and other information, mark them with the same batch number, and output the tracking track.

2. The radar target data association method based on graph neural network according to claim 1, characterized in that: The multi-dimensional features extracted from the detected radar traces and AD data include target amplitude envelope features, phase features, Doppler features, signal-to-noise ratio, RCS, target range size, azimuth size, and pitch size.

3. The radar target data association method based on graph neural network according to claim 1, characterized in that: In constructing graph data, the nodes of the graph include target tracks and point tracks to be associated, the edges of the graph are formed by connecting the target tracks and the points to be associated, and the connection weights of the edges represent the corresponding association relationships.

4. The radar target data association method based on graph neural network according to claim 1, characterized in that: The input of the graph neural network edge update is the target track, target point track, edge before update, and global variable, and the output is the updated edge. The module includes two fully connected layers and a ReLU function.

5. The radar target data association method based on graph neural network according to claim 1, characterized in that: The graph neural network node update condenses historical features into detection node features, updates the detection node features, and inputs are target tracks, target point tracks, updated edges, and global variables, and the output is the updated target point track.

6. The radar target data association method based on graph neural network according to claim 1, characterized in that: The global variable update of the graph neural network utilizes the target trajectory and target point trajectory connection information to mine the global information of the graph structure. The input is the global variables before the update, the aggregated nodes and edges, and the output is the updated global variables.

7. The radar target data association method based on graph neural network according to claim 1, characterized in that: The input of the graph neural network edge classification is the target track, the updated target point, the updated edge, and the updated global variable. The output is the final edge classification result, that is, the association relationship between the track and the point track.

8. The radar target data association method based on graph neural network according to claim 1, characterized in that: The training sample formation process is as follows: match the true value data with the radar traces, and use the matching results to extract the target amplitude envelope characteristics, phase characteristics, Doppler characteristics, signal-to-noise ratio, RCS, target range size, azimuth size, and pitch size information from the AD data. Then, the samples are equalized and the features are normalized to form training samples.

Images