Asynchronous robust track association method based on temporal encoding and graph representation

Abstract

The present application relates to the technical field of distributed radar data processing, and discloses an asynchronous robust track association method based on temporal encoding and graph representation. The method relates to an intra-track representation module and an inter-track representation module, wherein the intra-track representation module is used for extracting spatio-temporal features among different plots within a track, and the inter-track representation module is used for extracting, in a track representation space, geometric features between tracks from the same sensor. In the method, a general model of a generalized track association task is established. By means of end-to-end training and learning, multi-source track association and broken track association can be simultaneously solved by using one model, and better generalization performance and association accuracy are achieved under an asynchronous condition and a systematic bias.

Description

Asynchronous robust track association method based on time encoding and graph representation

[0001] The present application claims priority to the Chinese patent application No. 202410713363.3, filed on June 04, 2024, and entitled "Asynchronous robust track association method based on time encoding and graph representation", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of distributed radar data processing, in particular to an asynchronous robust track association method based on time encoding and graph representation. BACKGROUND

[0003] Target track association is a key technology of distributed fusion system, aiming to determine whether the tracks reported by different sensors are from the same target. Accurate track association is the basis for track segment completion and homogenous track fusion, which improves the accuracy of multi-source situation awareness and improves the identification efficiency. Ultimately, a clear, unified and complete multi-source fusion situation is obtained. However, in a complex electromagnetic environment, due to missed detection, sampling frequency, start time, communication delay and time synchronization error, the timestamps of different tracks are often asynchronous. In addition, due to calibration bias and platform attitude error, target measurements may have random noise and constant or progressive system bias. The combination of the above factors reduces the effectiveness of track association and limits the quality of high-level situation awareness fusion. Therefore, in a multi-source distributed fusion system, under the combined influence of random noise, system bias and asynchrony, track association is challenged.

[0004] For asynchronous tracks, time alignment techniques are usually used to obtain synchronous data by fitting, extrapolation or prediction to meet the requirements of the fusion center, but due to additional errors caused by target maneuvering and alignment performance, it increases the system complexity and reduces the fusion quality. Related methods directly use asynchronous tracks for association, but the track similarity measurement function constructed is not unbiased for homogenous asynchronous track data, and the computational complexity is high, and the association performance for short tracks is poor. For synchronous track association with system bias, by manually designing track features that are robust to system bias, good association performance is achieved, but these features are easily affected by measurement noise, and lack the ability to actively extract and select effective features. In some cases, there is a lack of representative research results on track association methods based on deep learning, and there is a lack of systematic research results on asynchronous track and robust association problems. SUMMARY

[0005] The purpose of the present application is to provide an asynchronous robust track association method based on time coding and graph representation for the general track association problem under the combined influence of random noise, system bias and asynchrony. By establishing a general model of the general track association task, through end-to-end training and learning, a model can be used to simultaneously solve multi-source track association and interrupted track association, and has better generalization performance and association accuracy under asynchronous and system bias conditions.

[0006] The present application provides an asynchronous robust track association method based on time coding and graph representation, which includes an intra-track representation module and an inter-track representation module, wherein the intra-track representation module is used to extract the space-time features between different point tracks in the track, and the inter-track representation module is used to extract the geometric features between the same sensor tracks in the track representation space, and specifically includes the following steps: step 1: using domain normalization on track point track data; step 2: using multiple point track mask strategies to simulate asynchronous association scenarios; step 3: inputting the point track after point track mask to the intra-track representation module to obtain intra-track representation; step 4: inputting the intra-track representation into the inter-track representation module to obtain inter-track representation; step 5: calculating the loss function to train the network model; step 6: after training the network model, directly using the output inter-track representation to calculate the similarity between the tracks, and using the nearest neighbor or optimal decision method to complete the track association.

[0007] The present application supports the neural network model design of the general track association task, such as multi-source track association and track segment association, through the asynchronous robust track association method based on time coding and graph representation. The proposed intra-track representation module embeds the timestamp into the intra-track representation through the constructed time coding, so that the model can simultaneously solve the multi-source asynchronous track association and track segment association tasks without any time alignment. The proposed inter-track representation module uses the K-order nearest neighbor with threshold to establish a track graph in the track representation space, and extracts the inter-track representation based on graph convolution and point convolution. Finally, the track association is realized by directly calculating the distance between the inter-track representations. The domain normalization and multiple point track mask strategies are used before data input, which enhances the generalization performance of the model. The proposed method has good performance for asynchronous track robust association scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0008] Fig. 1 is a flowchart of the asynchronous robust track association method based on time coding and graph representation provided by the embodiment of the present application.

[0009] Fig. 2 is a flowchart of the asynchronous robust track association method based on time coding and graph representation provided by the embodiment of the present application. DETAILED DESCRIPTION

[0010] In order to make the purposes, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in a clear and complete manner in combination with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0011] The present application will be described in further detail in combination with the flowchart of the embodiments of the present application described in combination with FIG. 1-FIG. 2.

[0012] The asynchronous robust track association method based on time coding and graph representation of the present embodiment has a twin-sibling network structure, mainly including a track intra-representation module and a track inter-representation module. The track intra-representation module is used to extract the space-time features between different point tracks in a track, and the track inter-representation module is used to extract the geometric features between tracks of the same sensor in the track representation space. The track intra-representation module includes three steps of time coding, point track feature extraction and track feature extraction. By embedding a timestamp into a point feature, asynchronous intra-track representation is obtained, so that the model can solve the generalized track association task, such as multi-source track association and track discontinuous association.

[0013] The track inter-representation module includes track graph construction and track inter-feature extraction. The track graph construction is constructed on the track representation space by using a K-order nearest neighbor method with a threshold, and then the track inter-representation is obtained by using graph convolution and multiple point convolution. Finally, the similarity between tracks is calculated directly using the track inter-representation to complete track association. In addition, before data input, domain normalization and point track mask are used to enhance the generalization performance of the model to various asynchronous association scenarios.

[0014] It is considered that two sensor systems can detect and track multiple targets in a distributed architecture. After stable tracking, the target track i obtained by the sensor system s is where N s is the number of target tracks observed by the sensor s. Each track is composed of multiple track points, that is where is the state of the track point at the timestamp , including position, velocity, even acceleration or attribute information, is the length of the track.

[0015] As shown in FIG. 1-FIG. 2, the present embodiment discloses an asynchronous robust track association method based on time coding and graph representation, including the following steps.

[0016] Step 1: Domain normalization is used for track point track data.

[0017] In practical applications, track-to-point data can be collected by radar (active / passive) to obtain target range, bearing, velocity, etc. point information, or by photoelectric sensors (such as infrared, visible light cameras) to collect target image features and space-time data.

[0018] Track-to-point data generally includes location domain, velocity domain, attribute domain, etc. Each domain has different value range and physical meaning, and should be normalized respectively. Taking a two-dimensional scene as an example, the kth point of target i of sensor s is represented as Wherein is the location domain information, is the velocity domain information. In addition, for different dimensions in the same domain, such as and After normalization, the point sequence should have translational invariance and rotational invariance to adapt to possible track data enhancement methods (such as rotation), so the data of different dimensions in the same domain should be scaled by the same scale. Therefore, domain normalization (DN) is proposed to preprocess the data.

[0019] First, the average value and standard deviation of the target state in different dimensions are calculated. Then, the mean and standard deviation are fused according to the domain. Finally, each dimension of the target track data is normalized according to the standard deviation of the domain and the average value of the dimension. The position domain normalization can be described as follows.

[0020] Wherein, x avg , y avg , x std and y std are the average value and standard deviation of the two sensors in the position dimension, p std is the standard deviation of the position domain. The normalization of other domains is similar to the normalization of the position domain. The track data normalized by the domain is consistent in different dimensions of the same domain, which is beneficial to improve the generalization performance of the model.

[0021] Step 2: Use multiple point mask strategies to simulate asynchronous association scenarios.

[0022] Different point mask strategies are used to simulate different asynchronous scenarios. Track point mask can also reduce the number of points in the track and improve the training speed.

[0023] (1) Random point mask, according to the given probability to cover each point in the training data set, used to simulate asynchronous track caused by missed detection.

[0024] (2) Asynchronous point mask, n consecutive points of a synchronous track are regarded as a point group, and the synchronous track is divided into multiple point groups, each of which contains n points. In the asynchronous test data set, for sensor 1, the last m-1 points of each point group are shielded, and for each point group in sensor 2, all track points except the mth point are shielded. When m = 1, they are synchronous tracks, and when m = 2,...,n, they are asynchronous tracks. The proximity of m to (n+2) / 2 can represent the time difference and the degree of asynchrony. The closer m is to (n+2) / 2, the greater the time difference and the degree of asynchrony, which simulates asynchronous tracks caused by different start-up times.

[0025] (3) Intermittent point mask, in each synchronous track pair, 2n+m consecutive points are randomly selected, and the other points are masked. Then, in sensor 1, the first n+m points of the selected points are masked, and in sensor 2, the last n+m points are masked. After processing, the homologous tracks do not overlap in time domain, and the time interval between the tracks is m points. The value of m represents the difficulty of intermittent association and is used to simulate the intermittent association scenario.

[0026] Step 3: input the point mask after the point mask into the track internal representation module to obtain the track internal representation.

[0027] The proposed track internal representation module includes three steps: time coding, point feature extraction, and track feature extraction.

[0028] 3.1 Time coding is composed of sine and cosine functions of different frequencies. The time coding of the kth track point is

[0029] where f is a hyperparameter and is regarded as the base frequency. f should take a large value, and f < 1 / T w , d model is the dimension of time coding, d = 1, 2,..., d model / 2.

[0030] To ensure the effect of time coding, on the one hand, the time coding within the association window T w should not be repeated; on the other hand, the base frequency f should not be too small to make the time coding flat. Therefore, f should take a large value, and f < 1 / T w . By embedding time coding, the model can solve the generalized track association task, such as multi-source track association and intermittent track association.

[0031] 3.2 Point feature extraction: use 1 fully connected layer to directly extract point features, and multiple target points are operated in parallel.

[0032] 3.3 Track feature extraction: The time encoding and tracklet features are concatenated and input into a recurrent neural network to extract features to obtain intra-track representations. Due to the embedding of the position, motion information in the tracklet and asynchronous time information in the time encoding, the obtained intra-track representations contain rich asynchronous track spatiotemporal information.

[0033] Step 4: The intra-track representations are input into an inter-track representation module to obtain inter-track representations.

[0034] The inter-track representation module in step 4 includes two steps of track graph construction and inter-track feature extraction.

[0035] Step 4.1: The intra-track representation of each track is taken as a node, and all target tracks observed by a sensor constitute a node set A K-Nearest Neighbor (KNN) method with a threshold is used to establish a directed graph.

[0036] Step 4.2: After the track graph is constructed, a graph convolution is used to obtain inter-track features. The graph convolution uses multiple different graph convolutions. After the inter-track features are obtained, a 1x1 point convolution is used to further expand the output dimension, enrich the inter-track feature extraction, and obtain the final inter-track representation. LayerNorm is used between the convolution layers to normalize the entire feature map, and the ReLU function is used to improve the nonlinear fitting ability of the model.

[0037] Preferably, step 4.1 is specifically: the Euclidean distance of each node to other nodes in the representation space is calculated, the nodes with a distance less than a threshold are taken as candidate nodes, if the number of candidate nodes is greater than K, the K candidate nodes with the smallest distance are selected to establish edges, if the number of candidate nodes is less than or equal to K, an edge is established with each candidate node, and after a self-loop is added to each node, a directed track graph of each sensor is obtained wherein is the set of edges.

[0038] Step 5: Calculate the loss function to train the network model.

[0039] For the track association task, it is required to minimize the distance between the representations of homologous tracks and maximize the distance between the representations of heterologous tracks. In order to meet these requirements, a loss function is proposed as follows.

[0040] wherein margin>0, and are inter-track representations from different sensors, and track j + is a homologous track of track i, is the heterologous track with the smallest distance to track i, and Npair is the number of homologous track pairs.

[0041] Step 6: After the network model is trained, the similarity between tracks is calculated directly using the output inter-track representation, and track association is completed using the nearest neighbor or optimal decision method.

[0042] In practical applications, after track association, target behavior analysis, threat assessment, cooperative tracking, and trajectory prediction can be performed, which can improve the battlefield situation awareness, traffic control, and security monitoring capabilities, and support intelligent decision-making and resource optimization scheduling.

[0043] The asynchronous robust track association method based on time encoding and graph representation of the present application supports the design of neural network models for general track association tasks, such as multi-source track association and track segment association. The proposed intra-track representation module embeds timestamps into intra-track representations through time encoding, enabling the model to simultaneously solve multi-source asynchronous track association and track segment association tasks without any time alignment. The proposed inter-track representation module uses thresholded KNN to establish a track graph in the track representation space and extracts inter-track representations based on graph convolution and point convolution. Finally, track association is achieved by directly calculating the distance between inter-track representations. Domain normalization and various point mask strategies are used before data input, enhancing the model's generalization performance. The proposed method performs well in asynchronous track robust association scenarios.

[0044] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist, they should be considered as within the scope of the present application.

[0045] The principles and implementation modes of the present application are described by applying specific examples in this paper, and the above embodiments are only used to help understand the method and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. An asynchronous robust track association method based on time encoding and graph representation, characterized in that The method comprises an intra-track representation module and an inter-track representation module, wherein the intra-track representation module is used to extract space-time features between different point tracks in a track, and the inter-track representation module is used to extract geometric features between tracks of the same sensor in a track representation space, and specifically comprises the following steps: Step 1: domain normalization is used for track point track data; Step 2: a variety of point track masking strategies are used to simulate an asynchronous association scenario; Step 3: the point track masked point track is input into the intra-track representation module to obtain intra-track representation; Step 4: the intra-track representation is input into the inter-track representation module to obtain inter-track representation; Step 5: a loss function is calculated to train the network model; Step 6: after the network model is trained, the inter-track representation output is directly used to calculate the similarity between tracks, and a nearest neighbor or optimal decision method is used to complete track association.

2. The asynchronous robust track association method based on time encoding and graph representation according to claim 1, characterized in that The calculation method of the domain normalization for the track point track data in step 1 is as follows: In a two-dimensional scenario, the kth point trace of the target i for the sensor s is denoted as for the trace corresponding to the timestamp, wherein for the position domain information, For the velocity domain information, the position domain is normalized as: where x avg , y avg , x std and y std are the mean and standard deviation of the two sensors in the position dimension, and p std is the standard deviation of the position domain.

3. The asynchronous robust track association method based on time encoding and graph representation according to claim 1, characterized in that The point track masking strategy in step 2 comprises: (1) random point track masking, each point track in the training data set is masked according to a given probability, which is used to simulate asynchronous tracks caused by missed detection; (2) asynchronous point track masking, n continuous points of a synchronous track are regarded as a point group, the synchronous track is divided into multiple point groups, each group contains n points, in the asynchronous test data set, for sensor 1, the last m-1 points of each point group are shielded, and for each point group in sensor 2, except for the mth point, other track points are shielded, when m=1, they are synchronous tracks, and when m=2,…,n, they are asynchronous tracks, the proximity of m to (n+2) / 2 can represent the time difference and the degree of asynchrony, which is used to simulate asynchronous tracks caused by different start-up times; (3) intermittent point track masking, in each synchronous track pair, 2n+m continuous points are randomly selected, and other points are masked, then in sensor 1, the first n+m points of the selected points are masked, and in sensor 2, the last n+m points are masked, after processing, the homologous tracks do not overlap in time domain, and the time interval between the tracks is m points, the value of m represents the difficulty of intermittent association, which is used to simulate the intermittent association scenario.

4. The asynchronous robust track association method based on time encoding and graph representation according to claim 1, characterized in that The intra-track representation module in step 3 comprises three steps of time coding, point track feature extraction and track feature extraction: Step 3.1 : Ignoring the sensor label s and the target label i, the time- encoded representation of the kthwaypoint is denoted as where f is a hyperparameter, viewed as the base frequency, and f < 1 / T w , d model is the dimension of the time encoding, d = 1, 2,..., d model / 2; Step 3.2: a fully connected layer is used to directly extract point track features, and multiple target point tracks are operated in parallel; Step 3.3: the time coding and the point track features are spliced and input into a recurrent neural network to extract features to obtain intra-track representation.

5. The asynchronous robust track association method based on time encoding and graph representation of claim 1, wherein The inter-track representation module in step 4 comprises two steps of track graph construction and inter-track feature extraction: Step 4.1: The intra-track representation of each track is taken as one node, and all target tracks observed by one sensor constitute the node set A K-order nearest neighbor method with a threshold is used to establish a directed graph; Step 4.2: after the track graph is constructed, a graph convolution is used to obtain inter-track features, and multiple different graph convolutions are used; after the inter-track features are obtained, a point convolution with a size of 1×1 is used to further expand the output dimension, enrich the inter-track feature extraction, and obtain the final inter-track representation; LayerNorm is used to normalize the entire feature map between convolution layers, and a ReLU function is used to improve the nonlinear fitting ability of the model.

6. The asynchronous robust track association method based on time encoding and graph representation of claim 5, wherein Step 4.1 is specifically: calculating the Euclidean distance of each node to other nodes in the feature space, taking the nodes with distance less than a threshold as candidate nodes, if the number of candidate nodes is greater than K, selecting the K candidate nodes with the smallest distance to establish edges, if the number of candidate nodes is less than or equal to K, establishing edges with each candidate node, and after adding a self-loop to each node, obtaining the directed track graph of each sensor wherein is a set of edges.

7. The asynchronous robust track association method based on time encoding and graph representation of claim 1, wherein The calculation method of the loss function described in step 5 is: margin > 0, and is an inter-track representation from different sensors, track j + is a homologous track to track i, is the minimum distance to the track i, N pair is the number of pairs of homologous tracks.

Images