A method and system for multi-source track data fusion based on LSTM network

Abstract

This invention discloses a multi-source track data fusion method and system based on LSTM networks, applied in the field of multi-source track fusion technology. The method includes the following steps: time registration of track points; construction of the input sequence for LSTM; calculation of the correlation probability between the original radar track points and existing fused track points using LSTM; classification of the original radar track points; quality management of the classification results based on the average correlation probability; and estimation of the target's future state by fusing data from multiple radars for a single target and employing a filtering algorithm. This invention introduces the LSTM algorithm, utilizing the contextual information of time-series data to capture the temporal continuity and spatial correlation of targets, effectively improving the accuracy of track association. The model is used for track identity judgment and track quality management, thereby improving the performance of the track fusion system.

Description

Technical Field

[0001] This invention relates to the field of multi-source track fusion technology, and more specifically to a method and system for multi-source track data fusion based on LSTM networks. Background Technology

[0002] Track fusion systems are systems used to integrate target detection information from multiple sensors (such as radar) to generate unified target tracking information. Their main purpose is to determine the identity of independent track data from multiple radars or sensors, i.e., track association, and generate fused tracks to obtain more accurate and comprehensive target track information, improving tracking accuracy and system robustness. Track association is one of the key technologies in multi-sensor information fusion; the correctness of track association directly affects the functionality and performance of the track fusion system. Currently, track association methods can be divided into five categories according to the different frameworks of the problems they handle: probability and statistics-based association methods, uncertainty information processing-based methods, target optimization constraint-based methods, signal processing-based methods, and target reference topology-based methods. However, in complex aerial target scenarios such as dense, intersecting, and close-range parallel targets, the accuracy of traditional track association methods decreases significantly.

[0003] When multiple targets are densely distributed in the airspace, the tracks detected by different radars may overlap or be close together. In such cases, traditional track association methods may struggle to distinguish the tracks of different targets, leading to association errors. When the tracks of targets intersect in the air, the track data returned by the radar may be interleaved, making it difficult for traditional algorithms to accurately determine which tracks belong to the same target. In scenarios where targets fly parallel to each other, the directions and speeds of different targets are very similar, making it difficult for traditional methods to maintain stable target differentiation over long periods. Therefore, how to correctly associate the tracks of aerial targets in complex scenarios such as dense, intersecting, and close-range parallel targets is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the present invention provides a multi-source track data fusion method and system based on LSTM network. By introducing the LSTM algorithm, the contextual information of time series data is used to capture the temporal continuity and spatial correlation of targets, accurately identify and distinguish the tracks of dense, intersecting, and close-range parallel targets, effectively improving the accuracy of track association. The rule model is used to judge track identity and manage track quality, thereby improving the accuracy of track association in complex aerial target scenarios such as dense, intersecting, and close-range parallel targets, and thus improving the performance of the track fusion system.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A multi-source track data fusion method based on LSTM networks includes the following steps:

[0007] S1. Using the time t of the original radar track point as a reference, the existing fused track point at time t-1 is extrapolated to time t, so that the original radar track point and the fused track point are comparable at the same time point.

[0008] S2. Normalize the state vectors of the original radar track points and the state vectors of any track point in the fused track set, construct a time series that simultaneously contains the original radar track points and the fused track points as the input sequence of the LSTM, and traverse all track points in the fused track set.

[0009] S3. Calculate the probability of whether they belong to the same target by using the correlation between the original track points of the LSTM radar and the existing fused track points.

[0010] S4. Based on the information of relevant probabilities, original radar track points and fused track points, classify the original radar track points, and classify them into new fused tracks or fused tracks associated with the original radar tracks according to the relevant probability threshold.

[0011] S5. Calculate the average probability of the sum of the correlation probabilities between multiple original radar track points of the target and existing fused track points. If the average probability between the original radar track points in the target's original radar track and the fused track points in a fused track is greater than a preset threshold, associate the current original radar track with the fused track. If the average probability between the original radar track points in the target's original radar track and the fused track points in all fused tracks is less than a preset threshold, then determine that the original radar track point is the track point of a newly discovered target, establish a fused track file, and generate a new fused track. If there is no data update for r consecutive detection cycles, then delete the track file.

[0012] S6. By fusing data from multiple radars for a single target, a filtering algorithm is used to estimate the target's future state.

[0013] Optionally, the original radar track point set in S1 is U l ={l1,l2,…,l m}, the merged waypoint set is U l ={l1,l2,…,l m}, where n is the number of fused track points, m is the number of original radar track points, and the track point of fused track i at time t is} The state vector, the original track j of radar l at time t is The state vector, Merge existing track points at time t-1 Extrapolation

[0014] Optionally, S2 specifically refers to: the original radar track points state vector With fused track set U s any waypoint in state vector Normalization is performed to obtain the normalized state vector. and For each element x in the waypoint state vector, its normalized value x′ is:

[0015]

[0016] Construct a time series that simultaneously contains both the original radar track points and the fused track points. As the input sequence for LSTM:

[0017]

[0018] In the formula, It is the number of radar scan cycles contained in the track, where T is the time step of one radar scan cycle, s. l (τ) is the input feature vector corresponding to time point τ, s l (τ) is derived from the normalized radar original track point feature vector. and normalized fused track point feature vector Combination structure:

[0019] Optionally, S3 specifically refers to:

[0020] S31. The constructed input sequence is processed through an LSTM network to obtain the probability distribution.

[0021]

[0022] In the formula, Indicates the original radar track point Merging track point with the i-th merging track The correlation probability between them, W p and b p These are the weight matrix and bias vector of the Softmax layer, respectively.

[0023] S32. Considering that relying solely on probability thresholds may lead to misjudgments of minority classes of tracks, an entropy H(P) is used to measure the matching confidence between the original radar track points and the fused track points:

[0024]

[0025] In the formula, H(P) represents the uncertainty of the correlation between the original radar track point and multiple fused track points. The larger the entropy value, the more uniform the distribution and the lower the confidence level; the smaller the entropy value, the more the probability is concentrated in a certain fused track point and the higher the confidence level.

[0026] Optionally, S4 specifically refers to:

[0027] Based on relevant probabilities and a preset threshold P th Information including entropy H(P), original radar track points, and fused track points is used to classify the original radar track points. When the correlation probability between the original radar track points and the fused track points is less than a preset threshold P... th If no fused track is found, the track point is determined to be the track point of the newly discovered target. A fused track file is established, and a new fused track is generated. The current original track point is used as the track point of the new track. When the correlation probability between the original radar track point and the fused track point is greater than a preset threshold P... th When considering the uncertainty of the track, the entropy H(P) of the correlation probability is calculated as an uncertainty measure to provide decision support. If the correlation probability P of the original radar track point is greater than the threshold P... th And the entropy H(P) is less than the set entropy threshold H t If the entropy H(P) is greater than H, then it is confirmed as a fused track associated with the original radar track; otherwise, if the entropy H(P) is greater than H th If the value is zero, it indicates that the track point has a high degree of uncertainty and is marked as a potential new target track.

[0028] Optionally, S6 specifically involves fusing data from multiple radars for a single target and using a filtering algorithm to generate the target's future state, including its position, direction of motion, and velocity. Specifically: if the fused track is a new fused track or the number of track points is less than 3, then the original track information of the current radar is output, and this original track information is used as the new track point for the associated fused track; if the number of fused track points is greater than 3, then maneuvering target filtering is applied to the track points before time t-1 of the fused track and the original track points of the current radar to generate a state vector at time t, which is then used as the new track point for the associated fused track, replacing the track point extrapolated to time t, and the new track point information of the fused track is output. Maneuvering target filtering includes interactive multi-model filtering algorithms, Singer model filtering algorithms, etc.

[0029] A multi-source track data fusion system based on LSTM networks executes the aforementioned multi-source track data fusion method based on LSTM networks, including: a time registration module, a track probability calculation module, a track association management module, a track quality management module, and a track state estimation module; the time registration module registers the fused track points, the track probability calculation module calculates the correlation probability between the original radar track points and the existing fused track points, the track association management module classifies the original radar track points, the track quality management module manages the quality of the track classification results, and the track state estimation module estimates the future state of the target.

[0030] As can be seen from the above technical solution, compared with the prior art, the present invention provides a multi-source track data fusion method and system based on LSTM network, which has the following beneficial effects:

[0031] 1. Improved accuracy of track association: This invention uses the LSTM algorithm to calculate the correlation probability between the original radar track and the fused track, which greatly improves the accuracy of track association. It is more robust and accurate than traditional methods, enabling the system to provide more accurate target tracking information in complex environments.

[0032] 2. Real-time performance and efficiency of the system: This invention utilizes the LSTM algorithm, which can quickly process data from multiple radars and achieve real-time track fusion. Even with a large amount of track point data, the system can still maintain a high computing speed and ensure the timeliness of the fusion results.

[0033] 3. System stability and reliability: This invention adopts a track quality management mechanism, which effectively prevents the impact of single correlation probability fluctuations through statistical analysis of relevant probabilities, enhances the stability of the system in target tracking, cleans up track data that has not been updated for a long time, avoids data redundancy, improves system resource utilization efficiency, and meets engineering application requirements;

[0034] 4. Social and Economic Benefits: The application of this invention can significantly improve the efficiency and accuracy of aerial target tracking systems, and is applicable to fields such as national defense and air traffic management. Efficient and accurate target tracking capabilities will help improve airspace safety, reduce accident risks, and thus bring significant social and economic benefits. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0036] Figure 1 This is a flowchart of the multi-source track data fusion method of the present invention;

[0037] Figure 2 This is a schematic diagram of track fusion deployment in an embodiment of the present invention;

[0038] Figure 3 This is a schematic diagram of the LSTM network structure in an embodiment of the present invention. Detailed Implementation

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0040] This invention discloses a multi-source track data fusion method based on LSTM networks, such as... Figure 1 As shown, it includes the following steps:

[0041] S1. Using the time t of the original radar track point as a reference, the existing fused track point at time t-1 is extrapolated to time t, so that the original radar track point and the fused track point are comparable at the same time point.

[0042] S2. Normalize the state vectors of the original radar track points and the state vectors of any track point in the fused track set, construct a time series that simultaneously contains the original radar track points and the fused track points as the input sequence of the LSTM, and traverse all track points in the fused track set.

[0043] S3. Calculate the correlation probability between the original radar track points and the existing fused track points using LSTM to determine whether they belong to the same target.

[0044] S4. Based on the information of relevant probabilities, original radar track points and fused track points, classify the original radar track points, and classify them into new fused tracks or fused tracks associated with the original radar tracks according to the relevant probability threshold.

[0045] S5. Calculate the average probability of the sum of the correlation probabilities between multiple original radar track points of the target and existing fused track points. If the average probability between the original radar track points in the target's original radar track and the fused track points in a fused track is greater than a preset threshold, associate the current original radar track with the fused track. If the average probability between the original radar track points in the target's original radar track and the fused track points in all fused tracks is less than a preset threshold, then determine that the original radar track point is the track point of a newly discovered target, establish a fused track file, and generate a new fused track. If there is no data update for r consecutive detection cycles, then delete the track file.

[0046] S6. By fusing data from multiple radars for a single target, a filtering algorithm is used to estimate the target's future state.

[0047] In this embodiment of the invention, the radar is deployed as follows: Figure 2 As shown, there are six radars, each no more than 5 km away from the track fusion center. Each radar can independently detect multiple targets in the air. The radars scan for targets in the air according to a period T. For each detected target, a data vector, i.e., a track point, is generated within that period, including: target ID, range, azimuth, elevation angle, detection time, etc. The set of track points of the target at each time moment is called the track. Each radar can obtain its own geographical location through positioning equipment and send the target data and location data detected each time to the track fusion center through the communication network. The fusion center performs multi-radar, multi-target track fusion.

[0048] In this embodiment of the invention, S6 employs one of the interactive multi-model filtering algorithm and the Singer model filtering algorithm.

[0049] Furthermore, the original radar track point set in S1 is U l ={l1,l2,…,l m}, the merged waypoint set is U l ={l1,l2,…,l m}, where n is the number of fused track points, m is the number of original radar track points, and the track point of fused track i at time t is} The state vector, the original track j of radar l at time t is The state vector, Merge existing track points at time t-1

[0050] Extrapolation

[0051] The input for time registration is a single original track point from a radar, or multiple or zero fused track points. Time registration is a prerequisite for subsequent track association. Only when the time is consistent can a correct association judgment be made between a single original track point from a radar and a fused track point. The time-registered data is transmitted to track correlation probability judgment and track association management.

[0052] Furthermore, S2 specifically refers to: the radar's original track point state vector With fused track set U s any waypoint in state vector Normalization is performed to obtain the normalized state vector. and For each element x in the waypoint state vector, its normalized value x′ is:

[0053]

[0054] Since the elements in the state vectors of different waypoints may have different orders of magnitude, direct processing by LSTM will cause the elements with larger values ​​to dominate during model training, which will lead to the model being biased towards these elements with larger values. To avoid the influence of element scale differences, normalization is required.

[0055] To fully utilize the capabilities of LSTM in processing sequential data and maximize the capture of the dynamic relationship and evolution patterns between the original radar track and the fused track, a time series data structure containing both original radar track points and fused track points is constructed. As the input sequence for LSTM:

[0056]

[0057] In the formula, It is the number of radar scan cycles contained in the track, s l (τ) is the input feature vector corresponding to time point τ, s l (τ) is derived from the normalized radar original track point feature vector. and normalized fused track point feature vector Combination structure:

[0058] In this embodiment of the invention, the structure of the LSTM network is as follows: Figure 3As shown, the core components of an LSTM network are the memory unit (Cell State Update) and three gating mechanisms: the Forget Gate, the Input Gate, and the Output Gate. Through these gating mechanisms, LSTM can effectively retain or discard historical information in the time series, thereby adapting to the modeling needs of long-term dependencies in the time series.

[0059] The forget gate determines which past information needs to be forgotten at the current moment, the input gate determines which new information needs to be written to the memory cell, updating the state of the memory cell is the core of LSTM, responsible for storing information, and the output gate determines the hidden state at the current moment and the final output.

[0060] In this embodiment of the invention, the loss function of LSTM is defined as the negative log-likelihood function:

[0061]

[0062] In the formula, It is an indication function, when the radar's original track point With merged waypoints The value is 1 if the objects belong to the same target, and 0 otherwise. The optimization process uses the backpropagation algorithm to adjust the model parameters by minimizing the loss function in order to maximize the correlation probability between the original track points and their true corresponding fused track points.

[0063] Furthermore, S3 specifically refers to:

[0064] S31, Forget Gate for the hidden state h of the previous moment. t-1 and the input sequence at the current time We perform weighted summation and use the Sigmoid activation function to achieve information forgetting and retention:

[0065] f t =σ(W f ·[h t-1 ,s l (t)]+b f )

[0066] In the formula, W f This is the weight matrix of the forget gate, b f σ is the bias vector of the forget gate, and σ(·) represents the Sigmoid activation function;

[0067] S32. The input gate generates a gating variable i that determines the writing ratio through a Sigmoid activation function. t :

[0068] i t =σ(Wi ·[h t-1 ,s l (t)]+b i )

[0069] A candidate memory content is generated using a tanh activation function.

[0070]

[0071] In the formula, W i and b i These are the weight matrix and bias vector of the input gate, respectively, W. C and b C These are the weight matrix and bias vector of the candidate memory content, respectively, and tanh(·) represents the hyperbolic tangent activation function;

[0072] S33, Update the memory cell to store the new memory cell state C t :

[0073]

[0074] S34. The output gate generates the control variable o through the Sigmoid function. t :

[0075] o t =σ(W o ·[h t-1 ,s l (t)]+b o )

[0076] Combined with the state C of the memory unit t After scaling using the tanh function, the hidden state h is obtained. t :

[0077] h t =o t ⊙tanh(C t )

[0078] In the formula, W o and b o These are the weight matrix and bias vector of the output gate, respectively;

[0079] S35, h is passed through a Softmax layer t Adjusted to a probability distribution, this measures the likelihood that the original radar track points and the fused track points belong to the same target:

[0080]

[0081] In the formula, Indicates the original radar track point Merging track point with the i-th merging track The correlation probability between them, W p and b p These are the weight matrix and bias vector of the Softmax layer, respectively.

[0082] Furthermore, S4 specifically refers to: based on the relevant probability and a preset threshold P th Information including entropy H(P), original radar track points, and fused track points is used to classify the original radar track points. When the correlation probability between the original radar track points and the fused track points is less than a preset threshold P... th If no fused track is found, the track point is determined to be the track point of the newly discovered target. A fused track file is established, and a new fused track is generated. The current original track point is used as the track point of the new track. When the correlation probability between the original radar track point and the fused track point is greater than a preset threshold P... th When considering the uncertainty of the track, the entropy H(P) of the correlation probability is calculated as an uncertainty measure to provide decision support. If the correlation probability P of the original radar track point is greater than the threshold P... th And the entropy H(P) is less than the set entropy threshold H t If the entropy H(P) is greater than H, then it is confirmed as a fused track associated with the original radar track; otherwise, if the entropy H(P) is greater than H th If the value is zero, it indicates that the track point has a high degree of uncertainty and is marked as a potential new target track.

[0083] Furthermore, S6 specifically involves fusing data from multiple radars for a single target and using filtering algorithms to generate the target's future state, including its position, direction of motion, and velocity. Specifically: if the fused track is a new fused track or the number of track points is less than 3, the original track information of the current radar is output, and this original track information serves as the new track point for the associated fused track; if the number of fused track points is greater than 3, maneuvering target filtering is applied to the track points before time t-1 and the original track points of the current radar to generate a state vector at time t, which is then used as the new track point for the associated fused track, replacing the track point extrapolated to time t, and the new track point information for the fused track is output. Maneuvering target filtering includes interactive multi-model filtering algorithms, Singer model filtering algorithms, etc.

[0084] and Figure 1Corresponding to the method described above, this invention also discloses a multi-source track data fusion system based on an LSTM network. The system executes the aforementioned multi-source track data fusion method based on an LSTM network, comprising: a time registration module, a track probability calculation module, a track association management module, a track quality management module, and a track state estimation module. The time registration module registers the fused track points; the track probability calculation module calculates the correlation probability between the original radar track points and the existing fused track points; the track association management module classifies the original radar track points; the track quality management module manages the quality of the track classification results; and the track state estimation module estimates the future state of the target.

[0085] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0086] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-source track data fusion method based on LSTM networks, characterized in that, The following steps are involved: S1. Using the time t of the original radar track point as a reference, the existing fused track point at time t-1 is extrapolated to time t, so that the original radar track point and the fused track point are comparable at the same time point. S2. Normalize the state vectors of the original radar track points and the state vectors of any track point in the fused track set, construct a time series that simultaneously contains the original radar track points and the fused track points as the input sequence of the LSTM, and traverse all track points in the fused track set. S3. Use LSTM to determine the correlation between the original radar track points and the existing fused track points, and calculate the probability that they are the same target. S4. Based on the information of relevant probabilities, original radar track points and fused track points, classify the original radar track points, and classify them into new fused tracks or fused tracks associated with the original radar tracks according to the relevant probability threshold. S5. Calculate the average probability of the sum of the correlation probabilities between multiple original radar track points of the target and existing fused track points. If the average probability between the original radar track points in the target's original radar track and the fused track points in a fused track is greater than a preset threshold, associate the current original radar track with the fused track. If the average probability between the original radar track points in the target's original radar track and the fused track points in all fused tracks is less than a preset threshold, then determine that the original radar track point is the track point of a newly discovered target, establish a fused track file, and generate a new fused track. If there is no data update for r consecutive detection cycles, then delete the track file. S6. By fusing data from multiple radars for a single target, a filtering algorithm is used to estimate the target's future state.

2. The multi-source track data fusion method based on LSTM network according to claim 1, characterized in that, The original radar trackpoint set in S1 is U l ={l1,l2,…,l m }, the merged waypoint set is U l ={l1,l2,…,l m }, where n is the number of fused track points, m is the number of original radar track points, and the track point of fused track i at time t is} The state vector, the original track j of radar l at time t is The state vector, Merge existing track points at time t-1 Extrapolation 3. The multi-source track data fusion method based on LSTM network according to claim 1, characterized in that, S2 specifically refers to: the original radar track point state vector With fused track set U s any waypoint in state vector Normalization is performed to obtain the normalized state vector. and For each element x in the waypoint state vector, its normalized value x′ is: Construct a time series that simultaneously contains both the original radar track points and the fused track points. As the input sequence for LSTM: In the formula, It is the number of radar scan cycles contained in the track, s l (τ) is the input feature vector corresponding to time point τ, s l (τ) is derived from the normalized radar original track point feature vector. and normalized fused track point feature vector Combination structure:

4. The multi-source track data fusion method based on LSTM network according to claim 1, characterized in that, S3 specifically refers to: S31. The constructed input sequence is processed through an LSTM network to obtain the probability distribution. In the formula, Indicates the original radar track point Merging track point with the i-th merging track The correlation probability between them, W p and b p These are the weight matrix and bias vector of the Softmax layer, respectively. S32. Considering that relying solely on probability thresholds may lead to misjudgments of minority classes of tracks, an entropy H(P) is used to measure the matching confidence between the original radar track points and the fused track points: In the formula, H(P) represents the uncertainty of the correlation between the original radar track point and multiple fused track points. The larger the entropy value, the more uniform the distribution and the lower the confidence level; the smaller the entropy value, the more the probability is concentrated in a certain fused track point and the higher the confidence level.

5. The multi-source track data fusion method based on LSTM network according to claim 1, characterized in that, S4 specifically refers to: based on the relevant probability and a preset threshold P th Information including entropy H(P), original radar track points, and fused track points is used to classify the original radar track points. When the correlation probability between the original radar track points and the fused track points is less than a preset threshold P, the classification is performed. th If no fused track is found, the track point is determined to be the track point of the newly discovered target. A fused track file is established, and a new fused track is generated. The current original track point is used as the track point of the new track. When the correlation probability between the original radar track point and the fused track point is greater than a preset threshold P... th When considering the uncertainty of the track, the entropy H(P) of the correlation probability is calculated as an uncertainty measure to provide decision support; if the correlation probability P of the original radar track point is greater than the threshold P... th And the entropy H(P) is less than the set entropy threshold H t If the entropy H(P) is greater than H, then it is confirmed as a fused track associated with the original radar track; otherwise, if the entropy H(P) is greater than H th If the value is zero, it indicates that the track point has a high degree of uncertainty and is marked as a potential new target track.

6. The multi-source track data fusion method based on LSTM network according to claim 1, characterized in that, S6 specifically refers to: generating the future state of a target, including its position, direction of motion, and velocity information, by fusing data from multiple radars for a single target and using a filtering algorithm.

7. A multi-source trajectory data fusion system based on LSTM networks, characterized in that, The method for multi-source track data fusion based on LSTM network according to any one of claims 1-6 includes: a time registration module, a track probability calculation module, a track association management module, a track quality management module, and a track state estimation module; the time registration module registers the fused track points, the track probability calculation module calculates the correlation probability between the original radar track points and the existing fused track points, the track association management module classifies the original radar track points, the track quality management module manages the quality of the track classification results, and the track state estimation module estimates the future state of the target.

Images