LSTM track filtering method and system based on man-machine cooperation track data set, and application

Abstract

The invention relates to the technical field of radar maneuvering target tracking, in particular to an LSTM track filtering method and system based on a man-machine cooperation track data set and application. Considering that in actual maneuvering target detection, a complex clutter environment and target diversity can seriously influence the tracking precision of a target, and a traditional tracking method is difficult to meet the requirements on data processing time delay and tracking precision. According to the invention, a high-quality track data set covering multiple target categories and complex environments is constructed through a man-machine cooperation mechanism and a fusion algorithm, accurate and automatic tracking of the maneuvering target is realized by using an LSTM technology based on the data set, and the tracking accuracy and real-time performance are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of radar maneuvering target tracking, and particularly to a track filtering method, system and application based on LSTM of a human-machine collaborative track data set. Background Art

[0002] In the field of radar maneuvering target tracking, the target diversity and environmental complexity caused by complex clutter environments have long affected the tracking accuracy. Traditional tracking methods are limited by data processing and algorithm accuracy and are difficult to meet actual requirements. Therefore, there is an urgent need for an innovative method to achieve accurate and automatic tracking by constructing a high-quality data set and combining advanced algorithms. The present invention aims to construct a high-quality track data set covering various target categories and complex environments through a human-machine collaborative mechanism, and use LSTM technology based on this data set to achieve accurate and automatic tracking of maneuvering targets, improving the accuracy and real-time performance of tracking. Summary of the Invention

[0003] The object of the present invention is to propose a track filtering method, system and application based on LSTM of a human-machine collaborative track data set for the problems existing in the background art; aiming to provide a comprehensive solution combining the construction of a high-quality track data set through human-machine collaboration and track filtering technology based on long short-term memory network (LSTM) to efficiently and accurately address the tracking challenges in complex clutter environments.

[0004] The technical solution of the present invention

[0005] The first aspect of the present invention provides a track filtering method based on LSTM of a human-machine collaborative track data set, including the following specific steps:

[0006] S1. Construct a human-machine interaction system including a human-machine interaction interface, allowing experts to annotate and correct the track data generated by a radar target detection algorithm;

[0007] S2. Use an interactive multiple model kalman filter to obtain the filtered position data and covariance matrix of the original track data;

[0008] S3. Obtain optical image data; use a convolutional neural network CNN to perform target feature extraction and positioning on the image data;

[0009] S4. Use the same interactive multiple model kalman filter to obtain the filtered result and covariance matrix of the image positioning data;

[0010] S5. Use the covariance intersection algorithm CI to fuse and process the original track data after manual annotation and image processing data to obtain the fused position result and the covariance matrix of the fused data;

[0011] S6. Output the track data;

[0012] S7. Clean and preprocess the track data obtained in step S6, and perform normalization processing.

[0013] S8. Use the normalized data to construct a training data set.

[0014] S9. Construct an LSTM training model based on incremental learning.

[0015] S10. Use the trained LSTM model to output the filtered position of the track.

[0016] In step S1, the human-computer interaction interface includes a data visualization module and an annotation tool set correction feedback module, which are used to annotate and process the track data.

[0017] In step S2, let the positioning data of the original track be X o =(x, y), and the position data set corrected by manual annotation is X ro ={(x roi , y roi ), i = 1, 2,..., N r , where i is the sample point serial number and N r is the number of data samples; use the interactive multiple model kalman filter to obtain the position data X r ={(x ri , y ri ), i = 1, 2,..., N} and the covariance matrix R r , where

[0018]

[0019] The values on the diagonal of R rxx , R ryy respectively represent the variances of the radar detection positions x and y variables, and the values on the non-diagonal of R rxy , R ryx are the covariances between the radar detection positions x and y, indicating the correlation between x and y.

[0020] In step S2, three basic models of constant velocity linear motion CV, constant acceleration linear motion CA, and constant velocity turning CT are used to divide the motion states.

[0021] In step S3, use an optical imaging device to capture the target image and provide optical image data for the data set.

[0022] In step S4, let the image data captured by the optical sensor be I, and the target position information obtained through image processing technology be X po ={(x poi , y poi), i = 1, 2, ..., N p , where N p is the number of data samples;

[0023] Based on the calibrated original filtered data, align the time, and use the same interacting multiple model Kalman filter to obtain the filtered result of the image positioning data as X p = {(x pi , y pi ), i = 1, 2, ..., N and the covariance matrix R p , where

[0024]

[0025] The values on the diagonal of R pxx , R pyy represent the variances of the x and y variables of the image detection position respectively. The values on the off-diagonal of R pxy , R pyx are the covariance between the x and y of the image detection position, indicating the correlation between x and y.

[0026] Using the covariance intersection algorithm CI, fuse and process the original track data and image processing data after manual annotation to obtain the fused position X = {(x i , y i ), i = 1, 2, ..., N, where i is the sample point serial number and N is the number of samples after the radar detection and image detection time alignment.

[0027]

[0028] R is the covariance matrix of the fused data:

[0029]

[0030] The values on the diagonal of R xx , R yy are the variances of the x and y variables of the fused position respectively. The values on the off-diagonal of R xy , R yx are the covariance between the x and y of the fused position, indicating the correlation between x and y.

[0031] In step S9, create an LSTM model. Use the continuous 5-point time, position plus the next point time to construct the input layer, with 16-dimensional features and 2-dimensional output features. The number of stacked LSTM layers is 1 layer, and the number of units in the hidden layer is adjusted to the optimal value according to the experiment;

[0032] Set the training parameters: set the initial learning rate and adopt the learning rate decay algorithm; obtain the optimal model through iterative training.

[0033] The second aspect of the present invention provides a track filtering system based on LSTM for a human-machine collaborative track data set, using the above method, including a data acquisition module, a signal-level processing module, a data-level processing module, and a human-machine interaction module;

[0034] Among them, the data acquisition module includes a millimeter-wave radar and a high-definition camera; the millimeter-wave radar is used to collect radar track data, and the high-definition camera is used to collect optical image data;

[0035] The signal-level processing module includes a radar echo detection and positioning unit and an image feature extraction and positioning unit; the radar echo detection and positioning unit is used to process the collected radar track data to detect and position the target track; the image feature extraction and positioning unit is used to extract features from the collected optical image data and perform track positioning;

[0036] The data-level processing module includes a multi-mode Kalman filtering unit, a multi-source track fusion unit, and an LSTM training unit based on incremental learning; the multi-mode Kalman filtering unit is used to filter the collected original track data and optical positioning image data to obtain position data and covariance matrices, filtering results and covariance matrices; the multi-source track fusion unit is used to fuse and process the original track data and image processing data after manual annotation to obtain a fused position result and the covariance matrix of the fused data; the LSTM training unit based on incremental learning is used to train the established model;

[0037] The human-machine interaction module includes a traffic situation map display interface, a manual annotation and modification module; the traffic situation map display interface is used to show the user the real-time situation of the current monitoring area; the manual annotation and modification module is used to manually annotate and modify the original data.

[0038] The third aspect of the present invention provides an application, using the above method and the above system, which is applied to tracking one or more moving targets such as ships, drones, vehicles, and pedestrians.

[0039] Compared with the prior art, the present invention has the following beneficial technical effects:

[0040] (1) The quality of the data set is significantly improved: Through human-machine collaboration and the fusion of optical auxiliary information, the constructed high-quality data set provides a solid data foundation for the training of the LSTM model.

[0041] (2) Precise automatic tracking: The track filtering technology based on LSTM realizes precise automatic tracking of moving targets, improving the tracking accuracy and real-time performance.

[0042] (3) Wide application prospects: The present invention has a wide range of applications. It is not only applicable to traditional ships and UAV tracking, but can also be extended to other mobile targets such as vehicles and pedestrians, with important application value and broad market prospects. Description of the Drawings

[0043] Figure 1 It is a flowchart of the track filtering method based on LSTM of the present invention for the human-machine collaborative track data set;

[0044] Figure 2 It is a flowchart for constructing a high-quality track data set in an embodiment of the present invention;

[0045] Figure 3 It is a flowchart for online training and updating of LSTM based on incremental learning in an embodiment of the present invention;

[0046] Figure 4 It is a diagram of the module composition of the urban traffic monitoring system in an embodiment of the present invention;

[0047] Figure 5 It is the target track obtained by simulated radar detection and image detection;

[0048] Figure 6 It is the target track after the fusion of simulated radar detection and image detection;

[0049] Figure 7 It is the filtered track based on LSTM by simulation. Detailed Implementation Manner

[0050] Embodiment 1

[0051] 1. Method for constructing a high-quality track data set for human-machine collaboration:

[0052] (1) Human-machine collaboration mechanism

[0053] A: Efficient human-machine interaction system: Design a system with an intuitive user interface that allows experts to easily annotate and correct the track data generated by the algorithm. The user interface includes a data visualization module, an annotation tool set, and a correction feedback mechanism to ensure the accuracy and integrity of the data set.

[0054] B: Implementation details: Let the positioning data of the original track be X o =(x, y), and the corrected position data set through manual annotation is X ro ={(x roi , y roi}, i = 1, 2,..., N r , where i is the sample point serial number, and N ris the number of data samples. Using the three basic models of constant velocity (CV), constant acceleration (CA), and constant turn (CT), the filtered position data X of the original data is obtained by using the interacting multiple model Kalman filter r ={(x ri , y ri ),}, i = 1, 2, ..., N and the covariance matrix R r , where

[0055]

[0056] The values on the diagonal, R rxx and R ryy represent the variances of the radar detection position variables x and y respectively. The values on the off-diagonal, R rxy and R ryx are the covariances between the radar detection positions x and y, indicating the correlation between x and y.

[0057] (2) Optical auxiliary information fusion

[0058] A: Application of optical sensors: Optical devices such as high-definition cameras are used to capture target images, providing rich visual information for the dataset.

[0059] B: Image processing technology: Convolutional neural networks (CNNs) are used to extract image features and fuse them with radar data to improve the quality and richness of the dataset.

[0060] C: Implementation details: Let the image data captured by the optical sensor be I, and the target position information obtained through image processing technology be X po ={(x poi , y poi ),}, i = 1, 2, ..., N p , where N p is the number of data samples. Aligning the time with the calibrated original filtered data as the reference, the filtered result of the image positioning data is obtained by using the same interacting multiple model Kalman filter as X p ={(x pi , y pi ),}, i = 1, 2, ..., N and the covariance matrix R p , where

[0061]

[0062] The values on the diagonal, R pxx and R pyy represent the variances of the image detection position variables x and y respectively. The values on the off-diagonal, R pxy and R pyxIt is the covariance between the image detection positions x and y, representing the correlation between x and y.

[0063] Using the covariance intersection (CI) algorithm, the original track data after manual annotation and the image processing data are fused to obtain the fused position X = {(x i , y i ), i = 1, 2,..., N, where i is the sample point sequence number and N is the number of samples after the radar detection and image detection times are aligned.

[0064]

[0065] R is the covariance matrix of the fused data:

[0066]

[0067] The values on the diagonal, R xx , R yy are the variances of the fused position x and y variables respectively, and the values on the off-diagonal, R xy , R yx are the covariance between the fused position x and y, representing the correlation between x and y.

[0068] (3) Diversity and complexity guarantee

[0069] A: Dataset design principle: Cover various target types such as aircraft, birds, drones, people, vehicles, ships, etc., as well as various environmental conditions such as urban traffic, airport runways, and ocean channels.

[0070] B: Implementation strategy: Through a carefully designed acquisition scheme and data processing method, ensure that the dataset has broad representativeness and practical application value.

[0071] 2. LSTM-based track filtering technology

[0072] (1) High-quality dataset support

[0073] A: Dataset quality guarantee: The high-quality dataset constructed using human-machine collaboration and optical auxiliary information provides reliable training data for the LSTM model.

[0074] B: Data preprocessing: Strictly clean and preprocess the track data to ensure data integrity and consistency. At the same time, perform normalization processing on all position data:

[0075]

[0076] where (t min , x min , y min ), (t max , x max,y max are the minimum and maximum values of the time t and the position coordinates (x, y), respectively.

[0077] C: Implementation details: Construct the input time series with 5 consecutive points. Since the time intervals are not necessarily consistent, the time is also included in the input sequence. The input layer consists of 16 - dimensional features, and the position of the next point is used as the result to construct the output.

[0078] X in ={(t i ' -5 ,x i ' -5 ,y i ' -5 ),(t i ' -4 ,x i ' -4 ,y i ' -4 ,(t i ' -3 ,x i ' -3 ,y i ' -3 ,),(t i ' -2 ,x i ' -2 ,y i ' -2 ,(t i ' -1 ,x i ' -1 ,y i ' -1 ,),t i '}, i = 6, 7,..., N (4)

[0079] X out ={(x i ', y i ')}, i = 6, 7,..., N (5)

[0080] Create an LSTM model with 16 - dimensional input features and 2 - dimensional output features. The number of stacked LSTM layers is 1, and the number of units in the hidden layer is adjusted to the optimal value according to experiments. Set the training parameters: Set the initial learning rate and use the learning rate decay algorithm. Obtain the optimal model through iterative training.

[0081] (2) Model optimization and online update

[0082] A: Online update mechanism: Combine the incremental learning (IL) technology to achieve online model update and continuously optimize the LSTM model parameters.

[0083] B: Implementation details: Collect the latest N IL pieces of data, construct N IL training samples by imitating formulas (3), (4), and (5), and use the incremental training samples to train and update the parameters of the pre-trained model. Finally, input the data from N+N IL -4 to N+N IL into the trained model, and after anti-normalizing the output data, obtain the position data at the N+N IL +1 moment, as shown in formula (6). When there is continuously updated data, continuously repeat this process to alternately implement the training and prediction processes.

[0084]

[0085] where (x out , y out ) is the output value of the model, and is the predicted value of the position.

[0086] The method of this embodiment can be widely applied: applicable to tracking various mobile targets such as ships, unmanned aerial vehicles, vehicles, pedestrians, etc., improving the technical practicality and market competitiveness. For example: it shows excellent performance and broad application prospects in fields such as urban traffic monitoring, airport security management, and maritime navigation.

[0087] Embodiment 2

[0088] This embodiment provides a trajectory filtering system based on LSTM of a human-machine collaborative trajectory dataset, which is processed using the method in Embodiment 1, and specifically includes a data acquisition module, a signal-level processing module, a data-level processing module, and a human-machine interaction module;

[0089] Among them, the data acquisition module includes a millimeter-wave radar and a high-definition camera; the millimeter-wave radar is used to collect radar trajectory data, and the high-definition camera is used to collect optical image data;

[0090] The signal-level processing module includes a radar echo detection and positioning unit and an image feature extraction and positioning unit; the radar echo detection and positioning unit is used to process the collected radar trajectory data to detect and locate the target trajectory; the image feature extraction and positioning unit is used to extract features from the collected optical image data and perform trajectory positioning;

[0091] The data-level processing module includes a multi-mode Kalman filtering unit, a multi-source track fusion unit, and an LSTM training unit based on incremental learning. The multi-mode Kalman filtering unit is used to filter the collected original track data and optical positioning image data to obtain position data, covariance matrices, filtering results, and covariance matrices. The multi-source track fusion unit is used to fuse and process the original track data and image processing data after manual annotation to obtain the fused position result and the covariance matrix of the fused data. The LSTM training unit based on incremental learning is used to train the established model.

[0092] The human-computer interaction module includes a traffic situation map display interface, a manual annotation and modification module. The traffic situation map display interface is used to show the user the real-time situation of the current monitoring area. The manual annotation and modification module is used to manually annotate and modify the original data.

[0093] The following uses a specific simulation case to introduce this solution in detail:

[0094] Suppose two ship targets move in a plane, with initial positions at [0, 12] km and [0, 10] km respectively. The movement process is divided into three segments: (1) Uniform linear motion, with headings both being 90°, speeds both being 15 m / s, and the duration being 400 s; (2) Uniform turning motion, the turning rate of target 1 is w 1 =-π / 400 rad / s, the turning rate of target 2 is w 2 =π / 400 rad / s, the duration is 400 s, and the speeds at the end of the stage are v t,1 , v t,2 respectively; (3) Uniform linear motion, the speeds of the two targets are kept as v t,1 , v t,2 respectively, and the duration is 400 s.

[0095] The detection errors of the radar in the x and y directions follow a Gaussian distribution with a mean of 80 m and a mean square deviation of 100 m, the detection period is 20 s, and the detection probability P d =0.6; the detection errors of the image processing track in the x and y directions follow a Gaussian distribution with a mean of 50 m and a mean square deviation of 50 m, the detection period is 50 s, and the detection probability P d ' = 1. The detected tracks are as Figure 5 shown, where the blue track is the radar detection track and the black track is the image detection track.

[0096] The filtered tracks X r , X p of the radar and the image and the covariance matrices R r , R p are obtained through Kalman filtering, and the fused track X is obtained using the CI fusion algorithm, asFigure 6 as shown, so as to construct a high-quality data set for LSTM model training.

[0097] Use formula (3) to normalize all data to [0, 1]. Construct the input time series with 5 consecutive points. Since the time intervals are not necessarily consistent, the time is also put into the input sequence. The input layer consists of 16-dimensional features, and the result of the next point is used to construct the output result. See formulas (4) and (5).

[0098] Create an LSTM model: 16-dimensional input layer, 128 neurons, ReLU activation layer, fully connected layer with 2 output units, and regression layer. Set the training parameters: maximum number of training times 200, gradient threshold 1, initial learning rate 0.005, train using the learning rate adjustment method, adjust the learning rate after 150 training times, learning rate adjustment factor 0.1, regularization parameter 10 -4 . After the training is completed, make predictions at intervals of 10s, and use formula (6) to denormalize the predicted values to obtain the filtered output result When a high-quality data set enters the system again, update the training on the existing net to obtain the latest net for predicting the subsequent positions of the ships.

[0099] Finally, the filtering result using LSTM is as Figure 7 shown. This example shows through simulation data that the technical gains generated by the trajectory filtering system after applying this technology are: (1) the trajectory point set of the ship target is denser, and the real-time performance of target tracking is higher; (2) the position tracking of the target is more accurate, the motion trend is more obvious, and the data quality is higher.

[0100] The above examples only perform simulations with two-dimensional data and can track targets such as ships, pedestrians, and cars; the method of this solution is also applicable to three-dimensional data and can track targets such as drones; just directly change the positioning data of the input trajectory of the model to three-dimensional data.

[0101] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to this. Various changes can be made without departing from the spirit of the present invention within the scope of knowledge possessed by those skilled in the art.

Claims

1. A track filtering method based on LSTM of human-machine collaborative track data set, characterized in that: The specific steps include: S1. Build a human-computer interaction system including a human-computer interaction interface to allow experts to annotate and correct the track data generated by the radar target detection algorithm; S2, using interactive multi-model Kalman filtering to obtain the position data and covariance matrix of the original track data after filtering; S3, obtain optical image data; use convolutional neural network (CNN) to extract and locate target features from image data; S4, using the same interactive multi-model Kalman filter to obtain the filtering results and covariance matrix of the image positioning data; S5, using the covariance intersection algorithm CI, the original track data and the image processing data after manual annotation are fused to obtain the fused position result and the covariance matrix of the fused data; S6, output track data; S7, cleaning and preprocessing the track data obtained in step S6, and normalizing it; S8. Use the normalized data to construct a training data set; S9. Build an LSTM training model based on incremental learning; S10. Use the trained LSTM model to output the track filtering position.

2. The LSTM track filtering method based on the human-machine collaborative track data set according to claim 1 is characterized in that: The human-computer interaction interface in step S1 includes a data visualization module and a labeling tool set correction feedback module, which are used to label and process the track data.

3. The track filtering method based on LSTM of human-machine collaborative track data set according to claim 1 is characterized in that: In step S2, the positioning data of the original track is assumed to be X o =(x,y), the position data set after manual annotation correction is X ro ={(x roi ,y roi )},i=1,2,...,N r , where i is the sample point number, N r is the number of data samples; the position data X after the original data is filtered is obtained by using interactive multi-model Kalman filtering r ={(x ri ,y ri )}, i = 1, 2, ..., N and the covariance matrix R r ,in The value R on the diagonal rxx , R ryy Respectively represent the variance of the radar detection position x and y variables, and the value R on the off-diagonal line rxy , R ryx is the covariance between the radar detection positions x and y, indicating the correlation between x and y.

4. The track filtering method based on LSTM of human-machine collaborative track data set according to claim 3 is characterized in that: In step S2, three basic models of uniform linear motion CV, uniform accelerated linear motion CA and uniform turning CT are used to divide the motion state.

5. The track filtering method based on LSTM of human-machine collaborative track data set according to claim 1 is characterized in that: In step S3, an optical camera device is used to capture a target image, providing optical image data for the data set.

6. The LSTM track filtering method based on the human-machine collaborative track data set according to claim 1 is characterized in that: In step S4, the image data captured by the optical sensor is denoted as I, and the target position information obtained by the image processing technology is denoted as X. po ={(x poi ,y poi )},i=1,2,...,N p , where N p is the number of data samples; Taking the calibrated original filtered data as the benchmark, the time is aligned and the filtering result of the image positioning data is obtained by using the same interactive multi-model Kalman filter as X p ={(x pi ,y pi )}, i = 1, 2, ..., N and the covariance matrix R p ,in The value R on the diagonal pxx , R pyy Respectively represent the variance of the image detection position x, y variables, and the value R on the off-diagonal line pxy , R pyx is the covariance between the image detection positions x and y, indicating the correlation between x and y.

7. The track filtering method based on LSTM of human-machine collaborative track data set according to claim 3 or 6, characterized in that: The covariance crossover algorithm CI is used to fuse the original track data and image processing data after manual annotation to obtain the fusion position X = {(x i ,y i )}, i = 1, 2, ..., N, where i is the sample point number and N is the number of samples after the radar detection and image detection are aligned in time. R is the covariance matrix of the fused data: The value R on the diagonal xx , R yy The variance of the fusion position x and y variables, and the value R on the off-diagonal line xy , R yx is the covariance between the fusion positions x and y, indicating the correlation between x and y.

8. The LSTM track filtering method based on the human-machine collaborative track data set according to claim 1 is characterized in that: In step S9, an LSTM model is created, and an input layer is constructed with 5 consecutive time points and positions plus the next time point, with 16-dimensional features and 2-dimensional output features. The number of LSTM stacking layers is 1, and the number of units in the hidden layer is adjusted to the optimal value according to experiments; Set training parameters: set the initial learning rate and use the learning rate decay algorithm; obtain the optimal model through iterative training.

9. A track filtering system based on LSTM of human-machine collaborative track data set, using the method described in any one of claims 1 to 8, characterized in that: It includes data acquisition module, signal level processing module, data level processing module and human-computer interaction module; The data acquisition module includes a millimeter-wave radar and a high-definition camera; the millimeter-wave radar is used to collect radar track data, and the high-definition camera is used to collect optical image data; The signal level processing module includes a radar echo detection and positioning unit and an image feature extraction and positioning unit; the radar echo detection and positioning unit is used to process the collected radar track data to detect and locate the target track; the image feature extraction and positioning unit is used to extract features from the collected optical image data and perform track positioning; The data-level processing module includes a multi-mode Kalman filter unit, a multi-source track fusion unit, and an LSTM training unit based on incremental learning; the multi-mode Kalman filter unit is used to filter the collected original track data and optical positioning image data to obtain position data and covariance matrix, filtering results and covariance matrix; the multi-source track fusion unit is used to fuse the manually annotated original track data and image processing data to obtain the fused position result and the covariance matrix of the fused data; the LSTM training unit based on incremental learning is used to train the established model; The human-computer interaction module includes a traffic situation map display interface and a manual annotation and modification module; the traffic situation map display interface is used to show the user the real-time situation of the current monitoring area; the manual annotation and modification module is used to manually annotate and modify the original data.

10. An application using the method according to any one of claims 1 to 8 and the system according to claim 9, characterized in that: Applied to tracking one or more maneuvering targets such as ships, UAVs, vehicles, and pedestrians.

Images