A millimeter wave radar multi-person tracking method based on interrupted track association

Abstract

The application relates to a millimeter wave radar multi-person tracking method based on interrupted track association, which comprises the following steps: obtaining millimeter wave radar echo data and performing pretreatment to obtain millimeter wave radar point cloud data; performing multi-target tracking on the millimeter wave radar point cloud data to obtain multi-target tracking tracks and corresponding multi-target time sequence point cloud data; constructing a new track set and an old track set and updating the same in the tracking process; when the updated new track set and the updated old track set are not empty, inputting the new track segment and the old track segment into a trained feature extraction network to obtain new track segment features and old track segment features; calculating the Euclidean distance between the new track segment features and the old track segment features to obtain a new-old track set matching cost matrix; and matching the new track segment and the old track segment according to the new-old track set matching cost matrix to obtain a track segment association result. The method solves the problem that the existing technology is prone to track interruption in practice.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of target tracking, and particularly relates to a millimeter wave radar multi-person tracking method based on interrupted track association. BACKGROUND

[0002] In recent years, the fields of safety protection, smart home and automatic driving develop rapidly, and accurate positioning and tracking of multiple moving targets are crucial in various fields. Pedestrians are very complex and agile targets, so multi-person target tracking is a challenging task in multi-target tracking, and has been widely concerned by domestic and foreign researchers. Millimeter wave radar has become one of the important sensors in multi-person target tracking because of its good resolution in distance, radial velocity and angle estimation, so it has important research significance to realize accurate tracking in multi-person scenes based on millimeter wave radar.

[0003] In recent years, the research on multi-person target tracking based on millimeter wave radar is also increasingly hot: Pegoraro et al. used millimeter wave radar to track multi-person targets and proposed an extended Kalman filter tracker to estimate the position, shape and extension of personnel targets. Rossi et al. tracked and identified indoor personnel based on millimeter wave radar signals, used Kalman filter and deep neural network for trajectory tracking and identification, so that the interaction between tracking and identification was realized, and the tracking accuracy and identification accuracy were improved. Li et al. proposed a multi-algorithm fusion framework (MFA) using mean shift algorithm and Kalman filter to track multi-person targets using millimeter wave radar. Knudde et al. focused on processing low signal-to-noise ratio and multipath effect, used Markov chain-based Monte Carlo data association method (MCMCDA) to complete the multi-person target tracking task in indoor environment. It can be seen that the multi-person target tracking method based on millimeter wave radar has been widely studied by many scholars. In addition, in view of the track interruption problem in multi-target tracking process, researchers have proposed a series of interrupted track association methods: Yeom et al. introduced interactive multiple model (IMM) in track segment association, which improved the tracking continuity of maneuvering targets in airborne early warning systems. In order to improve the utilization of prior information and reduce the dependence on motion model, document one uses position, velocity, acceleration and other prior information to connect the coarsely associated track segments through polynomial fitting, which effectively improves the continuity of track segments. In document two, the actual end time and start time of old and young tracks are also considered, and multi-dimensional (S-D) assignment is applied to track segment association. All the above track segment association algorithms are model-driven methods, which use a prior dynamic model to predict or backtrack track segments. When a prior model does not match the actual motion, the performance of track segment association will inevitably decline. With the rapid development of artificial intelligence, data-driven methods are gradually applied to track segment association: document three uses a tracking segment association double contrast neural network (TSADCNN) to complete the tracking segment association task, which achieves good results on simulation data.

[0004] The existing patent document "Multi-person tracking method based on millimeter wave radar" proposes a method for tracking multiple targets using millimeter wave radar. This method uses a multi-mode tracking method based on millimeter wave radar point cloud data, and different tracking methods are used for tracks with different tracking difficulties. The results of different tracking methods are finally fused to achieve multi-person tracking. However, this method still has some shortcomings. The method does not consider the complex situations such as mutual occlusion and close proximity of people in a multi-person scene. These complex situations can cause track interruption and other problems. When the multi-person tracking track is frequently interrupted, multiple short tracks with different IDs for the same target are generated, significantly increasing track fragmentation and seriously affecting downstream tasks: it is not conducive to behavior analysis and prediction of key figures, and to a large extent, it limits the application of the method in actual scenarios. Therefore, it is of great significance to propose a multi-person tracking method based on millimeter wave radar for track interruption.

[0005] The People's Liberation Army Navy Aviation University discloses an end-to-end interrupted track connection association deep learning method in its applied patent document "Interrupted track connection association deep learning method". The method mainly solves the problem that existing interrupted track association models require a large amount of prior information in actual engineering applications and are difficult to apply directly. However, this method still has some shortcomings. The method only uses the target state information (point track data) obtained by tracking. When the target is a person, due to the complex motion form and strong maneuverability of the person target, it is difficult to achieve the expected effect by only describing the motion characteristics.

[0006] Therefore, in combination with the rich measurement information of millimeter wave radar, a suitable interrupted track association method is designed to improve the track association accuracy and achieve robust multi-person target tracking, which is a problem to be solved at present. SUMMARY

[0007] To solve the above problems in the prior art, the present application provides a millimeter wave radar multi-person tracking method based on interrupted track association. The technical problem to be solved by the present application is solved by the following technical scheme:

[0008] The embodiment of the present application provides a millimeter wave radar multi-person tracking method based on interrupted track association, comprising the following steps:

[0009] Obtain millimeter wave radar echo data and perform preprocessing to obtain millimeter wave radar point cloud data;

[0010] Perform multi-target tracking on the millimeter wave radar point cloud data to obtain multi-target tracking tracks and their corresponding multi-target time sequence point cloud data, wherein a new track set and an old track set are constructed and updated during tracking;

[0011] When the updated new track set and the updated old track set are not empty, new track segments in the updated new track set and old track segments in the updated old track set are input into a trained feature extraction network to obtain new track segment features and old track segment features, wherein the trained feature extraction network extracts target subject motion features corresponding to point track data in new track segments and old track segments and target detail contour features corresponding to time sequence point cloud data, and splices and fuses the target subject motion features and the target detail contour features;

[0012] An Euclidean distance between the new track segment features and the old track segment features is calculated in a feature space to obtain a new-old track set matching cost matrix;

[0013] The new track segments and the old track segments are matched according to the new-old track set matching cost matrix to obtain a track segment association result.

[0014] In an embodiment of the present application, millimeter wave radar echo data is acquired and preprocessed to obtain millimeter wave radar point cloud data, including:

[0015] Millimeter wave radar echo data of a multi-person scene is acquired;

[0016] The millimeter wave radar echo data is sequentially subjected to orthogonal mixer mixing, low-pass filtering, discrete time sampling, arrangement according to fast time dimension sampling point number, slow time dimension sampling point number and antenna channel number, discrete time Fourier transform, peak detection and coordinate conversion to obtain the millimeter wave radar point cloud data:

[0017]

[0018] wherein [X, Y, Z] is a three-dimensional coordinate of a target point, V is a speed of the target point, θ is a pitch angle of the target, and φ is an azimuth angle of the target.

[0019] In an embodiment of the present application, the millimeter wave radar point cloud data is subjected to multi-target tracking to obtain multi-target tracking tracks and corresponding time sequence point cloud data, including:

[0020] Each frame of millimeter wave point cloud data is clustered by using a DBSCAN algorithm, and a centroid of each clustered point track cluster is condensed to obtain an observation value set;

[0021] Based on the observation value set, a multi-target is tracked by combining a Kalman filtering algorithm and a global nearest neighbor data association algorithm to obtain multi-target tracking tracks;

[0022] Point cloud data of a position corresponding to each target in each frame is extracted through the multi-target tracking tracks to obtain multi-target time sequence point cloud data corresponding to the multi-target tracking tracks.

[0023] In one embodiment of the present application, a new track set and an old track set are constructed in the tracking process, and the new track set and the old track set are updated, comprising:

[0024] In the tracking process, the multi-target tracking track is monitored, and a new track set and an old track set are constructed according to the length of the monitoring window, and the new track set and the old track set are updated; wherein,

[0025] The new track set represents a track set after the interruption occurs, and the old track set represents a track set before the interruption occurs;

[0026] The new track set update criterion includes deleting associated tracks, deleting terminated tracks, and adding all un-terminated tracks with starting time within the monitoring window;

[0027] The update criterion of the old track set includes deleting associated tracks, deleting tracks falling from the monitoring window, and adding all tracks deleted from the new track set.

[0028] In one embodiment of the present application, the feature extraction network comprises a network input module, a point track feature extraction module, a point cloud feature extraction module and a feature fusion strategy module, wherein,

[0029] The network input module is used to input several frames of point track data and several frames of time sequence point cloud data corresponding to the point track data;

[0030] The point track feature extraction module is connected with the network input module, and is used to extract target subject motion features in the point track data to obtain point track features;

[0031] The point cloud feature extraction module is connected with the network input module, and is used to extract target detail contour features in the time sequence point cloud data to obtain point cloud features;

[0032] The feature fusion strategy module is connected with the point track feature extraction module and the point cloud feature extraction module, and is used to use a weighted splicing strategy to multiply the point track features and the point cloud features by different weights and splice them together to obtain a feature vector.

[0033] In one embodiment of the present application, the point track feature extraction module comprises a time sequence feature extraction module and a local and global feature extraction module connected in sequence, the time sequence feature extraction module comprises one or more of LSTM, RNN and Bi-LSTM; and the local and global feature extraction module comprises one or more of RestNet18 and a convolutional neural network.

[0034] The point cloud feature extraction module is constructed based on PointNet or PointNet++, and includes a spatial feature extraction branch, a speed feature extraction branch, a fusion module, a cutting module, an aggregation module, and a time sequence feature extraction module, wherein the spatial feature extraction branch and the speed feature extraction branch are parallel and are both connected to the fusion module, and the fusion module, the cutting module, the aggregation module, and the time sequence feature extraction module are connected in sequence.

[0035] In an embodiment of the present application, the training method of the feature extraction network comprises:

[0036] According to the total number of people in a multi-person scene, data of 2-person walking combinations, 3-person walking combinations, and 4-person walking combinations are collected, each combination including close walking, crossing walking, and free walking, to obtain millimeter wave radar training echo data;

[0037] The millimeter wave radar training echo data is preprocessed to obtain millimeter wave radar training point cloud data;

[0038] The millimeter wave radar training point cloud data is subjected to multi-target tracking to obtain training multi-target tracking tracks and corresponding training multi-target time sequence point cloud data;

[0039] According to the optical video collected synchronously with the millimeter wave radar training echo data, the training multi-target tracking tracks are judged, if any two tracks belong to different track segments of the same target, a label is set to 1, if any two tracks do not belong to different track segments of the same target, a label is set to 0, to form label data; the training data set is formed by the training multi-target tracking tracks, the training multi-target time sequence point cloud data, and the label data;

[0040] The training data set is subjected to standardization processing to obtain a standardized training data set;

[0041] The standardized training data set is input into the feature extraction network, and a ternary loss function is used to train the feature extraction network to obtain the trained feature extraction network, wherein during network training, a ternary group is constructed according to the training data set to calculate the ternary loss function each time the network parameters are updated.

[0042] In an embodiment of the present application, the ternary group is constructed according to the training data set to calculate the ternary loss function, comprising:

[0043] A track segment is randomly selected from the training data set as a reference sample;

[0044] According to the reference sample, a track segment with a label of 1 is randomly selected from the training data set as a positive example sample, and if there is no positive example sample in the training data set, the reference sample is reselected;

[0045] According to the reference sample, a track segment with a label of 0 is randomly selected from the training data set as a negative example sample;

[0046] The ternary loss function is calculated using the reference sample, the positive example sample and the negative example sample:

[0047] L = max (d (a, p) - d (a, n) + margin, 0)

[0048] Wherein, d represents calculating the Euclidean distance, a represents the reference sample, p represents the positive example sample of the same class as the reference sample, n represents the negative example sample of different classes from the reference sample, and margin represents a constant greater than 0.

[0049] In an embodiment of the application, the new and old track set matching cost matrix is:

[0050]

[0051]

[0052] Wherein, d represents calculating the Euclidean distance, x represents the xth track feature vector in the new track set, y represents the yth track feature vector in the old track set, and n represents the dimension of the feature vector.

[0053] In an embodiment of the application, the new track segment and the old track segment are matched according to the new and old track set matching cost matrix to obtain a track segment association result, comprising:

[0054] According to the new and old track set matching cost matrix, the new track segment and the old track segment are matched using the Hungarian matching algorithm to obtain a track segment association result.

[0055] Compared with the prior art, the application has the following advantages:

[0056] 1. In the multi-person tracking method of the present application, a new track set and an old track set are constructed and updated during the process of multi-target tracking. When the updated new track set and the updated old track set are not empty, it represents that there is an interrupted track. At this time, the trained feature network is used for interrupted track association operation, so that the method can monitor the track state when the multi-person tracking track is frequently interrupted, thereby associating the interrupted track. Finally, the tracks belonging to the same target broken track all maintain consistent target ID, solving the tracking track interruption problem that is easily encountered in practice and not considered in the prior art, greatly improving the continuity and robustness of multi-person target tracking, and being more suitable for actual millimeter wave radar multi-person tracking scenes such as safety protection, smart home, and automatic driving compared with the prior art.

[0057] 2. In the multi-person tracking method of the present application, the trained feature extraction network first extracts target main body motion features corresponding to the point track data and target detail contour features corresponding to the time sequence point cloud data, and then splices and fuses the target main body motion features and the target detail contour features. On the basis of the motion features, the contour features of the point cloud data are combined, so that the target track segment feature representation is more complete, and the accuracy of interrupted track association during multi-person target tracking can be effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 A flowchart of a millimeter wave radar multi-person tracking method based on interrupted track association provided by an embodiment of the present application is shown.

[0059] Figure 2 A millimeter wave radar multi-person tracking method based on interrupted track association provided by an embodiment of the present application is shown.

[0060] Figure 3 A multi-person tracking result example diagram based on millimeter wave radar point cloud in the X-Y plane provided by an embodiment of the present application is shown.

[0061] Figure 4 A schematic diagram of a feature extraction network model for an interrupted track association task provided by an embodiment of the present application is shown.

[0062] Figure 5 A schematic diagram of a point track feature extraction module provided by an embodiment of the present application is shown.

[0063] Figure 6 A schematic diagram of a point cloud feature extraction module provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0064] The present application will be further described in detail below in conjunction with specific embodiments, but the embodiments of the present application are not limited thereto.

[0065] Embodiment one

[0066] The embodiment constructs a millimeter wave radar multi-person tracking method based on interrupted track association, aiming at the track interruption problem caused by the large number of targets, complex motion state and cross occlusion between targets in a multi-person scene, and realizes association by measuring the similarity between track segments before and after interruption and one-to-one matching according to the similarity cost, solving the difficulty that the existing millimeter wave radar multi-person tracking method cannot associate interrupted tracks. Specifically, the method extracts track segment features through the design of interrupted track association feature extraction network, combines millimeter wave radar point clouds and target tracks obtained by tracking, and solves the problem that the current interrupted track association technology cannot be applied to multi-person target tracking; a millimeter wave radar multi-person walking training data set is constructed for network training process and method verification test process; the millimeter wave radar multi-person tracking method based on interrupted track association is designed, and robust tracking of multi-person targets is realized.

[0067] Please refer to Figure 1 and Figure 2 , Figure 1 The flowchart of the millimeter wave radar multi-person tracking method based on interrupted track association provided by the embodiment of the application is shown in Figure 2 The flowchart of the millimeter wave radar multi-person tracking method based on interrupted track association provided by the embodiment of the application is shown in. The millimeter wave radar multi-person tracking method based on interrupted track association comprises the following steps:

[0068] S1, obtaining millimeter wave radar echo data and preprocessing to obtain millimeter wave radar point cloud data.

[0069] Specifically, it comprises the following steps:

[0070] S11, obtaining millimeter wave radar echo data of a multi-person scene.

[0071] Specifically, the millimeter wave radar transmits a linear frequency continuous wave composed of continuous linear frequency signals, and the radar configuration parameters mainly include the number of transmitting antennas T x , the number of receiving antennas R x , the carrier frequency f c , the frequency modulation time τ, the frequency modulation slope S, the ADC sampling rate f s , the number of sampling points N s of each linear frequency signal, the linear frequency signal repetition period T c , the number of linear frequency signals contained in each frame N c , the total number of frames N f , and the observation time T c ×N c ×N fThese parameters are determined by the required distance resolution, unambiguous distance, speed resolution, unambiguous speed and other performance indicators in the actual measurement scene. For example, in an indoor scene, the T x transmit antennas periodically transmit a linear frequency modulated continuous wave, which is reflected by a walking human body and received by R x receive antennas. Each acquisition sample has T x ×R x channels of echo data.

[0072] S12, after obtaining the millimeter wave radar echo data to be tested in the actual scene, the millimeter wave radar echo data is preprocessed, which includes: sequentially performing orthogonal mixer mixing, low-pass filtering, discrete time sampling, arranging according to the number of fast time dimension sampling points, the number of slow time dimension sampling points and the number of antenna channels, discrete time Fourier transform, peak detection and coordinate conversion on the millimeter wave radar echo data, to obtain millimeter wave radar point cloud data.

[0073] The specific steps of preprocessing are:

[0074] S121, mixing the T x ×R x channel echo signals in each frame of millimeter wave radar echo data with the corresponding transmission signals through the orthogonal mixer, and performing low-pass filtering on the mixed signals to obtain the difference frequency signals in complex exponential form:

[0075]

[0076] wherein, is the echo signal received by the i-th channel, is the corresponding transmission signal, LPF is a low-pass filter, is the difference frequency signal of the i-th channel, is the frequency of the difference frequency signal, is the phase of the difference frequency signal, and A is the amplitude of the difference frequency signal.

[0077] The above operations are performed on the T x ×R x channel data to obtain T x ×R x channel difference frequency signals.

[0078] S122, the difference frequency signal obtained by the above steps is in analog signal form, in order to facilitate the subsequent digital signal processing process, the difference frequency signal is discretely sampled to convert it into a discrete time signal form and saved:

[0079]

[0080]

[0081] wherein, is the difference frequency signal in the form of the i-th channel discrete-time signal, N s is the number of discrete-time sampling points, is the discrete-time angular frequency, F s is the discrete-time sampling frequency.

[0082] The obtained T x ×R x channel discrete-time signals are arranged according to the number of fast-time dimension sampling points N, the number of slow-time dimension sampling points M, and the number of antenna channels L = T x ×R x , and the following signal form can be obtained:

[0083]

[0084] The discrete-time Fourier transform is performed on each dimension of the signal to obtain a range-Doppler-angle (RDA) three-dimensional tensor:

[0085] S(n,m,l) = DFT{x IF}

[0086] wherein, S(n,m,l) represents the RDA tensor, DFT represents the discrete-time Fourier transform, n represents the number of fast-time dimension DFT points, m represents the number of slow-time dimension DFT points, and l represents the number of antenna channel dimension DFT points.

[0087] S123, peak detection is performed on the RDA tensor S(n,m,l) to obtain the state information of the target:

[0088] [R,D,A] = CFAR{S(n,m,l)}

[0089] wherein, CFAR represents the constant false alarm rate detection method, R represents the distance of the target, D represents the Doppler velocity of the target, and A represents the angle of the target, including the pitch angle θ and the azimuth angle i.e.

[0090] After the obtained target state information is converted through coordinate conversion, the millimeter wave radar point cloud data is obtained:

[0091]

[0092] wherein, [X,Y,Z] is the three-dimensional coordinates of the target point, and V is the speed of the target point. Thus, the millimeter wave radar point cloud data [X,Y,Z,V] in each frame has been obtained, and the N f frame point cloud data of the millimeter wave radar is recorded as P = {P1, P2, …, P n , …, P Nf}.

[0093] S2, multi-target tracking is performed on the millimeter wave radar point cloud data to obtain multi-target tracking tracks and corresponding multi-target time series point cloud data, wherein a new track set and an old track set are constructed and updated during tracking. Specifically, the steps include:

[0094] S21, DBSCAN algorithm is used to cluster each frame of millimeter wave point cloud data, and the centroid of each clustered point cluster is condensed to obtain an observation value set.

[0095] Specifically, DBSCAN algorithm is used to cluster each frame of point cloud data in the X-Y plane, and the centroid of each clustered point cluster is condensed to obtain N f observation value set Z corresponding to the frame of point cloud data P = {Z1, Z2, …, Z n ,…,Z Nf}.

[0096] S22, based on the observation value set, the multi-target is tracked by combining the Kalman filter algorithm and the global nearest neighbor data association algorithm to obtain the multi-target tracking track.

[0097] Specifically, based on the observation value set, the multi-target tracking is performed on each data by combining the Kalman filter algorithm (KF) and the global nearest neighbor data association algorithm (GNN) to obtain the multi-target tracking track represents the mth track, X lm is the state information corresponding to the target in the l m frame, and the total number of frames tracked is L m . The tracking result is shown in the following table, Figure 3 Figure 3 is a multi-person tracking result example provided by the embodiment of the application based on the millimeter wave radar point cloud in the X-Y plane, Figure 3 In the figure, the tracks are interrupted due to occlusion and intersection between multiple people.

[0098] S23, the point cloud data corresponding to the position of each target in each frame is extracted from the multi-target tracking track to obtain the multi-target time series point cloud data corresponding to the multi-target tracking track.

[0099] Specifically, the embodiment aims to more effectively and accurately associate the interrupted tracks, and to perform the subsequent interrupted track association task, the point cloud data corresponding to the position of each target in each frame is extracted from the tracked track to obtain the point cloud data corresponding to the multi-target tracking track ​represents the time point cloud data corresponding to the mth track. Further, the method of extracting the point cloud data of the corresponding position of each target in each frame through the tracked track is: according to the target position information in the tracked track and the prior knowledge of the target, a range box is set in the scene-level point cloud data, and the data in the range box is recorded as the point cloud data corresponding to the multi-target tracking track.

[0100] wherein, in the tracking process, the multi-target tracking track is monitored, a new track set and an old track set are constructed according to the monitoring window length U, and the new track set and the old track set are updated. The new track set represents the track set after the interruption occurs, and the old track set represents the track set before the interruption occurs. The new track set update criteria include: deleting the associated track, deleting the terminated track, and adding all un-terminated tracks with a start time within the monitoring window. The update criteria of the old track set include: deleting the associated track, deleting the track falling from the monitoring window, and adding all tracks deleted from the new track set.

[0101] S3, when the updated new track set and the updated old track set are not empty, input the new track segment in the updated new track set and the old track segment in the updated old track set into the trained feature extraction network to obtain the new track segment feature and the old track segment feature, wherein the trained feature extraction network extracts the target subject motion feature corresponding to the point data in the new track segment and the old track segment and the target detail contour feature corresponding to the time point cloud data, and splices and fuses the target subject motion feature and the target detail contour feature. Specifically, it includes the following steps:

[0102] S31, obtaining the trained feature extraction network.

[0103] In this embodiment, the trained feature extraction network is obtained by training the feature extraction network using a pre-constructed training data set. The training method of the feature extraction network includes the following steps:

[0104] S311, constructing a training data set and performing standardization processing on the training data set to obtain a standardized training data set.

[0105] On the one hand, the related technologies based on interrupted track association are all based on simulation data for research, and there is a lack of measured data set to verify the method. On the other hand, the method of mining track segment features through deep neural network needs a large amount of training data, and the measured data needs to cover as many actual situations as possible. The above problems prompt this embodiment to construct as many multi-person tracking data sets as possible under various walking modes, so as to generate track interruption problems caused by various complex situations and verify the practicability of the method of this embodiment.

[0106] The embodiment is designed to design a multi-person target tracking algorithm based on interrupted track association. In order to ensure the universality of the designed algorithm (not affected by specific personnel), the personnel in the test and the training are ensured not to overlap when the training data and the test data are divided. The multi-person walking data obtained by combining the collected P n people The multi-person walking data obtained by combining the remaining P people are used as test data.

[0107] The construction method of the training data set and the test data set includes:

[0108] S3111, according to the total number of people in the multi-person scene, data of 2-person walking combination, 3-person walking combination and 4-person walking combination are collected, each combination includes close walking, crossing walking and free walking, and millimeter wave radar training echo data are obtained.

[0109] Specifically, the number of people walking in the multi-person scene includes: 2 people, 3 people, 4 people; the walking mode includes: close walking, crossing walking, free walking. The total number of people collected is P n people. The combination is collected in multiple 2-person, 3-person and 4-person combinations, and 15 groups (5 groups of close walking, 5 groups of crossing walking and 5 groups of free walking) of samples are collected in each combination.

[0110] S3112, the millimeter wave radar training echo data are preprocessed to obtain millimeter wave radar training point cloud data.

[0111] For specific preprocessing process, please refer to S121-S123 in step S1, which will not be repeated here.

[0112] S3113, the millimeter wave radar training point cloud data are subjected to multi-target tracking to obtain training multi-target tracking tracks and corresponding training multi-target time sequence point cloud data.

[0113] For specific process, please refer to steps S21-S23, which will not be repeated here.

[0114] S3114, according to the optical video collected synchronously with the millimeter wave radar training echo data, the training multi-target tracking tracks are judged. If any two tracks belong to different track segments of the same target, the label is set to 1, and if any two tracks do not belong to different track segments of the same target, the label is set to 0, and the label data are formed. Further, the training multi-target tracking tracks, the training multi-target time sequence point cloud data and the label data form a training data set.

[0115] S3115, the construction method of the test data includes: combining P people can obtain N testThe group contains multi-person walking data of 2, 3, 4 persons, in order to measure the performance of the technology provided in the embodiment, the tagging process is consistent with the step (1d) of constructing the test data set.

[0116] S3116, standardize the training data set and the test data set.

[0117] Specifically, when training the network, in order to avoid the influence of different distribution ranges of different dimensional data on network training, it is necessary to standardize each dimension of the data, and the 0-1 standardization method is used to standardize the data:

[0118]

[0119] Wherein, x represents a certain dimension in the data, x min represents the minimum value in the dimension data, x max represents the maximum value in the dimension data. After standardization, the same length window L is used to intercept all the data.

[0120] S312, input the training data set into the feature extraction network, and train the feature extraction network using a ternary loss function, to obtain the trained feature extraction network, wherein during network training, a ternary group is constructed according to the training data set to calculate the ternary loss function each time the network parameters are updated.

[0121] S3121, design a feature extraction network for interrupted track association tasks.

[0122] The feature extraction network designed by the application is based on the idea of track segment matching, uses the point track information in the track and the corresponding time sequence point cloud information to mine track segment features, and through the constraint of loss, makes the different track segment features of the same target continuously close in the feature space, and finally realizes the accurate association of interrupted tracks by measuring the Euclidean distance between the features.

[0123] Please refer to Figure 4 , Figure 4 The feature extraction network model for interrupted track association tasks provided by the embodiment of the application is shown in the schematic diagram. The feature extraction network model for interrupted track association tasks mainly includes five components: a network input module, a point track feature extraction module, a point cloud feature extraction module, and a feature fusion strategy module.

[0124] (1) The network input module determines the network input track length L according to the actual situation, if the track in the training data set is long, the long track is divided into multiple short tracks with a length of L. The network input is L frames of point track data and corresponding L frames of time sequence point cloud data.

[0125] (2) The point feature extraction module is connected with the network input module, and is used for extracting target subject motion features in the point data, and obtaining point features.

[0126] Specifically, the point data reflects target motion trends and other motion information. For different track segments of the same target, the motion information can be associated to a certain extent.

[0127] The point feature extraction module comprises a time sequence feature extraction module and a local and global feature extraction module connected in sequence, the time sequence feature extraction module comprises one or more of LSTM, RNN and Bi-LSTM, and the local and global feature extraction module comprises one or more of RestNet18 and a convolutional neural network.

[0128] Please refer to Figure 5 , Figure 5 The point feature extraction module provided by the embodiment of the application is shown in the schematic diagram. In order to fully mine the target motion information in the point data, the embodiment proposes a network structure of LSTM+ResNet18. The point feature extraction module firstly inputs the point data of L frames according to the time sequence relationship into a single-layer LSTM network with a length of L, and preliminarily extracts the time sequence features. The number of hidden layer units of the LSTM is also set to L, the hidden output of each LSTM time step unit is spliced into an LxL feature map according to the time sequence relationship, and then it is sent into the ResNet18. With the powerful feature extraction capability of the residual network, the local and global features of the input data can be learned, and finally more rich feature representations can be extracted. The output vector dimension of the ResNet18 is 1xM, and M is set to 256 in the application.

[0129] It should be noted that the LSTM of the point feature extraction module can be replaced by other recurrent neural networks such as RNN, Bi-LSTM or time sequence convolution network, and the RestNet18 can also be replaced by other more complex convolutional neural networks.

[0130] (3) The point cloud feature extraction module is connected with the network input module, and is used for extracting target detail contour features in the time sequence point cloud data, and obtaining point cloud features.

[0131] Specifically, the point cloud data reflects the shape, size and other detail information of the target, and the features that can be reflected are more effective for the association between different track segments of the same target.

[0132] Please refer to Figure 6 , Figure 6A schematic diagram of the point cloud feature extraction module provided by the embodiment of the present application. In order to extract the point cloud data features, the point cloud feature extraction module of the embodiment takes PointNet as the reference network, combines the time sequence feature extraction structure, extracts the shape and other detailed features of the target in the motion process, and includes a spatial feature extraction branch, a speed feature extraction branch, a fusion module, a cutting module, an aggregation module and a time sequence feature extraction module. The spatial feature extraction branch and the speed feature extraction branch are parallel and are both connected to the fusion module. The fusion module, the cutting module, the aggregation module and the time sequence feature extraction module are sequentially connected.

[0133] The input data of the point cloud feature extraction module is L frames of time sequence point cloud data. The size of each frame of time sequence point cloud data can be represented as NXS, N is the number of point clouds, and S = [X, Y, Z, V] is the feature of each point. For each frame of point cloud, it is divided into two branches of spatial feature extraction and speed feature extraction, and the spatial features S1 = [X, Y, Z] and the speed features S2 = [V] contained therein are extracted through a multi-layer perception (MLP) respectively; the fusion module splices the two features and then extracts the fused features through an MLP again; through the cutting module, the fused features of the N point cloud features are cut into four parts according to the Z value; in the aggregation module, the features of different cutting parts are aggregated through an average pooling layer and a maximum pooling layer respectively, corresponding to the feature representation of the different heights of the personnel target, and thus the feature output of each frame of point cloud is 1x4xW1, and the network parameters between different frames are shared; in the time sequence feature extraction module, the features of all frames are obtained in sequence through the feature extraction network according to the time sequence, and the features of all frames are Lx4xW1, and finally after the time sequence feature extraction module extracts the time sequence features between different frames, the output features are 4xW2, and the first splicing of the output features obtains the final point cloud features 1xM. In the present application, W1 = 64 and W2 = 64 are set.

[0134] It should be noted that the network structure of the point cloud feature extraction module can be replaced by other point cloud feature extraction networks such as PointNet++.

[0135] (4) The feature fusion strategy module is connected with the track feature extraction module and the point cloud feature extraction module, and is used to multiply the track features and the point cloud features by different weights and splice them together to obtain a feature vector.

[0136] Specifically, since the track data contains the motion features of the target subject, and the point cloud data reflects the shape, size and other detailed information of the target, the two have complementarity in feature representation, and therefore fusing the features of the two modules is beneficial to the interrupted track association task. The feature fusion strategy adopted by the embodiment is a weighted splicing strategy, which multiplies the track features and the point cloud features by different weights and splices them together, and finally obtains a 1x512 feature vector.

[0137] S3122, constructing triplets according to the training dataset to calculate a triplet loss function.

[0138] Specifically, the triplet loss constraint is used to pull the distance between different trajectory segment features of the same target closer in the feature space, and to push the distance between trajectory segment features of different targets farther. The calculation of the triplet loss requires three samples to form a triplet, and the three samples are a reference sample, a positive sample of the same class as the reference sample, and a negative sample of a different class from the reference sample. The purpose of optimizing the loss is to pull the distance between the reference sample and the positive sample closer, and to push the distance between the reference sample and the negative sample farther.

[0139] During network training, the specific process of calculating the triplet loss function when updating the network parameters each time is as follows:

[0140] (1) randomly select a track segment from the training dataset as a reference sample;

[0141] (2) according to the reference sample, randomly select a track segment with a label of 1 from the training dataset as a positive sample, and if there is no positive sample in the training dataset, then reselect the reference sample;

[0142] (3) according to the reference sample, randomly select a track segment with a label of 0 from the training dataset as a negative sample;

[0143] (4) calculate the triplet loss function using the reference sample, the positive sample, and the negative sample:

[0144] L = max(d(a, p) - d(a, n) + margin, 0)

[0145] where d represents the calculation of the Euclidean distance, a represents the reference sample, p represents the positive sample of the same class as the reference sample, n represents the negative sample of a different class from the reference sample, and margin represents a constant greater than 0.

[0146] Further, after obtaining the feature extraction network and the triplet loss function, the standardized training dataset is input into the feature extraction network, and the feature extraction network is trained using the triplet loss function to obtain the trained feature extraction network.

[0147] After obtaining the trained feature extraction network, the feature extraction network can be tested using the test dataset.

[0148] S32, when the updated new track set and the updated old track set are not empty, the new track segments in the updated new track set and the old track segments in the updated old track set and their corresponding time point cloud data are standardized, and then the standardized data is input into the trained feature extraction network. The trained feature extraction network extracts the target subject motion features corresponding to the point data in the new track segments and the old track segments and the target detail contour features corresponding to the time point cloud data, and splices and fuses the target subject motion features and the target detail contour features to obtain new track segment features and old track segment features.

[0149] S4, calculate the Euclidean distance between the new track segment features and the old track segment features in the feature space to obtain a new-old track set matching cost matrix.

[0150] Specifically, the new-old track set matching cost matrix is:

[0151]

[0152]

[0153] Wherein, d represents the calculation of Euclidean distance, x represents the xth track feature vector in the new track set, y represents the yth track feature vector in the old track set, and n represents the dimension of the feature vector.

[0154] S5, according to the new-old track set matching cost matrix, the new track segments and the old track segments are matched to obtain a track segment association result.

[0155] Specifically, according to the new-old track set matching cost matrix, the new track segments and the old track segments are matched by using the Hungarian matching algorithm to obtain a track segment association result.

[0156] In the multi-person tracking method of the embodiment, the new track set and the old track set are constructed and updated during multi-target tracking. When the updated new track set and the updated old track set are not empty, it represents that there is an interrupted track. At this time, the trained feature network is used for interrupted track association operation, so that the method can monitor the track state when the multi-person tracking track is frequently interrupted, thereby associating the interrupted track. Finally, the tracks belonging to the same target broken track all maintain consistent target ID, solving the tracking track interruption problem which is easily encountered in actual application and not considered in the prior art, greatly improving the continuity and robustness of multi-person target tracking, and being more suitable for actual millimeter wave radar multi-person tracking scenes such as safety protection, smart home and automatic driving, etc.

[0157] In the multi-person tracking method of the embodiment, the trained feature extraction network first extracts target subject motion features corresponding to the track data and target detail contour features corresponding to the time sequence point cloud data, and then splices and fuses the target subject motion features and the target detail contour features. On the basis of the motion features, the contour features and the like of the point cloud data are combined, so that the target track segment feature representation is more complete, and the accuracy of the interrupted track association in multi-person target tracking can be effectively improved.

[0158] The effect of the present application will be further described below in combination with experimental data:

[0159] 1. Experimental conditions and experimental content

[0160] The software platform used in this experiment is: Windows 10 operating system, Matlab R2021b, python 3.6, pytorch.

[0161] The hardware platform used in this experiment is: Dell T7910 workstation, CPU: Intel Core(TM) i7-4770, GPU: NVIDIA GTX 1080Ti, radar millimeter wave radar TI Awr2243.

[0162] The millimeter wave radar parameters used in this experiment are: carrier frequency 77GHz, frequency modulation bandwidth 1200MHz, frequency modulation time 12us, frequency modulation slope 100MHz / us, sampling frequency 12.5Mhz, linear frequency modulation signal repetition period 15.5us, idle time 3.5us, number of sampling points for each linear frequency modulation 128, and number of linear frequency modulation signals per frame 128.

[0163] The experimental data uses a measured millimeter wave radar multi-person tracking data set. A total of 32 subjects participated in the data collection, which were combined into multiple 2-person, 3-person, and 4-person combinations for data collection. The data set contains a total of 1320 groups of data, each group of data has a collection time of 8s, and the walking methods include close walking, crossing walking, and free walking. The training set contains 990 groups of data of 24 people, and contains a total of 5193 tracks for training. The test set contains 330 groups of data of 8 people, and contains a total of 1432 tracks for testing.

[0164] 2. Experimental results and analysis

[0165] The evaluation index of the test data set is: track association accuracy = number of correctly associated tracks / total number of tracks. The experimental results are shown in Table 1:

[0166] Table 1 Track association accuracy under different multi-person walking methods

[0167]

[0168] As can be seen in combination with Table 1, the method proposed in the embodiment can achieve an association accuracy of 80% on the test data, in which the accuracy of close walking is the lowest because close walking is prone to mutation of the motion direction before and after the track interruption, resulting in greater difficulty in association, and the accuracy of crossing walking is the highest because the track interruption before and after crossing occlusion causes the motion mode of the multi-person target to be simple, the association difficulty is small, and free walking contains most of the multi-person target walking modes, and the average accuracy is similar. The multi-person tracking method based on interrupted track association is first proposed in the embodiment, which expands the application of the millimeter wave radar in the actual multi-person scene and has high practical application value.

[0169] In summary, the interrupted track association method based on fusion of point cloud features and point track features is first proposed in the embodiment, which can extract the target subject motion features and target detail contour features at the same time and splice and fuse them through learning weights, and can effectively improve the interrupted track association accuracy.

[0170] The multi-person tracking method based on interrupted track association of the millimeter wave radar is first proposed in the embodiment for the multi-person track interruption problem, and the interrupted track association method designed in the application is combined to perform real-time interrupted track association through monitoring of the track state in the tracking process, so that the multi-person tracking of the millimeter wave radar is more robust.

[0171] The embodiment details the collection and construction process of the multi-person tracking data set of the millimeter wave radar, the collected actual measurement data as much as possible contains various situations of multi-person walking, verifies the universality of the proposed method, and the personnel of the training set and the test set are different, which verifies the generalization of the proposed method, so the constructed data set has important practical significance.

[0172] The above is a further detailed description of the application in combination with a specific preferred embodiment, and the specific implementation of the application cannot be limited to these descriptions. For ordinary skilled persons in the technical field to which the application belongs, some simple deductions or substitutions can be made without departing from the concept of the application, and all of them should be regarded as falling within the protection scope of the application.

Claims

1. A method for multi-person tracking based on interrupted track association using millimeter wave radar, characterized in that, The method comprises the steps of: acquiring millimeter wave radar echo data and preprocessing to obtain millimeter wave radar point cloud data; performing multi-target tracking on the millimeter wave radar point cloud data to obtain multi-target tracking tracks and corresponding multi-target time series point cloud data, wherein a new track set and an old track set are constructed and updated during tracking; when the updated new track set and the updated old track set are not empty, inputting new track segments in the updated new track set and old track segments in the updated old track set into a trained feature extraction network to obtain new track segment features and old track segment features, wherein the trained feature extraction network extracts target main body motion features and target detail contour features corresponding to point cloud data in the new track segments and the old track segments, and fuses the target main body motion features and the target detail contour features; calculating the Euclidean distance between the new track segment features and the old track segment features in a feature space to obtain a new-old track set matching cost matrix; matching the new track segments and the old track segments according to the new-old track set matching cost matrix to obtain track segment association results.

2. The method of claim 1, wherein, The method comprises the steps of: acquiring millimeter wave radar echo data and preprocessing to obtain millimeter wave radar point cloud data, comprising: acquiring millimeter wave radar echo data of a multi-person scene; Wherein, [X, Y, Z] is the three-dimensional coordinates of the target point, V is the speed of the target point, θ is the pitch angle of the target, is the azimuth angle of the target.

3. The method of claim 1, wherein, sequentially performing orthogonal mixer mixing, low-pass filtering, discrete time sampling, arranging according to fast time dimension sampling point number, slow time dimension sampling point number and antenna channel number, discrete time Fourier transform, peak detection and coordinate conversion on the millimeter wave radar echo data to obtain the millimeter wave radar point cloud data; performing multi-target tracking on the millimeter wave radar point cloud data to obtain multi-target tracking tracks and corresponding time series point cloud data, comprising: performing clustering on each frame of millimeter wave point cloud data using a DBSCAN algorithm, and performing centroid condensation on each clustered point cluster to obtain an observation value set; tracking multiple targets based on the observation value set, combining a Kalman filtering algorithm and a global nearest neighbor data association algorithm to obtain multi-target tracking tracks; 4. The method of claim 1, wherein, extracting point cloud data corresponding to the position of each target in each frame through the multi-target tracking tracks to obtain multi-target time series point cloud data corresponding to the multi-target tracking tracks. constructing a new track set and an old track set during tracking, and updating the new track set and the old track set, comprising: during tracking, monitoring the multi-target tracking tracks, constructing a new track set and an old track set according to the length of the monitoring window, and updating the new track set and the old track set; wherein, the new track set represents a track set after interruption, and the old track set represents a track set before interruption; the new track set update criteria include deleting associated tracks, deleting terminated tracks and adding all un-terminated tracks with starting times within the monitoring window; the old track update criteria include deleting associated tracks, deleting tracks falling from the monitoring window and adding all tracks deleted from the new track set.

5. The method of claim 1, wherein, The feature extraction network comprises a network input module, a track feature extraction module, a point cloud feature extraction module and a feature fusion strategy module, wherein The network input module is configured to input a plurality of frames of track data and a plurality of frames of time-series point cloud data corresponding to the track data; The track feature extraction module is connected to the network input module and configured to extract target subject motion features in the track data to obtain track features; The point cloud feature extraction module is connected to the network input module and configured to extract target detail contour features in the time-series point cloud data to obtain point cloud features; The feature fusion strategy module is connected to the track feature extraction module and the point cloud feature extraction module and configured to use a weighted splicing strategy to multiply the track features and the point cloud features by different weights and splice them together to obtain a feature vector.

6. The millimeter wave radar multi-person tracking method based on interrupted track association according to claim 5, wherein The track feature extraction module comprises a time-series feature extraction module and a local and global feature extraction module connected in sequence, the time-series feature extraction module comprises one or more of LSTM, RNN and Bi-LSTM, and the local and global feature extraction module comprises one or more of RestNet18 and a convolutional neural network; The point cloud feature extraction module is constructed based on PointNet or PointNet++ and comprises a spatial feature extraction branch, a velocity feature extraction branch, a fusion module, a cutting module, an aggregation module and a time-series feature extraction module, wherein the spatial feature extraction branch and the velocity feature extraction branch are parallel and both are connected to the fusion module, and the fusion module, the cutting module, the aggregation module and the time-series feature extraction module are connected in sequence.

7. The interrupt track association based millimeter wave radar multi-person tracking method according to claim 1, characterized in that, The training method of the feature extraction network comprises: Data is collected for 2-person walking combinations, 3-person walking combinations and 4-person walking combinations according to the total number of persons in a multi-person scenario, each combination comprising close walking, cross walking and free walking, to obtain millimeter wave radar training echo data; The millimeter wave radar training echo data is preprocessed to obtain millimeter wave radar training point cloud data; Multi-target tracking is performed on the millimeter wave radar training point cloud data to obtain training multi-target tracking tracks and corresponding training multi-target time-series point cloud data; The training multi-target tracking tracks are judged according to optical video collected synchronously with the millimeter wave radar training echo data, any two tracks are set as label 1 if they belong to different track segments of the same target, and any two tracks are set as label 0 if they do not belong to different track segments of the same target, to form label data; a training data set is formed from the training multi-target tracking tracks, the training multi-target time-series point cloud data and the label data; The training data set is standardized to obtain a standardized training data set; The standardized training data set is input into the feature extraction network, and the feature extraction network is trained using a triplet loss function, to obtain the trained feature extraction network, wherein during network training, a triplet is constructed according to the training data set to calculate the triplet loss function each time network parameters are updated.

8. The method of claim 7, wherein, According to the training data set, a triplet is constructed to calculate the triplet loss function, comprising: randomly selecting a track segment from the training data set as a reference sample; According to the reference sample, randomly selecting a track segment with a label of 1 from the training data set as a positive example sample, and if there is no positive example sample in the training data set, reselecting the reference sample; According to the reference sample, randomly selecting a track segment with a label of 0 from the training data set as a negative example sample; The triplet loss function is calculated using the reference sample, the positive example sample and the negative example sample: L = max (d (a, p) - d (a, n) + margin, 0) Wherein d represents calculating the Euclidean distance, a represents the reference sample, p represents the positive example sample of the same class as the reference sample, n represents the negative example sample of different classes from the reference sample, and margin represents a constant greater than 0.

9. The interrupt track association based millimeter wave radar multi-person tracking method according to claim 1, characterized in that, The new and old track set matching cost matrix is: Wherein d represents calculating the Euclidean distance, x represents the xth track feature vector in the new track set, y represents the yth track feature vector in the old track set, and n represents the dimension of the feature vector.

10. The method of claim 1, wherein, According to the new and old track set matching cost matrix, the new track segment and the old track segment are matched to obtain a track segment association result, comprising: According to the new and old track set matching cost matrix, the new track segment and the old track segment are matched using the Hungarian matching algorithm to obtain a track segment association result.

Images