Multi-target tracking method and system based on spatiotemporal trajectory association

Abstract

The present invention provides a multi-target tracking method and system based on spatiotemporal trajectory association, the method comprising: matching targets between adjacent frames in a multi-target tracking video, updating each target in each frame and the historical trajectory state corresponding to each target, and constructing a current frame candidate trajectory sequence and a historical candidate trajectory sequence according to the matching result; extracting features from the current frame candidate trajectory sequence to obtain the underlying features corresponding to each frame, and performing feature enhancement processing on the underlying features of the current frame according to the historical trajectory fragments and the underlying features of the historical frames to obtain trajectory features; based on visual similarity, spatial similarity and temporal similarity, according to the trajectory features, performing trajectory association matching on the candidate trajectories in the current frame candidate trajectory sequence and the candidate trajectories in the historical candidate trajectory sequence to obtain target tracking results in the multi-target tracking video. The present invention improves the accuracy of multi-target tracking and reduces mismatching and identity jumps.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a multi-target tracking method and system based on spatiotemporal trajectory association. Background Art

[0002] Multi-target tracking is one of the most attractive tasks in computer vision, which aims to automatically generate trajectories of moving targets. Existing multi-target tracking methods can be divided into two categories: tracking-by-detection (TBD) and detection-free tracking (DFT).

[0003] With the rapid development of multi-target detection, the TBD algorithm has become an effective multi-target tracking paradigm. The TBD algorithm divides multi-target tracking into four independent modules, including target detection, feature extraction, similarity calculation, and data association. Since the existing TBD algorithm only uses visual similarity to match different trajectories, it is easy to cause mismatches, and the targets in the scene are constantly changing, which can also cause identity jumps. For example, trajectories with similar visual features but distant in time and space are incorrectly associated, resulting in the accuracy of existing multi-target tracking needs to be further improved.

[0004] Therefore, there is an urgent need for a multi-target tracking method and system based on spatiotemporal trajectory association to solve the above problems. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention provides a multi-target tracking method and system based on spatiotemporal trajectory association.

[0006] The present invention provides a multi-target tracking method based on spatiotemporal trajectory association, comprising:

[0007] Matching targets between adjacent frames in a multi-target tracking video, updating each target in each frame of the multi-target tracking video and the historical trajectory state corresponding to each target, and constructing a current frame candidate trajectory sequence and a historical candidate trajectory sequence according to the matching results;

[0008] Extract features of the candidate trajectory sequence of the current frame to obtain underlying features corresponding to each frame of each trajectory segment, and enhance the underlying features of the current frame according to the underlying features of the historical trajectory segments and historical frames to obtain trajectory features corresponding to the candidate trajectory sequence of the current frame;

[0009] Based on visual similarity, spatial similarity and temporal similarity, and according to the trajectory features, candidate trajectories in the current frame candidate trajectory sequence are matched with candidate trajectories in the historical candidate trajectory sequence to obtain target tracking results in the multi-target tracking video.

[0010] According to a multi-target tracking method based on spatiotemporal trajectory association provided by the present invention, after performing trajectory association matching on the candidate trajectory in the current frame candidate trajectory sequence and the candidate trajectory in the historical candidate trajectory sequence based on the visual similarity, spatial similarity and temporal similarity according to the trajectory features and obtaining the target tracking result in the multi-target tracking video, the method further includes:

[0011] The target tracking results in the multi-target tracking video are subjected to trajectory post-processing by a bidirectional interpolation algorithm to obtain optimized target tracking results.

[0012] According to a multi-target tracking method based on spatiotemporal trajectory association provided by the present invention, before extracting features from the candidate trajectory sequence of the current frame to obtain underlying features corresponding to each frame of the image in each trajectory segment, and performing feature enhancement processing on the underlying features of the current frame according to the underlying features of the historical trajectory segments and the historical frames to obtain trajectory features corresponding to the candidate trajectory sequence of the current frame, the method further includes:

[0013] A spatiotemporal attention network is constructed through a ResNet-50 network, a global attention mechanism and a local attention mechanism, so as to extract features of the candidate trajectory sequence of the current frame through the ResNet-50 network of the spatiotemporal attention network to obtain underlying features, and the underlying features are enhanced based on the global attention mechanism and the local attention mechanism to obtain trajectory features.

[0014] According to a multi-target tracking method based on spatiotemporal trajectory association provided by the present invention, the underlying features of the current frame are enhanced according to the historical trajectory segments and the underlying features of the historical frames to obtain the trajectory features corresponding to the candidate trajectory sequence of the current frame, including:

[0015] Based on the historical candidate trajectory sequence corresponding to the current frame candidate trajectory sequence, obtaining the global attention feature of the historical trajectory segment;

[0016] The first underlying feature is subjected to feature enhancement processing through the second underlying feature to obtain a local attention feature; wherein the first underlying feature and the second underlying feature are respectively the underlying feature of the current frame and the underlying feature of the historical frame in the same trajectory segment of the candidate trajectory sequence of the current frame;

[0017] Fusing the global attention feature and the local attention feature to obtain a fused attention feature;

[0018] According to the fused attention feature, the first bottom-level feature is subjected to feature enhancement processing to obtain a third bottom-level feature;

[0019] Processing the third bottom-level features through a feature learner, a global average pooling layer, and a temporal average pooling layer to obtain trajectory segment features;

[0020] According to the trajectory segment features, trajectory features corresponding to the candidate trajectory sequence of the current frame are constructed.

[0021] According to a multi-target tracking method based on spatiotemporal trajectory association provided by the present invention, based on visual similarity, spatial similarity and temporal similarity, according to the trajectory features, candidate trajectories in the candidate trajectory sequence of the current frame are matched with candidate trajectories in the historical candidate trajectory sequence to obtain target tracking results in the multi-target tracking video, including:

[0022] Acquire a visual similarity between the current frame candidate trajectory sequence and the historical candidate trajectory sequence according to the trajectory features of the current frame candidate trajectory sequence and the trajectory features of the historical candidate trajectory sequence;

[0023] According to the trajectory start coordinates and trajectory end coordinates of the current frame candidate trajectory sequence and the historical candidate trajectory sequence, obtaining the spatial similarity between the current frame candidate trajectory sequence and the historical candidate trajectory sequence;

[0024] According to the start frame and the end frame of the current frame candidate trajectory sequence and the historical candidate trajectory sequence, obtaining the time similarity between the current frame candidate trajectory sequence and the historical candidate trajectory sequence;

[0025] According to the visual similarity, the spatial similarity and the temporal similarity, a loss matrix between the current frame candidate trajectory sequence and the historical candidate trajectory sequence is constructed, and the loss matrix is ​​solved by the Hungarian algorithm to perform trajectory association matching on the candidate trajectories in the current frame candidate trajectory sequence and the candidate trajectories in the historical candidate trajectory sequence to obtain the target tracking result in the multi-target tracking video.

[0026] According to a multi-target tracking method based on spatiotemporal trajectory association provided by the present invention, the formula of the visual similarity is:

[0027]

[0028] in, Represents historical candidate trajectories The trajectory characteristics of Represents the candidate trajectory of the current frame The trajectory characteristics, S v Indicates visual similarity;

[0029] The formula for the spatial similarity is:

[0030]

[0031] in, Represents the candidate trajectory of the current frame The starting coordinates of Represents the historical candidate trajectory predicted by the Kalman filter The ending coordinates, S d Indicates spatial distance;

[0032] Through maximum and minimum normalization, the spatial distance S d Transform and obtain the spatial similarity S s ;

[0033] The formula for the temporal similarity is:

[0034]

[0035] Among them, λ represents the smoothing factor, γ represents the shrinkage factor, and Respectively represent S s Candidate tracks The start and end frames of Represents historical candidate trajectories The end frame of , Δt represents the historical candidate trajectory and S s Candidate tracks The time interval between t Indicates temporal similarity;

[0036] The formula of the loss matrix is:

[0037]

[0038] The present invention also provides a multi-target tracking system based on time-space trajectory association, comprising:

[0039] A trajectory generation module is used to match targets between adjacent frames in a multi-target tracking video, update each target in each frame of the multi-target tracking video and the historical trajectory state corresponding to each target, and construct a current frame candidate trajectory sequence and a historical candidate trajectory sequence according to the matching results;

[0040] A trajectory feature generation module is used to extract features from the candidate trajectory sequence of the current frame, obtain the underlying features corresponding to each frame in each trajectory segment, and perform feature enhancement processing on the underlying features of the current frame according to the underlying features of the historical trajectory segments and the historical frames, to obtain the trajectory features corresponding to the candidate trajectory sequence of the current frame;

[0041] The spatiotemporal trajectory association module is used to perform trajectory association matching on the candidate trajectories in the current frame candidate trajectory sequence and the candidate trajectories in the historical candidate trajectory sequence based on visual similarity, spatial similarity and temporal similarity according to the trajectory features, so as to obtain the target tracking result in the multi-target tracking video.

[0042] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a multi-target tracking method based on spatiotemporal trajectory association as described above is implemented.

[0043] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the multi-target tracking method based on spatiotemporal trajectory association as described above is implemented.

[0044] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the multi-target tracking method based on spatiotemporal trajectory association as described above is implemented.

[0045] The multi-target tracking method and system based on spatiotemporal trajectory association provided by the present invention utilize the spatiotemporal relationship within and between trajectories to enhance the features of the trajectories, thereby combining the obtained discriminative trajectory features with the spatiotemporal relationship to associate the current trajectory with the historical trajectory, thereby improving the accuracy of multi-target tracking and reducing mismatching and identity jumps. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A schematic diagram of the process flow of the multi-target tracking method based on spatiotemporal trajectory association provided by the present invention;

[0048] Figure 2 A schematic diagram of target tracking post-processing based on a bidirectional interpolation algorithm provided by the present invention;

[0049] Figure 3 A schematic diagram of the structure of the spatiotemporal attention network provided by the present invention;

[0050] Figure 4 A schematic diagram of a spatiotemporal enhancement module provided by the present invention;

[0051] Figure 5 An overall schematic diagram of the multi-target tracking method based on spatiotemporal trajectory association provided by the present invention;

[0052] Figure 6 A schematic diagram of the structure of a multi-target tracking system based on spatiotemporal trajectory association provided by the present invention;

[0053] Figure 7 This is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] Existing multi-target tracking algorithms are mainly divided into frame matching-based frameworks and trajectory matching-based frameworks. Among them, the frame matching-based framework uses historical frame or historical trajectory features to match the current frame candidate frame, and learns discriminative features by combining various information, such as visual features, motion information and topological information, or focuses on generating trajectory features using trajectory confidence models. However, the frame matching-based framework is not reliable enough for the visual similarity of occluded targets, and it is easy to cause identity jumps.

[0056] Different from the frame matching based framework, the trajectory matching based framework considers the spatiotemporal information of each trajectory to handle long-term occlusion. Existing trajectory matching based frameworks use convolutional neural networks or long short-term memory networks to obtain robust trajectory descriptions.

[0057] Since existing multi-target tracking algorithms only use visual similarity to match different trajectories, there are the following problems: first, these algorithms treat each target within the trajectory independently, ignoring the spatiotemporal information, which can easily lead to mismatching; second, these algorithms ignore the spatiotemporal relationship between different trajectories, which can easily lead to identity jumps, for example, incorrectly associating trajectories that are distant in time and space but have similar visual features.

[0058] In order to solve the above problems, the present invention integrates spatiotemporal information into trajectory description and trajectory association. By introducing spatiotemporal attention and spatiotemporal relationship, discriminative trajectory features are extracted and different trajectories are matched, the reliability of trajectory association can be improved.

[0059] In the present invention, in order to extract discriminative trajectory features, spatiotemporal attention is used to focus on the most significant and complementary parts of each trajectory, thereby generating a robust visual feature description. After obtaining robust trajectory features using spatiotemporal information, visual similarity and spatiotemporal relationships are integrated into a unified metric function through trajectory association. However, constructing an effective metric function faces the following challenges: first, due to the uncertain motion trajectory and speed, the spatiotemporal relationship is extremely difficult to estimate, that is, a person may appear at any time and anywhere; second, due to the unreliability of the immeasurable spatiotemporal relationship, it is difficult to construct a reliable metric function. Therefore, the present invention also uses a logical model to convert the spatiotemporal relationship between trajectories into spatiotemporal confidence.

[0060] Figure 1 A schematic diagram of the process flow of the multi-target tracking method based on spatiotemporal trajectory association provided by the present invention is shown in FIG. Figure 1 As shown, the present invention provides a multi-target tracking method based on spatiotemporal trajectory association, comprising:

[0061] Step 101, matching targets between adjacent frames in a multi-target tracking video, updating each target in each frame of the multi-target tracking video and the historical trajectory state corresponding to each target, and constructing a current frame candidate trajectory sequence and a historical candidate trajectory sequence according to the matching results.

[0062] In the present invention, for multi-target tracking in a video stream, the trajectory generation of each target is intended to generate short trajectories between adjacent frames, which is a basic part of multi-target tracking. Existing multi-target tracking algorithms have achieved excellent results in short trajectory generation, for example, Tracktor, CenterTrack, Deepsort, JDE, and CTracker. Therefore, these existing algorithms can be regarded as methods for generating short-term trajectories. With the generation of short-term trajectories, the present invention merges short-term trajectories into long-term trajectories. Since trajectory matching is a complementary component of trajectory generation, a robust trajectory matching algorithm can be plug-and-play to existing algorithms, thereby further improving the performance of existing algorithms.

[0063] In the present invention, the existing algorithm Tracktor is used to generate short-term trajectories, that is, to obtain candidate trajectories corresponding to each target in each frame of the multi-target tracking video. Specifically, the present invention first uses the regression model of Faster R-CNN to generate candidate targets for each frame. Specifically, according to the target detection results of the current frame and the tracking results of the previous frame, these results are regressed to generate candidate targets for the current frame; then, the loss matrix between the candidate targets of the current frame and the tracking results of the previous frame is generated using the Intersection of Union (IoU) distance; finally, the Hungarian algorithm is used to optimize the loss matrix to generate the trajectory. It is worth noting that once a trajectory is not successfully matched to the corresponding target in the next frame, it is deemed that the trajectory is terminated.

[0064] Through the above method, the target detection in the video stream is completed. Further, it is necessary to associate the trajectories of the same target in each frame. The present invention decomposes the candidate trajectories in the entire video stream into video sequences of fixed length according to the order of video frames. That is, the candidate trajectory sequence, where is a trajectory segment, including N = 4 frames of pictures, n represents the total number of video sequences, for example, the candidate trajectory sequence contains 3 trajectory segments, each segment contains 4 frames of pictures, and each frame of pictures is arranged in the order of video frames. It should be noted that in the present invention, the candidate trajectory sequence of the current frame is composed of the historical trajectories that failed to match between two adjacent frames. When matching the targets between adjacent frames in the multi-target tracking video, the historical trajectories that failed to match construct the candidate trajectory sequence of the current frame, and the historical trajectories that successfully matched directly enter the adjacent frame target matching of the next frame. For each candidate trajectory of the current frame, all previous frame trajectories are used as its historical candidate trajectory sequence.

[0065] Step 102, feature extraction is performed on the candidate trajectory sequence of the current frame to obtain the underlying features corresponding to each frame of each trajectory segment, and feature enhancement processing is performed on the underlying features of the current frame according to the underlying features of the historical trajectory segments and the historical frames to obtain the trajectory features corresponding to the candidate trajectory sequence of the current frame.

[0066] In existing multi-target tracking algorithms, each target in the sequence is treated independently, while the spatiotemporal information within the trajectory is ignored. Different from global average pooling, local attention is used to describe trajectory features in re-identification tasks. However, local attention only considers the current video sequence and ignores the features of historical video sequences. In order to make full use of the spatiotemporal clues of historical trajectory segments, this paper proposes an improved spatial-temporal attention neural network (STANet) that uses global-local attention to generate robust trajectory segment features. STANet classifies trajectory segments into As input, output fragment features

[0067]

[0068] Furthermore, the sequence-level features are fused to generate trajectory features

[0069]

[0070] Among them, TAP represents the temporal average pooling layer, Represents the trajectory features corresponding to the i-th trajectory segment sequence.

[0071] Step 103 , based on visual similarity, spatial similarity and temporal similarity and according to the trajectory features, the candidate trajectories in the current frame candidate trajectory sequence are matched with the candidate trajectories in the historical candidate trajectory sequence to obtain the target tracking result in the multi-target tracking video.

[0072] In the present invention, a loss matrix is ​​constructed by visual similarity, spatial similarity and temporal similarity, and then trajectory features are obtained according to the above embodiment. The candidate trajectories in the candidate trajectory sequence of the current frame are matched with the candidate trajectories in the historical candidate trajectory sequence to obtain the optimal matching result, and the long-term trajectory of each target in the multi-target tracking video is obtained, that is, the target tracking result is obtained.

[0073] The multi-target tracking method based on spatiotemporal trajectory association provided by the present invention utilizes the spatiotemporal relationship within and between trajectories to enhance the features of the trajectories, thereby combining the obtained discriminative trajectory features with the spatiotemporal relationship to associate the current trajectory with the historical trajectory, thereby improving the accuracy of multi-target tracking and reducing mismatching and identity jumps.

[0074] On the basis of the above embodiment, after performing trajectory association matching on the candidate trajectories in the candidate trajectory sequence of the current frame and the candidate trajectories in the historical candidate trajectory sequence based on the visual similarity, spatial similarity and temporal similarity and according to the trajectory features, and obtaining the target tracking result in the multi-target tracking video, the method further includes:

[0075] The target tracking results in the multi-target tracking video are subjected to trajectory post-processing by a bidirectional interpolation algorithm to obtain optimized target tracking results.

[0076] In the present invention, by associating the current frame candidate trajectory with the historical candidate trajectory in time and space, different trajectories can be correctly matched, but the objects blocked between the trajectories may still be undetected. To solve this problem, the existing algorithm uses linear interpolation to restore the historical candidate trajectory. Candidate trajectory with current frame However, the linear interpolation algorithm ignores the motion information inside the trajectory, resulting in many misdetected targets. Therefore, the present invention provides a method for recovering lost targets using a bidirectional interpolation algorithm.

[0077] Figure 2 The target tracking post-processing diagram based on the bidirectional interpolation algorithm provided by the present invention can be referred to Figure 2 As shown in FIG. 1 , the bidirectional interpolation algorithm aims to generate a smooth trajectory. The algorithm uses a Kalman filter to consider the motion information of the trajectory. Specifically, the present invention divides the lost trajectory segment into an upper half and a lower half. In order to recover the lost target in the upper half, first, a historical candidate trajectory is generated. Select track with current frame Then, the linear interpolation coordinates and the virtual coordinates are calculated (the historical candidate trajectory is predicted by the Kalman filter). The intermediate position between the virtual coordinates of the trajectory start frame) is used to update the Kalman filter; then, the lost target in the upper half is restored by iteratively executing the above process. In addition, the lost target operation in the lower half is the reverse operation of the upper half, except that the Kalman filter is constructed from the future frame to the past frame. Finally, the bidirectional interpolation algorithm can be used to recover the lost target based on the historical candidate trajectory. Candidate trajectory with current frame The matching results in a smooth trajectory.

[0078] On the basis of the above embodiment, before extracting features from the candidate trajectory sequence of the current frame to obtain underlying features corresponding to each frame of the image in each trajectory segment, and performing feature enhancement processing on the underlying features of the current frame according to the underlying features of the historical trajectory segments and the historical frames to obtain trajectory features corresponding to the candidate trajectory sequence of the current frame, the method further includes:

[0079] A spatiotemporal attention network is constructed through a ResNet-50 network, a global attention mechanism and a local attention mechanism, so as to extract features of the candidate trajectory sequence of the current frame through the ResNet-50 network of the spatiotemporal attention network to obtain underlying features, and the underlying features are enhanced based on the global attention mechanism and the local attention mechanism to obtain trajectory features.

[0080] On the basis of the above embodiment, the feature enhancement processing is performed on the underlying features of the current frame according to the historical trajectory fragments and the underlying features of the historical frames to obtain the trajectory features corresponding to the candidate trajectory sequence of the current frame, including:

[0081] Based on the historical candidate trajectory sequence corresponding to the current frame candidate trajectory sequence, obtaining the global attention feature of the historical trajectory segment;

[0082] The first underlying feature is subjected to feature enhancement processing through the second underlying feature to obtain a local attention feature; wherein the first underlying feature and the second underlying feature are respectively the underlying feature of the current frame and the underlying feature of the historical frame in the same trajectory segment of the candidate trajectory sequence of the current frame;

[0083] Fusing the global attention feature and the local attention feature to obtain a fused attention feature;

[0084] According to the fused attention feature, the first bottom-level feature is subjected to feature enhancement processing to obtain a third bottom-level feature;

[0085] Processing the third bottom-level features through a feature learner, a global average pooling layer, and a temporal average pooling layer to obtain trajectory segment features;

[0086] According to the trajectory segment features, trajectory features corresponding to the candidate trajectory sequence of the current frame are constructed.

[0087] In the present invention, the key problem of generating robust trajectory features is how to generate video sequences Corresponding trajectory segment features However, existing algorithms always treat each target in the video sequence independently and ignore the spatiotemporal information within the trajectory. Different from global average pooling, local attention is used to describe trajectory features in the re-identification task. However, local attention only considers the current video sequence and ignores the features of historical video sequences. In order to make full use of the spatiotemporal clues of historical trajectory segments, this paper proposes an improved spatiotemporal attention network (STANet) that uses global-local attention to generate robust trajectory segment features. In this paper, STANet converts trajectory segments into As input, output trajectory segment features

[0088]

[0089] Among them, G i represents the global attention feature generated by the historical information of the i-th trajectory segment, It stands for STANet. It can be understood that for the video stream collected by the same acquisition module (for example, a camera), when performing multi-target tracking, the i-th track segment can be used as the current frame track segment, the previous i-1 track segments can be used as the historical track segments of the current frame track segment, and the i-th track segment is subjected to feature enhancement processing.

[0090] For multi-target tracking tasks, motion blur and occlusion can cause the current candidate segment to contain insufficient information and lead to identity jumps, while historical trajectory segments can provide some discriminative clues to enhance the description of the current segment. Figure 3 This is a schematic diagram of the structure of the spatiotemporal attention network provided by the present invention, in which the spatiotemporal boosting module (Spatial-TemporalBoosting, referred to as STB) is constructed based on the global attention mechanism and the local attention mechanism, which can be referred to Figure 3 As shown, in the present invention, the global clues of the historical fragments are used to obtain the global attention feature G i :

[0091]

[0092] in, F i-1,t (t∈[1,N]) represents the underlying features of the t-th frame of the i-1-th segment, and * represents the dot product.

[0093] Furthermore, given a trajectory segment First, the backbone network ResNet-50 is used to extract the corresponding underlying features F i,t , where I i,t With F i,tRespectively represent the t-th frame image and the underlying features in the ith trajectory segment. For the convenience of description, in the present invention, I i,t With F i,t Represented as I t With F t .

[0094] Furthermore, in the present invention, a spatiotemporal enhancement module (STB) is constructed based on the global attention mechanism and the local attention mechanism. Figure 4 For a schematic diagram of the spatiotemporal enhancement module provided by the present invention, please refer to Figure 4 As shown, the spatiotemporal enhancement module uses spatiotemporal clues to enhance the current frame feature F i , the enhanced feature is defined as F i The spatiotemporal enhancement module is a global-local attention mechanism that includes global attention and local attention. (H = 256, W = 128, C1 = 2048) describes the attention of the historical track segment; different from the global attention, the local attention focuses on the video frame of the same track segment. The feature F of the previous frame of the current frame track segment p (p∈[1,t-1]), the second bottom-level feature, is used to enhance the current frame image feature F t , which is the first bottom-level feature. Therefore, the local attention feature L t It can be expressed as:

[0095] L t =F1*…F p …*F t-1 ;

[0096] Then, the global attention feature G is fused i With the local attention feature L t , we get the global-local attention feature, that is, the fused attention feature:

[0097] Λ i,t =G i *L t ;

[0098] Among them, Λ i,t represents the global-local attention feature of the tth frame in the i-th trajectory segment. For the convenience of description, the present invention will i,t Simplified to Λ t . Used to enhance the current frame features Therefore, the present invention utilizes the global-local attention feature Λ t With the current frame feature F t Inferring the attention map a t :

[0099] a t =Λ t *F t ;

[0100] Furthermore, using the attention map a t and the underlying feature F of the current frame t , generate enhanced features F t ′=a t +F t That is, the third bottom-level feature is obtained.

[0101] Next, the enhanced feature F is transformed into t ′ is mapped to f t ∈R 1×C (C=1024):

[0102] f t =GAP(φ(F t ′));

[0103] The feature learner φ is the fourth layer of the ResNet-50 network. Finally, the temporal average pooling layer (TAP) uses f i,t (t∈[1,N]) Generate trajectory segment features Then generate trajectory features

[0104] On the basis of the above embodiment, based on visual similarity, spatial similarity and temporal similarity, according to the trajectory features, the candidate trajectories in the candidate trajectory sequence of the current frame are matched with the candidate trajectories in the historical candidate trajectory sequence to obtain the target tracking result in the multi-target tracking video, including:

[0105] Acquire a visual similarity between the current frame candidate trajectory sequence and the historical candidate trajectory sequence according to the trajectory features of the current frame candidate trajectory sequence and the trajectory features of the historical candidate trajectory sequence;

[0106] According to the trajectory start coordinates and trajectory end coordinates of the current frame candidate trajectory sequence and the historical candidate trajectory sequence, obtaining the spatial similarity between the current frame candidate trajectory sequence and the historical candidate trajectory sequence;

[0107] According to the start frame and the end frame of the current frame candidate trajectory sequence and the historical candidate trajectory sequence, obtaining the time similarity between the current frame candidate trajectory sequence and the historical candidate trajectory sequence;

[0108] According to the visual similarity, the spatial similarity and the temporal similarity, a loss matrix between the current frame candidate trajectory sequence and the historical candidate trajectory sequence is constructed, and the loss matrix is ​​solved by the Hungarian algorithm to perform trajectory association matching on the candidate trajectories in the current frame candidate trajectory sequence and the candidate trajectories in the historical candidate trajectory sequence to obtain the target tracking result in the multi-target tracking video.

[0109] In the present invention, the temporal and spatial relationships are used to correctly match the trajectories between different current time periods and historical time periods. Candidate trajectory with current frame j The data association between them is explained, where T is the historical trajectory set.

[0110] Specifically, given a historical candidate trajectory Candidate trajectory with current frame Historical candidate trajectory Explain, among which, represents the trajectory characteristics, and Respectively represent the start frame and end frame of the trajectory, and Represents the starting and ending coordinates of the trajectory.

[0111] Furthermore, a loss matrix between the current frame candidate trajectory sequence and the historical candidate trajectory sequence is constructed through visual similarity, spatial similarity and temporal similarity. The formula of the loss matrix is:

[0112]

[0113] in, Used to measure historical candidate trajectories Select track with current frame The visual similarity between Used to express the spatial similarity between different trajectories. Used to express the temporal similarity between different trajectories.

[0114] In the present invention, the historical trajectory set T and the current frame candidate trajectory The loss matrix between is defined as And through the Hungarian algorithm, the optimal matching result is obtained according to the loss matrix, so as to obtain the long-term trajectory.

[0115] In this invention, visual similarity is used to measure the historical candidate trajectories. Candidate trajectory with current frame The present invention uses the cosine similarity of the trajectory features generated by STANet to represent the visual similarity between trajectories. The formula for the visual similarity is:

[0116]

[0117] In the present invention, wherein, Represents historical candidate trajectories The trajectory characteristics of Represents the candidate trajectory of the current frame Trajectory characteristics of S v represents the visual similarity, The higher the value, the more likely it is that a candidate historical trajectory Candidate trajectory with current frame The higher the similarity between them, the better. It should be noted that existing algorithms mostly use visual similarity to calculate the data association between trajectories. Therefore, visual similarity can be used as a benchmark for trajectory matching.

[0118] Furthermore, in addition to visual similarity, the continuity of target motion between adjacent trajectories is another valuable prior knowledge for multi-target tracking tasks. That is, the spatial displacement between adjacent trajectories of the same target is relatively small. Since there is a time domain interval between adjacent trajectories, the key to calculating the spatial distance between trajectories is to calculate the spatial displacement in the same time frame. In order to reduce the spatial domain deviation caused by the time domain interval, the present invention constructs a Kalman filter for each trajectory to predict its position in the entire video sequence.

[0119] Specifically, the historical candidate trajectory Candidate trajectory with current frame Represents the spatial similarity from the historical trajectory to the candidate trajectory. Candidate trajectory with current frame There is a time interval between them, and the spatial similarity between the two trajectories cannot be directly calculated. To solve this problem, the present invention first uses a Kalman filter to predict the historical candidate trajectories. exist Virtual coordinates of time That is, the historical candidate trajectory is obtained by prediction The ending coordinates of Represents the candidate trajectory of the current frame The starting frame; then, using the virtual coordinates To calculate the spatial distance between trajectories Normalize the spatial distance by maximum and minimum normalization Then we get the spatial similarity The higher the historical candidate trajectory Candidate trajectory with current frame The greater the probability of belonging to the same target, the greater the probability of belonging to the same target. Therefore, the present invention uses the formula of spatial distance as the formula of spatial similarity:

[0120]

[0121] in, Represents the candidate trajectory of the current frame The starting coordinates of Represents the historical candidate trajectory predicted by the Kalman filter The ending coordinates, S d Indicates spatial distance;

[0122] Through maximum and minimum normalization, the spatial distance S d Transform to get the spatial similarity.

[0123] Furthermore, when the target is blocked, it will still reappear in the continuous time domain. In other words, the historical candidate trajectory Candidate trajectory with current frame The smaller the time interval between them, the greater the probability that the two trajectories belong to the same target. Since the target’s trajectory and speed are uncertain, the historical candidate trajectories Candidate trajectory with current frame The time domain interval Δt between the two cannot be directly used to calculate the time similarity. Therefore, the present invention proposes to use a logical model to convert the time domain interval into time similarity, and the formula of the time similarity is:

[0124]

[0125]

[0126] Wherein, λ represents a smoothing factor, γ represents a shrinkage factor, and in the present invention, γ=1 / L is set, and L represents the length of the video sequence; and Represent the candidate trajectories of the current frame The start and end frames of Represents historical candidate trajectories The end frame of , Δt represents the historical candidate trajectory and the candidate trajectory of the current frame The time interval between t Indicates the time similarity, The higher the value, the more likely it is that a candidate historical trajectory Candidate trajectory with current frame The greater the probability of belonging to the same target.

[0127] Figure 5 The overall schematic diagram of the multi-target tracking method based on spatiotemporal trajectory association provided by the present invention can be referred to Figure 5 As shown, the present invention first generates continuous frame trajectories using spatial similarity; then, through the spatiotemporal attention network, the spatiotemporal attention of the entire trajectory is captured to generate discriminative trajectory features; then, through spatiotemporal trajectory matching, the spatiotemporal relationship between trajectories is considered to generate high-quality complete trajectories with less identity jumps; finally, based on the bidirectional interpolation algorithm, the occluded targets between the matched trajectories are restored to generate a smooth trajectory.

[0128] In one embodiment, based on three public data sets, MOT15, MOT16 and MOT17, the method provided by the present invention is compared with the existing method for research experiments. From the experimental results, it can be seen that the method of the present invention has a significant effect on the accuracy of multi-target tracking, and improves the ability of the multi-target tracking algorithm to solve occlusion and false detection.

[0129] The multi-target tracking system based on spatiotemporal trajectory association provided by the present invention is described below. The multi-target tracking system based on spatiotemporal trajectory association described below and the multi-target tracking method based on spatiotemporal trajectory association described above can correspond to each other.

[0130] Figure 6 The schematic diagram of the structure of the multi-target tracking system based on spatiotemporal trajectory association provided by the present invention is as follows: Figure 6 As shown, the present invention provides a multi-target tracking system based on spatiotemporal trajectory association, including a trajectory generation module 601, a trajectory feature generation module 602 and a spatiotemporal trajectory association module 603, wherein the trajectory generation module 601 is used to match targets between adjacent frames in a multi-target tracking video, update each target in each frame of the multi-target tracking video and the historical trajectory state corresponding to each target, and construct a current frame candidate trajectory sequence and a historical candidate trajectory sequence according to the matching result; the trajectory feature generation module 602 is used to extract features from the current frame candidate trajectory sequence, obtain the underlying features corresponding to each frame in each trajectory segment, and perform feature enhancement processing on the underlying features of the current frame according to the historical trajectory segments and the underlying features of the historical frames, to obtain the trajectory features corresponding to the current frame candidate trajectory sequence; the spatiotemporal trajectory association module 603 is used to perform trajectory association matching on the candidate trajectories in the current frame candidate trajectory sequence and the candidate trajectories in the historical candidate trajectory sequence based on visual similarity, spatial similarity and temporal similarity according to the trajectory features, to obtain the target tracking result in the multi-target tracking video.

[0131] The multi-target tracking system based on spatiotemporal trajectory association provided by the present invention utilizes the spatiotemporal relationship within and between trajectories to enhance the features of the trajectories, thereby combining the obtained discriminative trajectory features with the spatiotemporal relationship to associate the current trajectory with the historical trajectory, thereby improving the accuracy of multi-target tracking and reducing mismatching and identity jumps.

[0132] The system provided by the present invention is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for the specific processes and detailed contents, which will not be repeated here.

[0133] Figure 7 A schematic diagram of the structure of an electronic device provided by the present invention, such as Figure 7 As shown, the electronic device may include: a processor (Processor) 701, a communication interface (Communications Interface) 702, a memory (Memory) 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704. The processor 701 can call the logic instructions in the memory 703 to execute a multi-target tracking method based on spatiotemporal trajectory association, the method comprising: matching targets between adjacent frames in a multi-target tracking video, updating each target in each frame of the multi-target tracking video and the historical trajectory state corresponding to each target, and constructing a current frame candidate trajectory sequence and a historical candidate trajectory sequence according to the matching result; performing feature extraction on the current frame candidate trajectory sequence to obtain the underlying features corresponding to each frame of each trajectory segment, and performing feature enhancement processing on the underlying features of the current frame according to the historical trajectory segments and the underlying features of the historical frames to obtain the trajectory features corresponding to the current frame candidate trajectory sequence; based on visual similarity, spatial similarity and temporal similarity, according to the trajectory features, performing trajectory association matching on the candidate trajectories in the current frame candidate trajectory sequence and the candidate trajectories in the historical candidate trajectory sequence to obtain the target tracking result in the multi-target tracking video.

[0134] In addition, the logic instructions in the above-mentioned memory 703 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0135] On the other hand, the present invention also provides a computer program product, the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program includes program instructions, when the program instructions are executed by the computer, the computer can execute the multi-target tracking method based on spatiotemporal trajectory association provided by the above methods, the method includes: matching targets between adjacent frames in a multi-target tracking video, updating each target in each frame of the multi-target tracking video and the historical trajectory state corresponding to each target, and constructing a current frame candidate trajectory sequence and a historical candidate trajectory sequence according to the matching result; extracting features from the current frame candidate trajectory sequence to obtain the underlying features corresponding to each frame of each trajectory segment, and performing feature enhancement processing on the underlying features of the current frame according to the historical trajectory segments and the underlying features of the historical frames to obtain the trajectory features corresponding to the current frame candidate trajectory sequence; based on visual similarity, spatial similarity and temporal similarity, according to the trajectory features, performing trajectory association matching between the candidate trajectories in the current frame candidate trajectory sequence and the candidate trajectories in the historical candidate trajectory sequence, and obtaining the target tracking result in the multi-target tracking video. .

[0136] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the processor executes the multi-target tracking method based on spatiotemporal trajectory association provided by the above-mentioned embodiments, the method comprising: matching targets between adjacent frames in a multi-target tracking video, updating each target in each frame of the multi-target tracking video and the historical trajectory state corresponding to each target, and constructing a current frame candidate trajectory sequence and a historical candidate trajectory sequence according to the matching results; performing feature extraction on the current frame candidate trajectory sequence to obtain the underlying features corresponding to each frame in each trajectory segment, and performing feature enhancement processing on the underlying features of the current frame according to the historical trajectory segments and the underlying features of the historical frames to obtain the trajectory features corresponding to the current frame candidate trajectory sequence; based on visual similarity, spatial similarity and temporal similarity, according to the trajectory features, performing trajectory association matching on the candidate trajectories in the current frame candidate trajectory sequence and the candidate trajectories in the historical candidate trajectory sequence to obtain the target tracking result in the multi-target tracking video.

[0137] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0138] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-target tracking method based on spatiotemporal trajectory association, characterized in that: include: Matching targets between adjacent frames in a multi-target tracking video, updating each target in each frame of the multi-target tracking video and the historical trajectory state corresponding to each target, and constructing a current frame candidate trajectory sequence and a historical candidate trajectory sequence according to the matching results; Extract features of the candidate trajectory sequence of the current frame to obtain underlying features corresponding to each frame of each trajectory segment, and enhance the underlying features of the current frame according to the underlying features of the historical trajectory segments and historical frames to obtain trajectory features corresponding to the candidate trajectory sequence of the current frame; Based on visual similarity, spatial similarity and temporal similarity, according to the trajectory features, the candidate trajectories in the current frame candidate trajectory sequence are matched with the candidate trajectories in the historical candidate trajectory sequence to obtain the target tracking result in the multi-target tracking video; The step of performing feature enhancement processing on the underlying features of the current frame according to the historical trajectory segments and the underlying features of the historical frames to obtain trajectory features corresponding to the candidate trajectory sequence of the current frame includes: Based on the historical candidate trajectory sequence corresponding to the current frame candidate trajectory sequence, obtaining the global attention feature of the historical trajectory segment; The first underlying feature is subjected to feature enhancement processing through the second underlying feature to obtain a local attention feature; wherein the first underlying feature and the second underlying feature are respectively the underlying feature of the current frame and the underlying feature of the historical frame in the same trajectory segment of the candidate trajectory sequence of the current frame; Fusing the global attention feature and the local attention feature to obtain a fused attention feature; According to the fused attention feature, the first bottom-level feature is subjected to feature enhancement processing to obtain a third bottom-level feature; Processing the third bottom-level features through a feature learner, a global average pooling layer, and a temporal average pooling layer to obtain trajectory segment features; Constructing trajectory features corresponding to the candidate trajectory sequence of the current frame according to the trajectory segment features; The method of performing trajectory association matching on the candidate trajectory in the current frame candidate trajectory sequence and the candidate trajectory in the historical candidate trajectory sequence based on the visual similarity, spatial similarity and temporal similarity according to the trajectory features to obtain the target tracking result in the multi-target tracking video includes: Acquire a visual similarity between the current frame candidate trajectory sequence and the historical candidate trajectory sequence according to the trajectory features of the current frame candidate trajectory sequence and the trajectory features of the historical candidate trajectory sequence; According to the trajectory start coordinates and trajectory end coordinates of the current frame candidate trajectory sequence and the historical candidate trajectory sequence, obtaining the spatial similarity between the current frame candidate trajectory sequence and the historical candidate trajectory sequence; According to the start frame and the end frame of the current frame candidate trajectory sequence and the historical candidate trajectory sequence, obtaining the time similarity between the current frame candidate trajectory sequence and the historical candidate trajectory sequence; According to the visual similarity, the spatial similarity and the temporal similarity, a loss matrix between the current frame candidate trajectory sequence and the historical candidate trajectory sequence is constructed, and the loss matrix is ​​solved by the Hungarian algorithm to perform trajectory association matching on the candidate trajectories in the current frame candidate trajectory sequence and the candidate trajectories in the historical candidate trajectory sequence to obtain the target tracking result in the multi-target tracking video.

2. The multi-target tracking method based on spatiotemporal trajectory association according to claim 1 is characterized in that: After performing trajectory association matching on the candidate trajectories in the candidate trajectory sequence of the current frame and the candidate trajectories in the historical candidate trajectory sequence based on the visual similarity, spatial similarity and temporal similarity and according to the trajectory features to obtain the target tracking result in the multi-target tracking video, the method further includes: The target tracking results in the multi-target tracking video are subjected to trajectory post-processing by a bidirectional interpolation algorithm to obtain optimized target tracking results.

3. The multi-target tracking method based on spatiotemporal trajectory association according to claim 1, characterized in that: Before extracting features from the candidate trajectory sequence of the current frame to obtain underlying features corresponding to each frame of the image in each trajectory segment, and performing feature enhancement processing on the underlying features of the current frame according to the underlying features of the historical trajectory segments and the historical frames to obtain trajectory features corresponding to the candidate trajectory sequence of the current frame, the method further includes: A spatiotemporal attention network is constructed through a ResNet-50 network, a global attention mechanism and a local attention mechanism, so as to extract features of the candidate trajectory sequence of the current frame through the ResNet-50 network of the spatiotemporal attention network to obtain underlying features, and the underlying features are enhanced based on the global attention mechanism and the local attention mechanism to obtain trajectory features.

4. The multi-target tracking method based on spatiotemporal trajectory association according to claim 1, characterized in that: The formula for the visual similarity is: in, Represents historical candidate trajectories The trajectory characteristics of Represents the candidate trajectory of the current frame The trajectory characteristics, S v Indicates visual similarity; The formula for the spatial similarity is: in, Represents the candidate trajectory of the current frame The starting coordinates of Represents the historical candidate trajectory predicted by the Kalman filter The ending coordinates, S d Indicates spatial distance; Through maximum and minimum normalization, the spatial distance S d Transform and obtain the spatial similarity S s ; The formula for the temporal similarity is: Among them, λ represents the smoothing factor, γ represents the shrinkage factor, and Represent the candidate trajectories of the current frame The start and end frames of Represents historical candidate trajectories The end frame of , Δt represents the historical candidate trajectory and the candidate trajectory of the current frame The time interval between t Indicates temporal similarity; The formula of the loss matrix is:

5. A multi-target tracking system based on spatiotemporal trajectory association, characterized in that: include: A trajectory generation module is used to match targets between adjacent frames in a multi-target tracking video, update each target in each frame of the multi-target tracking video and the historical trajectory state corresponding to each target, and construct a current frame candidate trajectory sequence and a historical candidate trajectory sequence according to the matching results; A trajectory feature generation module is used to extract features from the candidate trajectory sequence of the current frame, obtain the underlying features corresponding to each frame in each trajectory segment, and perform feature enhancement processing on the underlying features of the current frame according to the underlying features of the historical trajectory segments and historical frames, to obtain the trajectory features corresponding to the selected trajectory sequence of the current frame; A spatiotemporal trajectory association module, configured to perform trajectory association matching on the candidate trajectories in the candidate trajectory sequence of the current frame and the candidate trajectories in the historical candidate trajectory sequence based on visual similarity, spatial similarity and temporal similarity according to the trajectory features, so as to obtain target tracking results in the multi-target tracking video; The trajectory feature generation module is specifically used for: Based on the historical candidate trajectory sequence corresponding to the current frame candidate trajectory sequence, obtaining the global attention feature of the historical trajectory segment; The first underlying feature is subjected to feature enhancement processing through the second underlying feature to obtain a local attention feature; wherein the first underlying feature and the second underlying feature are respectively the underlying feature of the current frame and the underlying feature of the historical frame in the same trajectory segment of the candidate trajectory sequence of the current frame; Fusing the global attention feature and the local attention feature to obtain a fused attention feature; According to the fused attention feature, the first bottom-level feature is subjected to feature enhancement processing to obtain a third bottom-level feature; Processing the third bottom-level features through a feature learner, a global average pooling layer, and a temporal average pooling layer to obtain trajectory segment features; Constructing trajectory features corresponding to the candidate trajectory sequence of the current frame according to the trajectory segment features; The spatiotemporal trajectory association module is specifically used for: Acquire a visual similarity between the current frame candidate trajectory sequence and the historical candidate trajectory sequence according to the trajectory features of the current frame candidate trajectory sequence and the trajectory features of the historical candidate trajectory sequence; According to the trajectory start coordinates and trajectory end coordinates of the current frame candidate trajectory sequence and the historical candidate trajectory sequence, obtaining the spatial similarity between the current frame candidate trajectory sequence and the historical candidate trajectory sequence; According to the start frame and the end frame of the current frame candidate trajectory sequence and the historical candidate trajectory sequence, obtaining the time similarity between the current frame candidate trajectory sequence and the historical candidate trajectory sequence; According to the visual similarity, the spatial similarity and the temporal similarity, a loss matrix between the current frame candidate trajectory sequence and the historical candidate trajectory sequence is constructed, and the loss matrix is ​​solved by the Hungarian algorithm to perform trajectory association matching on the candidate trajectories in the current frame candidate trajectory sequence and the candidate trajectories in the historical candidate trajectory sequence to obtain the target tracking result in the multi-target tracking video.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the multi-target tracking method based on spatiotemporal trajectory association as described in any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-target tracking method based on spatiotemporal trajectory association as described in any one of claims 1 to 4 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multi-target tracking method based on spatiotemporal trajectory association as described in any one of claims 1 to 4 is implemented.

Images