A spatio-temporal fusion multi-target tracking method, device, equipment and medium

Abstract

The application belongs to the technical field of multi-target tracking, and relates to a spatio-temporal fusion multi-target tracking method, device, equipment and medium. The method comprises the following steps: acquiring continuous frame images and performing detection processing to obtain targets in historical frames and a current frame image; for the targets in the historical frame images, a trajectory time sequence diagram of each target is established; for the targets in the current frame image, a target space diagram is established when the detection box of the target is high confidence; node feature propagation and fusion are respectively performed on the trajectory time sequence diagram and the target space diagram to obtain trajectory features and space features; feature coding is respectively performed by using a time encoder and a space encoder, and the decoders are simultaneously input to perform spatio-temporal fusion to obtain target features; prediction is performed according to the target features to obtain position changes and template features, and the position changes and the template features are matched with the targets in the current frame image to realize multi-target tracking. The application can guarantee the accuracy of detection and tracking.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-target tracking, and in particular to a spatio-temporal fusion multi-target tracking method, device, equipment and medium. BACKGROUND

[0002] Multi-target tracking is a challenging key technology in computer vision, which is widely used in automatic driving, intelligent monitoring and other fields. The main task of multi-target tracking is to track multiple targets in the field of view at the same time, and to distinguish different targets. The key technical points include dynamically determining the number of targets in the frame and keeping the number of existing target trajectories and assigning numbers to newly appearing targets. The conventional workflow of multi-target tracking is as follows: (1) a plurality of continuous sequences are given; (2) a target detector is run to obtain the bounding box of the target; (3) for each detected object, different features are calculated, usually appearance and motion features; (4) similarity calculation between targets is performed according to the extracted features, the probability that two objects belong to the same target is calculated, and matching is performed according to the probability. Multi-target tracking algorithms can be divided into tracking based on detection and end-to-end tracking paradigm. In the current research, the tracking based on detection algorithm has better performance in tracking stability and accuracy.

[0003] After the detector obtains the target box, the subsequent tracker is essentially a similarity matching, which aims to maximize the feature distance between different objects in the feature space, while minimizing the feature distance of the same object. The features used by the tracker can be divided into motion features and appearance features. Early trackers are mostly based on motion models, using Kalman filter to predict the position of the current frame according to the position of the target in the previous key frame, and then matching with the detection result. Some algorithms directly calculate the overlap between the trajectory (previous frame) and the detection, without using Kalman filter to predict the future position. This method can achieve an inference speed of nearly 100kfps (detection time is not calculated), and this type of method works well when the object motion is small, but performs poorly in crowded scenes with occlusion and fast human movement. Some appearance feature-based methods crop part of the image region, output to the Re-ID network, extract appearance features, calculate the similarity between the trajectory and the feature, and use the Hungarian algorithm for matching. This type of method can handle fast motion and occlusion, but the Re-ID network is limited in development, the extracted features have low robustness and are difficult to train. In the tracking algorithm, combining motion and appearance features can reduce the situation of mis-matching and missing-matching, and the key to improving the performance of the algorithm is how to extract and fuse the features and how to design the matching mechanism.

[0004] In the prior art, the multi-target tracking technology framework can be divided into the detection and tracking joint paradigm and the tracking based on detection paradigm.

[0005] The detection and tracking joint paradigm is to integrate the detection task and the tracking task into one model. DT proposes a joint detection and tracking network architecture for frame-based object detection and cross-frame tracking regression. The input of this network is a video sequence composed of multiple frames. While performing object detection, it predicts the displacement of the target bounding box between frames. Tracktor predicts the position of the target in the next frame by using the bounding box regression of the detector, converting the detector into a tracker. CenterTrack uses pairs of adjacent frames as input to predict the detection box of the target and associates the target between two frames by point distance, thereby predicting the displacement of the target and completing the tracking of the target. It also provides a point-based heatmap trajectory. The model can match targets at any location, even if the boxes are completely non-overlapping. FairMOT proposes to solve the fairness problem in the learning of anchor-based single-stage multi-object tracking structure (such as CenterTrack and FairMOT). These models usually use pre-defined anchors for object detection, but due to the different number and distribution of targets of different sizes, this can lead to unfair competition between targets of different sizes. FairMOT uses a keypoint-based solution that can adaptively detect the size and distribution of targets and reduce the unfair impact between target sizes. This method can improve the robustness and scalability of the model, while improving the fairness of multi-object tracking. Since the detection and tracking are integrated into one network, the multi-task joint training is not ideal, and the accuracy is low, so most current methods use the detection-based tracking paradigm to maximize the accuracy of detection and tracking.

[0006] The detection-based tracking paradigm is the paradigm adopted by most modern trackers, where a detector finds all targets in each individual frame, and tracking is a bounding box association problem. SORT is a representative work of the detection-based tracking paradigm, whose main idea is to separate the target detection and target tracking, using a detector to detect all targets in each frame, applying Kalman filter to track the bounding boxes, and using binary matching to associate each bounding box with the detection with the highest overlap in its current frame. DeepSORT introduces deep visual features for target association in the SORT framework, influenced by the rapid development of convolutional neural networks and target detectors. In addition to using motion features of Kalman filter for matching, target matching based on appearance features is also a hot research topic, but in many cases the robustness of appearance features is insufficient, especially when the scene is crowded, the target representation is rough (such as bounding box), and the target appearance is difficult to distinguish. Appearance features are prone to confusion, leading to missed and false detections. Some works construct a target detection graph structure on all frames to represent the association problem, such as multicut. However, due to the large computational resource requirements of the graph structure, global optimization needs to be performed for large graph calculations, which limits the application of graph-based methods in online tracking. For the tracking task, the feature association extraction of the trajectory changes over time is very important, and the traditional deep network will have a forgetting situation due to the limitation of the structure, and it is difficult to effectively associate the features of different sequences. In recent years, the emergence of the Transformer algorithm is more effective for processing information association of different modalities and different sequences, which has greatly improved the performance of various perception tasks. The application of the Transformer structure in the multi-target tracking task has also emerged a lot of work, TransTrack and TrackFormer are the pioneering research of applying Transformer in MOT, both of which use DETR for target detection and feature extraction, and apply transformer to model the space-time relationship between trajectories and detections. Trackformer

[35] uses Transformer to encode the object features of all frames, and uses tracking queries to associate the detection results of different frames. TransTrack uses the Trackformer architecture to use the target features of the previous frame as the query of the current frame, and introduces a set of learned target queries to detect newly appearing targets. The TransMOT framework combines spatial graph structure and Transformer to model the spatial relationship between trajectories and detections, and uses spatial and temporal Transformer encoders to learn the spatial and temporal relationship between targets.TransCenter proposes a multi-object tracking framework of twin Transformer, taking the adjacent two frames as input, and two Transformer branches are used for tracking and detection respectively, and finally predicting the target center heat map, target size and displacement. MOTR takes video sequence as input, and uses query-key mechanism for inter-frame propagation and update of tracking frame. But the traditional DETR structure applied in the tracking task has low performance, and there is still a lot of room for improvement for the design of query-key mechanism and the related structure of attention mechanism.

[0007] However, multi-object tracking algorithms can be divided into tracking based on detection and end-to-end tracking paradigm. In the current research, the tracking algorithm integrating detection task and tracking task into one model is not ideal in multi-task joint training, and the precision is low. For the tracking task, the spatio-temporal structure of the target is necessary to solve long-time tracking and occlusion jump. The modeling ability of traditional social models or motion models is limited, and the traditional deep network will have a forgetting situation due to the limitation of structure, and it is difficult to effectively associate the features of different sequences. In recent years, the emergence of Transformer algorithm is more effective for processing information association of different modalities and different sequences. Most multi-object tracking works based on the DETR of the detection architecture, and their performance is relatively low. Since multi-object tracking will process many objects at the same time, the spatio-temporal processing performance of Transformer is not enough to model the spatio-temporal dependence of multiple target trajectories, and it needs a large amount of data and a large amount of computing resources to extract spatio-temporal structure. The deep learning model based on graph convolution has been verified in many tasks in recent years in processing spatio-temporal structure. Therefore, graph+Transformer is used to model spatio-temporal structure. The Transformer architecture is a deterministic reasoning network. Since the information contained by the same target at each time node is different, the extraction of the general characteristics of the trajectory is essentially uncertain reasoning, so an uncertainty mechanism is added to the traditional attention mechanism. Only relying on single frame / adjacent frame has low performance in the past multi-object tracking work, and only relying on the propagation between adjacent frames has weak long-time dependence learning ability. Due to the interaction between targets and the motion of the target itself, the current frame / previous frame is not enough to represent the features of the entire trajectory, and it is necessary to extract from multiple frames to improve the accuracy of the template for trajectory description. Therefore, the trajectory features of multiple frames are used to explicitly establish the trajectory time graph and the single frame space graph. SUMMARY

[0008] Therefore, it is necessary to solve the above technical problems, provide a spatio-temporal fusion multi-object tracking method, device, equipment and medium, which can ensure the accuracy of detection and tracking.

[0009] The spatio-temporal fusion multi-object tracking method comprises:

[0010] Acquire continuous frame images and perform detection processing to obtain the target in the historical frame image and the target in the current frame image;

[0011] For the target in the historical frame image, feature extraction is performed according to the positions of the same target in different frame images to establish a trajectory time sequence diagram of each target; for the target in the current frame image, when the detection box of the target is high confidence, feature extraction is performed according to the positions of different targets in the current frame image to establish a target space diagram;

[0012] The graph convolutional neural network is used to perform node feature propagation and fusion on the trajectory time sequence diagram and the target space diagram respectively to obtain trajectory features and space features;

[0013] The time encoder is used to perform feature encoding on the trajectory features, the space encoder is used to perform feature encoding on the space features, and the decoder is simultaneously input to perform space-time fusion to obtain target features;

[0014] According to the target features, the position change and the template features are obtained, and the target in the current frame image is matched to realize multi-target tracking.

[0015] In one embodiment, when the time encoder is used to perform feature encoding on the trajectory features and the space encoder is used to perform feature encoding on the space features, the time encoder and the space encoder introduce uncertainty noise to generate multiple attention distributions, and sampling is performed according to the probability distribution of each category.

[0016] In one embodiment, introducing uncertainty noise to generate multiple attention distributions and sampling according to the probability distribution of each category includes:

[0017]

[0018] g i =-log(-log(u)),u~U(0,1)

[0019] In the formula, is a random attention distribution, q i is the query vector of the i-th attention head, is the transpose of the key vector of the i-th attention head, g i is a random vector sampled from the Gumbel distribution, τ is a temperature parameter, and u is a random number from the uniform distribution [0, 1].

[0020] In one embodiment, an additional random attention is added in front of the query vector and the key vector to obtain a new key vector:

[0021]

[0022]

[0023] wherein, is the sampled categorical distribution, G is the Gumbel-softmax function, τ1 is a temperature coefficient, C is a cluster center, is a new key vector obtained by weighted average according to the weight of each center.

[0024] In one embodiment, the time encoder and the space encoder each comprise a plurality of Transformer encoding layers, and each Transformer encoding layer comprises, connected in sequence: a multi-head self-attention layer and a fully connected feed-forward neural network layer.

[0025] In one embodiment, the position change and the template feature are obtained by predicting according to the target feature, and the multi-target tracking is realized by matching the target in the current frame image, including:

[0026] The position change and the template feature are obtained by predicting according to the target feature;

[0027] For the position change, the IOU distance with the detection box of the current frame is calculated as the position similarity;

[0028] For the template feature, the feature similarity with the high-confidence detection box of the current frame is calculated;

[0029] According to the position similarity and the feature similarity, the Hungarian algorithm is used to match the target in the current frame image, and the multi-target tracking is realized.

[0030] In one embodiment, for the target in the current frame image, when the detection box of the target is low-confidence, the Kalman filtering algorithm is used to extract the feature, predict the position change, and match the target in the current frame image according to the position of different targets in the current frame image, and the multi-target tracking is realized.

[0031] The space-time fusion multi-target tracking device comprises:

[0032] The acquisition module is configured to acquire continuous frame images and perform detection processing to obtain the targets in the historical frame images and the targets in the current frame image;

[0033] The composition module is configured to, for the targets in the historical frame images, extract the features according to the positions of the same target in different frame images to establish a trajectory time sequence diagram of each target; and for the targets in the current frame image, when the detection box of the target is high-confidence, extract the features according to the positions of different targets in the current frame image to establish a target space diagram.

[0034] a convolution module, configured to perform node feature propagation and fusion on the trajectory time sequence graph and the target space graph respectively by using a graph convolutional neural network, to obtain trajectory features and space features;

[0035] a fusion module, configured to perform feature coding on the trajectory features by using a time encoder, perform feature coding on the space features by using a space encoder, and input a decoder simultaneously for spatio-temporal fusion to obtain target features;

[0036] a tracking module, configured to perform prediction according to the target features, to obtain position changes and template features, and to match the target features with targets in the current frame image to realize multi-target tracking.

[0037] A computer device, comprising a memory and a processor, the memory storing a computer program, and the processor realizing the following steps when executing the computer program:

[0038] obtaining continuous frame images and performing detection processing to obtain targets in historical frame images and targets in a current frame image;

[0039] for the targets in the historical frame images, performing feature extraction according to positions of the same target in different frame images to establish a trajectory time sequence graph of each target, and for the targets in the current frame image, when a detection box of a target is high confidence, performing feature extraction according to positions of different targets in the current frame image to establish a target space graph;

[0040] performing node feature propagation and fusion on the trajectory time sequence graph and the target space graph respectively by using a graph convolutional neural network to obtain trajectory features and space features;

[0041] performing feature coding on the trajectory features by using a time encoder, performing feature coding on the space features by using a space encoder, and inputting a decoder simultaneously for spatio-temporal fusion to obtain target features;

[0042] performing prediction according to the target features to obtain position changes and template features, and matching the target features with targets in the current frame image to realize multi-target tracking.

[0043] A computer readable storage medium, storing a computer program, the computer program being executed by a processor to realize the following steps:

[0044] obtaining continuous frame images and performing detection processing to obtain targets in historical frame images and targets in a current frame image;

[0045] For the target in the historical frame image, feature extraction is performed according to the positions of the same target in different frame images, and a trajectory time sequence diagram of each target is established; for the target in the current frame image, when the detection box of the target is high confidence, feature extraction is performed according to the positions of different targets in the current frame image, and a target space diagram is established;

[0046] The trajectory feature and the space feature are obtained by adopting a graph convolutional neural network to perform node feature propagation and fusion on the trajectory time sequence diagram and the target space diagram respectively;

[0047] The trajectory feature is encoded by adopting a time encoder, the space feature is encoded by adopting a space encoder, and the target feature is obtained by simultaneously inputting the decoder for spatio-temporal fusion;

[0048] According to the target feature, the position change and the template feature are obtained, and are matched with the target in the current frame image, so as to realize multi-target tracking.

[0049] The above-mentioned spatio-temporal fusion multi-target tracking method, device, equipment and medium are a spatio-temporal multi-modal graph fusion non-deterministic multi-target tracking method, follow the tracking paradigm based on detection, construct a two-stage tracking algorithm framework, perform hierarchical tracking according to detection confidence, combine motion features and appearance features, robustly cope with irregular target motion and appearance changes in long-time tracking, so as to improve the robustness of the tracking algorithm and maximize the accuracy of detection and tracking. The present application designs a spatio-temporal graph fusion uncertainty tracking network, and the spatio-temporal graph information fusion architecture adopts a coding-decoding structure based on Transformer, and an uncertainty mechanism is introduced therein, that is, an uncertainty attention mechanism is adopted to form the coding-decoding structure, so as to adaptively learn the appearance template and the displacement, fuse the multi-modal features such as target representation, motion trajectory and confidence, model the multi-frame time sequence information of the trajectory and the spatial relationship of the candidate target by using the graph structure, and explicitly describe the target features independently. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 It is a scene diagram for application of the spatio-temporal fusion multi-target tracking method in one embodiment;

[0051] Figure 2 It is a flowchart of the spatio-temporal fusion multi-target tracking method in one embodiment;

[0052] Figure 3 It is a hierarchical random attention calculation schematic diagram in one embodiment;

[0053] Figure 4 It is a hierarchical random attention calculation schematic diagram in one embodiment;

[0054] Figure 5 A schematic diagram of a prediction head in one embodiment;

[0055] Figure 6 A schematic diagram of a framework of a spatio-temporal graph fusion uncertainty tracking network in one embodiment;

[0056] Figure 7 A schematic diagram of a framework of a spatio-temporal fusion multi-target tracking method in one embodiment;

[0057] Figure 8 A tracking result diagram of a street scene small target in one specific embodiment, wherein (a) is a first frame sequence, (b) is a second frame sequence, and (c) is a third frame sequence;

[0058] Figure 9 A tracking result diagram of a crowded group of people in one specific embodiment, wherein (a) is a first frame sequence, (b) is a second frame sequence, and (c) is a third frame sequence;

[0059] Figure 10 A tracking result diagram of an occlusion in one specific embodiment, wherein (a) is a first frame sequence, (b) is a second frame sequence, and (c) is a third frame sequence;

[0060] Figure 11 A tracking result diagram of a campus clothing similar and crowded group in one specific embodiment, wherein (a) is a first frame sequence, (b) is a second frame sequence, and (c) is a third frame sequence;

[0061] Figure 12 A tracking result diagram of an off-road scene occlusion in one specific embodiment, wherein (a) is a first frame sequence, (b) is a second frame sequence, and (c) is a third frame sequence;

[0062] Figure 13 A tracking result diagram of an off-road scene occlusion between targets in one specific embodiment, wherein (a) is a first frame sequence, (b) is a second frame sequence, and (c) is a third frame sequence;

[0063] Figure 14 A structural block diagram of a spatio-temporal fusion multi-target tracking device in one embodiment;

[0064] Figure 15 An internal structure diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0065] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0066] It should be noted that all directional indications, such as upper, lower, left, right, front, back, etc., described in the embodiments of the present application are only used to explain the relative position relationship, movement condition, etc. between components in a certain posture (as shown in the drawings), and if the certain posture changes, the directional indications also change accordingly.

[0067] In addition, the description such as "first", "second" and the like in the present application is only for the purpose of description, and should not be understood as indicating or implying the relative importance of the technical features indicated or implying the number of technical features indicated. Therefore, the features defined as "first" and "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple groups" is at least two groups, such as two groups, three groups, etc., unless otherwise specifically limited.

[0068] In the present application, unless otherwise specifically defined and limited, the terms "connection", "fixing" and the like should be understood broadly, for example, "fixing" can be fixed connection, or detachable connection, or integral; can be mechanical connection, or electrical connection, or physical connection or wireless communication connection; can be directly connected, or indirectly connected through intermediate medium, or internal communication of two elements or interaction relationship between two elements, unless otherwise specifically limited. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0069] In addition, the technical solutions of each embodiment of the present application can be combined with each other, but it must be based on the fact that those of ordinary skill in the art can realize it, and when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, nor is it within the scope of protection claimed by the present application.

[0070] The method provided by the present application can be applied to the application environment as shown in Figure 1 The terminal 102 communicates with the server 104 through the network, and the terminal 102 can include but is not limited to various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices, and the server 104 can be a server corresponding to various portal websites, work system backends, etc.

[0071] The application provides a spatio-temporal fusion multi-target tracking method, as shown in Figure 2 In one embodiment, the method is applied to a terminal in Figure 1 , and includes the following steps.

[0072] In step 202, continuous frame images are acquired and detection processing is performed to obtain targets in historical frame images and targets in a current frame image.

[0073] Specifically, the continuous frame images refer to a series of images with consecutive frame numbers, including the current frame image and the historical frame images before the current frame image.

[0074] The detection processing refers to inputting the images (current frame image or historical frame image) into a detector to obtain the targets in the images.

[0075] The specific detector can adopt YOLOV7, YOLOX or other detection networks. Among them, YOLOV7 is a lightweight and efficient target detection algorithm framework, which can maintain a fast detection speed without affecting the detection accuracy, which is very necessary for the real-time operation of the two-stage tracking algorithm in the application under the condition of ensuring the accuracy. YOLOv7 is an anchor-based detector that adopts the innovative idea of YOLOv5 and uses adaptive anchors. The adaptive anchors are generated by a clustering algorithm to adapt to the size and proportion of the target, avoiding the difficulties caused by manual setting of anchors. Compared with the fixed anchor-based detector, the adaptive anchor can adapt to a wider range of target sizes and proportions, enhancing the robustness of the model. The network structure of YOLOv7 mainly consists of three parts: feature extraction network (Backbone), feature pyramid network (FPN) and detection head (Head). The backbone network of YOLOv7 feature extraction is Darknet53. Similar to YOLOv5, YOLOv7 also adopts FPN structure, which can fuse features of different scales to generate a multi-layer feature pyramid, which can help YOLOv7 network to detect targets of different sizes. The final prediction head part of the network adopts the head structure of YOLOv5, which adopts multi-scale prediction and SPP structure, which can better adapt to targets of different scales and sizes, improving the detection accuracy. YOLOv7 adds adaptive anchors, multi-scale prediction, SPP-FPN structure and other frontiers in the field of target detection to the traditional YOLO series, which significantly improves the accuracy while achieving high-efficiency real-time inference speed.

[0076] In this step, a detector is designed.

[0077] Step 204, for the target in the historical frame image, feature extraction is performed according to the position of the same target in different frame images to establish a trajectory time sequence diagram of each target; for the target in the current frame image, when the detection box of the target is high confidence, feature extraction is performed according to the position of different targets in the current frame image to establish a target space diagram.

[0078] Specifically: for the historical trajectory and the target of the current frame, first, the feature of each target region is extracted by using a convolutional neural network, and then the spatio-temporal features of the historical trajectory and the detection box of the current frame are modeled by using a graph structure.

[0079] In the multi-target tracking (MOT) task, the detection and tracking of targets are usually performed in a single frame / adjacent frame manner. However, due to the interaction between targets and the motion of the targets themselves, only using the information of the current frame / previous frame is insufficient to represent the features of the entire trajectory, which can lead to inaccurate description of the trajectory. In order to improve the accuracy of the final template in describing the trajectory and predicting the displacement of the trajectory, the present method uses the target feature of the previous n frames of the current frame to represent the historical trajectory, models and predicts the motion trajectory of the target by using multi-frame information, which can more accurately describe the motion law and behavior pattern of the target. At the same time, multi-frame information can better capture the association and interaction between targets, thereby improving the recognition and differentiation ability of the targets.

[0080] A graph structure is established for each trajectory, which is more accurate in describing the trajectory itself than the previous tracking method of establishing a target space diagram for each frame of the history. The trajectory time sequence information is not disturbed by other trajectories, the information extraction is not redundant, and the number of saved frames of the trajectory can be increased, further improving the accuracy of tracking. The graph structure is composed of nodes and edges, and is a flexible and expandable data structure. Compared with the traditional linear data structure, the graph structure can represent data of any shape, has better expandability and adaptability. The processing of new nodes or edges is more efficient than the matrix structure, and nodes or edges can be easily added or deleted. In the matrix structure, missing data needs to be filled with special values (such as 0), which introduces additional noise. For depicting the spatio-temporal relationship of the trajectory in the tracking task, the graph structure can intuitively and efficiently represent the connection of each target at different time points, which is more in line with human thinking and provides more interpretability.

[0081] In this step, a spatio-temporal multi-modal data graph modeling is designed.

[0082] Step 206, a graph convolutional neural network is used to propagate and fuse the node features of the trajectory time sequence diagram and the target space diagram respectively to obtain the trajectory features and the spatial features.

[0083] Specifically: after modeling the trajectory time sequence diagram and the target space diagram of the current frame, a graph convolutional neural network (GCN) is used to fuse and transfer the features of each diagram. For each node feature, the GCN exchanges information with its neighbor nodes to obtain a new node representation.

[0084] The basic formula of GCN information transmission is:

[0085]

[0086] wherein H (1) denotes the node feature matrix of the lth layer GCN, wherein each row represents a feature vector of a node, and the matrix concatenates all the node features of the input diagram; is the core part of the GCN, representing the process of normalizing and weighting the adjacency matrix, which is to eliminate the influence of uneven weights between nodes with different degrees, and also to avoid gradient explosion; A represents the adjacency matrix of the graph structure; is to add a unit matrix to the adjacency matrix: The unit matrix I is used to ensure that each node has a self-loop connection; D represents the degree matrix of the graph, and the diagonal elements are the sum of the degrees of each node; W (l) is the weight matrix of the lth layer GCN, which is used to map the feature vector of each node to a new feature space; σ(·) is an activation function, and the node feature matrix H (1+1) of the next layer GCN is finally obtained.

[0087] The multi-layer GCN is used to perform layer-by-layer transmission of graph information and extraction of features, and the features of the fused nodes are obtained. The feature map extracted by the multi-layer GCN is input into the subsequent space-time encoder and decoder for space-time fusion.

[0088] In this step, a graph convolutional encoder is designed.

[0089] In step 208, the trajectory features are encoded by a time encoder, and the space features are encoded by a space encoder, and the decoder is input at the same time for space-time fusion to obtain target features.

[0090] Specifically:

[0091] The trajectory features and space features extracted by the multi-layer GCN are further extracted by the time encoder and the space encoder respectively. The time encoder and the space encoder are both composed of multiple Transformer encoding layers, the difference being that the time encoder encodes the time dimension of each historical trajectory, and the space encoder encodes the candidate target space dimension of the current frame.

[0092] Both the temporal encoder and the spatial encoder include multiple Transformer coding layers, which consist of a multi-head self-attention layer and a fully connected feedforward neural network layer connected in sequence.

[0093] like Figure 3 As shown, the structure of a single spatiotemporal coding layer is presented. The input features first pass through a multi-head self-attention layer to calculate a weighted input vector representation, where each dimension of the feature is influenced by the other dimensions. The weighted output of the self-attention layer and the original input features are then subjected to residual connection and normalization operations by the Add&Norm module. The subsequent feedforward layer is a fully connected feedforward neural network layer that outputs a new vector through two linear transformations and an activation function. Its function is to perform nonlinear transformation and recombination on the features at each position, thereby enhancing the expressive power of the model. The output of the feedforward layer is again subjected to residual connection and normalization operations by the Add&Norm module to finally obtain the output features. The Transformer coding layers are stacked together to form a multi-layer spatiotemporal encoder.

[0094] The spatiotemporal encoder in this application is not a standard self-attention mechanism. The standard Transformer attention architecture is a deterministic inference network. For heteroscedastic data (such as different frame representations of a trajectory), some frame inputs may have more noise outputs than others. Therefore, the sample feature fusion and propagation are dynamically adjusted by modeling its uncertainty. In view of this, this application applies a novel uncertain attention mechanism to the encoding layer, enabling the Transformer to have uncertain inference capabilities. Unlike traditional attention mechanisms, this mechanism generates multiple attention distributions by introducing uncertain noise, thereby achieving the purpose of uncertain inference.

[0095] Assuming there are n attention heads in the Transformer encoding layer, q i k i and v i Let represent the query vector, key vector, and value vector of the i-th attention head, respectively. The input graph features have a dimension of x×d, representing x targets, each with a feature dimension of d. The attention calculation process is as follows:

[0096]

[0097]

[0098] Where d represents the dimension of the input feature, a i q i and ki The calculated attention distribution h i represents the output of the i-th attention head.

[0099] When the trajectory features are encoded by the time encoder and the spatial features are encoded by the space encoder, the time encoder and the space encoder introduce uncertainty noise to generate multiple attention distributions, and each category is sampled according to the probability distribution, rather than using a softmax function to calculate the probability of each category and selecting the category with the maximum probability as the output, in order to pay attention to the hidden information between the probabilities and better utilize the inherent association behind the data.

[0100] Specifically:

[0101]

[0102] g i = -log(-log(u)), u ~ U(0, 1)

[0103] In the formula, q is a random attention distribution, i is the query vector of the i-th attention head, is the transpose of the key vector of the i-th attention head, g i is a random vector sampled from the Gumbel distribution, τ is a temperature parameter, and u is a random number from the uniform distribution [0, 1].

[0104] It should be noted that the smaller τ is, the closer the sampling distribution is to the one-hot vector, which means that the model is more inclined to choose the element with the maximum probability; the larger τ is, the smoother the sampling distribution is, which means that the model is more likely to explore elements with smaller probabilities; therefore, by adjusting the temperature parameter τ, the exploration ability and the degree of certainty of the model can be controlled, and in practice τ will be adjusted in some pre-set manner to balance the trade-off between prediction performance and uncertainty estimation.

[0105] GUMBEL-softmax achieves this by replacing the log-probability of softmax with the form of log-sample GUMBEL distribution, the self-attention mechanism of sampling from Gumbel distribution is an approximation of the upper bound, the randomness comes from the Gumbel sampling classification distribution, and the Gumbel distribution is used for sampling, which increases the randomness of model decision-making and makes the decision space more smooth, and the sampling process is not in the computation graph, so the subsequent derivation can be performed.

[0106] An additional random attention is added in front of the query vector and the key vector, which is equivalent to regularizing the key vector to obtain a new key vector, which participates in the subsequent random attention calculation, so as to further make the Transformer model have a certain randomness while maintaining the prediction performance. Finally, one attention head needs to go through two times of Gumbel sampling, forming a hierarchical random attention mechanism.

[0107] As shown in Figure 4 , the calculation process of the hierarchical random attention mechanism is given. First, a set of cluster centers C is defined, and the dimension of each cluster center is the same as that of each key vector k. Each key head randomly pays attention to a set of learnable cluster centers C using the Gumbel-Softmax distribution, C∈R d×c , to obtain a new key vector:

[0108]

[0109]

[0110] In the formula, is the sampled classification distribution, G is the Gumbel-softmax function, τ1 is the temperature coefficient, C is the cluster center, is the new key vector obtained by weighted average according to the weight of each center.

[0111] The new key vector is used in the subsequent random self-attention calculation:

[0112]

[0113]

[0114] Where, is the output of the final i-th random attention head, and τ2 is the temperature coefficient.

[0115] The random attention mechanism samples twice by Gumbel distribution, and randomly pays attention to the key vector and the value respectively, thereby increasing the uncertainty of the model. The attention mechanism in each encoding layer is replaced by the hierarchical random attention mechanism. The feature of each trajectory and the spatial graph feature of the target in the current frame are input into the decoder for fusion processing after being encoded by the space-time encoder. The decoder is composed of a multi-head cross-attention structure. The decoder takes the spatial graph feature of the current frame as the query and the trajectory graph feature as the key value, and calculates the output space-time fusion feature in the multi-head cross-attention structure. The input is input into the subsequent prediction head to predict the template feature and the position change.

[0116] In this step, an uncertainty space-time encoder-decoder is designed.

[0117] Step 210: Predict based on target features to obtain position changes and template features, and match them with targets in the current frame image to achieve multi-target tracking.

[0118] Specifically:

[0119] Predicting based on target features yields positional changes and template features. For positional changes, the Intersection over Union (IOU) distance with the current frame's detection bounding box is calculated as positional similarity. For template features, feature similarity with the current frame's high-confidence detection bounding box is calculated. Based on positional and feature similarities, the Hungarian algorithm is used to match targets in the current frame image, achieving multi-target tracking.

[0120] It should be noted that, because fully connected operations have higher spatial sensitivity and a stronger ability to distinguish between complete and partial objects than convolutional operations, they are more suitable for classification tasks that require sufficient semantic information. The prediction head is composed of fully connected operations, and its structure is as follows: Figure 5 As shown, the input is M×512 dimensional features, where M represents the number of historical trajectories. The template feature prediction head consists of a multilayer perceptron with a 512-dimensional input and a 256-dimensional output, and the position regression head consists of a multilayer perceptron with a 512-dimensional input and a 4-dimensional output. Each fully connected layer uses the ReLU activation function, and the output of the last layer does not use an activation function. The final network outputs a 256-dimensional template feature and a 4-dimensional position coordinate for each historical trajectory.

[0121] In this step, a decoupled prediction head is designed.

[0122] In this embodiment, a detector was designed first, and then a spatiotemporal graph fusion uncertainty tracker was designed, which consists of spatiotemporal multimodal data graph modeling, graph convolutional coding, uncertainty spatiotemporal codec, and decoupled prediction head.

[0123] The spatiotemporal graph fusion uncertainty tracker aims to learn the appearance feature templates and displacement values ​​of historical trajectories, and then use the learned trajectory features to match the detected high-confidence bounding boxes.

[0124] Given the current frame F t Test results and all historical trajectories from time tn to time t-1 Where N represents the number of detected targets at time t, and M represents the number of historical trajectories from time tn to time t-1. Detection results Including target area and the location of the target Ultimately, the output of this model is the historical trajectory S. t-1 Feature template and its position in the current image frame t The network obtains the historical trajectory template features and displacement values, respectively, and performs similarity calculation with the target frame to obtain a correlation matrix Q N×M With matching trajectories and candidate targets, the calculation of the correlation matrix Q can be represented by the following formula:

[0125]

[0126] wherein Q ij represents the matching similarity of the ith detected target and the jth historical trajectory, represents the similarity between the trajectory template and the candidate target features, which can be measured by the inner product space isometric, E represents the feature encoder, IOU represents the intersection over union (IOU) distance between the network predicted trajectory displacement value and the candidate target frame, the correlation matrix Q considers the similarity of the appearance features and the motion features, β represents the weight ratio coefficient of the appearance feature and the motion feature similarity.

[0127] The spatio-temporal graph fusion uncertainty tracking network, as shown in Figure 6 , contains graph modeling of the time sequence relationship of multiple frames of trajectories and the spatial relationship of the current frame detection frame, uncertainty spatio-temporal encoder, and finally the prediction head of the template features and the position.

[0128] As shown in Figure 7As shown, in this embodiment, two stages of detector and tracker are divided, the detector uses YOLOv7 network framework, the tracker combines motion model and spatio-temporal fusion uncertainty tracking model, and tracks according to the confidence level of the detection frame. Specifically: after the target in the image is detected by using the YOLOv7 detector, the target frame position and confidence obtained by the detector are input into the designed hierarchical tracker. The tracker divides the detection frame into high confidence and low confidence frames according to the confidence threshold for hierarchical processing. For high confidence detection frame matching, a deep learning tracker, namely a spatio-temporal graph fusion uncertainty tracker, is adopted. The tracker is responsible for predicting the displacement value of the historical trajectory and the appearance template feature. For the predicted appearance template feature and the high confidence detection frame feature of the current frame, the similarity is calculated. For the predicted historical trajectory displacement value, the IOU distance between the current frame detection frame position is calculated as the similarity. The two similarity values are comprehensively utilized to match the unmatched target frame and historical trajectory by using the Hungarian algorithm. The motion feature is used to match the frames of the unmatched high confidence frame and low confidence frame. The Kalman filtering algorithm is used to predict the target position, and the position distance is calculated as the similarity. The Hungarian algorithm is used for matching (that is, for the target in the current frame image, when the detection frame of the target is low confidence, according to the position of different targets in the current frame image, the Kalman filtering algorithm is used for feature extraction, the position change is predicted, and the target in the current frame image is matched to realize multi-target tracking). The hierarchical tracker updates the target trajectory position, newly builds the target trajectory, and removes the unmatched historical trajectory according to the matching result.

[0129] The above-mentioned spatio-temporal fusion multi-target tracking method is a spatio-temporal multi-modal graph fusion non-deterministic multi-target tracking method, which follows the tracking paradigm based on detection, constructs a two-stage tracking algorithm framework, performs hierarchical tracking according to the detection confidence, combines motion features and appearance features, robustly copes with irregular target motion and appearance changes in long-time tracking, and improves the robustness of the tracking algorithm to maximize the accuracy of detection and tracking. The application designs a spatio-temporal graph fusion uncertainty tracking network. The spatio-temporal graph information fusion architecture adopts a coding-decoding structure based on Transformer, and an uncertainty mechanism is introduced therein, that is, an uncertainty attention mechanism is used to form the coding-decoding structure to adaptively learn the appearance template and the displacement. The multi-modal features such as target representation, motion trajectory and confidence are fused, the graph structure is used to model the multi-frame time sequence information of the trajectory and the spatial relationship of the candidate target, and the target features are explicitly described independently.

[0130] In view of the problem that the multi-task joint training of the tracking algorithm integrating the detection task and the tracking task into one model in the prior art is not ideal and the accuracy is low, the detection-based tracking paradigm is adopted to maximize the accuracy of detection and tracking. For the tracking task, the spatiotemporal structure of the target is necessary to solve long-time tracking and occlusion jump, the modeling capability of the traditional social model or motion model is limited, and the traditional deep network will exist forgetting due to the limitation of the structure, and it is difficult to effectively associate the features of different sequences. In recent years, the emergence of the Transformer algorithm is more effective for processing information association of different modalities and different sequences. Most multi-target tracking works are based on the DETR of the detection architecture, and the performance is relatively low. Since the multi-target tracking processes many objects at the same time, the spatiotemporal processing performance of the Transformer is not enough to model the spatiotemporal dependence of multiple target trajectories, and the extraction of the spatiotemporal structure needs a large amount of data and consumes a large amount of computing resources. The deep learning model based on graph convolution has been verified in handling spatiotemporal structure in recent years in multiple tasks. Therefore, the graph+Transformer is adopted to model the spatiotemporal structure. The Transformer architecture is a deterministic reasoning network, and since the information contained by the same target at each time node is different, the general feature extraction of the trajectory is essentially uncertain reasoning, and therefore the uncertainty mechanism is added in the traditional attention mechanism. Only relying on single frame / adjacent frame has low performance in the past multi-target tracking work, and only relying on the propagation between adjacent frames has weak long-time dependence learning ability. Due to the interaction between targets and the motion of the target itself, the current frame / previous frame is not enough to represent the features of the entire trajectory, and it is necessary to extract from multiple frames to improve the accuracy of the template for trajectory description. Therefore, the multi-frame target trajectory features are adopted, and the trajectory time graph and the single-frame spatial graph are explicitly established.

[0131] It should be understood that, although Figure 2 The steps in the flowchart of the method are shown in sequence according to the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, Figure 2 At least part of the steps in the method can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0132] In a specific embodiment, multi-target tracking is performed for different scenes.

[0133] Figures 8 to 10The tracking visualization results on the public dataset MOT17 are shown. (a)(b)(c) in each figure represent the tracking results of a sequence of consecutive frames, and different boxes represent different targets, each of which is marked with its ID number in the upper left corner.

[0134] Figures 11 to 13 The tracking results in the campus scene and the off-road scene are shown.

[0135] The above visualization examples contain difficult scenes such as occlusion, crowding, small targets, similar targets, etc. The method can still stably track targets in these scenes.

[0136] The application also provides a spatio-temporal fusion multi-target tracking device, as shown in Figure 14 In one embodiment, the device comprises an acquisition module 1402, a composition module 1404, a convolution module 1406, a fusion module 1408 and a tracking module 1410, wherein:

[0137] The acquisition module 1402 is configured to acquire consecutive frame images and perform detection processing to obtain targets in historical frame images and targets in a current frame image.

[0138] The composition module 1404 is configured to, for the targets in the historical frame images, perform feature extraction according to the positions of the same target in different frame images to establish a trajectory time sequence diagram of each target; and for the targets in the current frame image, when the detection box of the target is high confidence, perform feature extraction according to the positions of different targets in the current frame image to establish a target space diagram.

[0139] The convolution module 1406 is configured to use a graph convolutional neural network to perform node feature propagation and fusion on the trajectory time sequence diagram and the target space diagram respectively to obtain trajectory features and space features.

[0140] The fusion module 1408 is configured to use a time encoder to perform feature encoding on the trajectory features, use a space encoder to perform feature encoding on the space features, and simultaneously input a decoder for spatio-temporal fusion to obtain target features.

[0141] The tracking module 1410 is configured to perform prediction according to the target features to obtain position changes and template features, and match the position changes and the template features with the targets in the current frame image to realize multi-target tracking.

[0142] The specific limitation of the multi-target tracking device of spatio-temporal fusion can refer to the limitation of the multi-target tracking method of spatio-temporal fusion in the above, which will not be repeated here. Each module in the above device can be realized by software, hardware and combination thereof in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to call and execute the operation corresponding to each module by the processor.

[0143] In one embodiment, a computer device, which can be a terminal, is provided, and an internal structure diagram thereof can be as shown in FIG. 1. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected by a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a multi-target tracking method of spatio-temporal fusion. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc. Figure 15 The person skilled in the art can understand that the structure shown in FIG. 1 is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0144] Figure 15 The person skilled in the art can understand that the structure shown in FIG. 1 is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0145] In one embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program, and the processor implements the steps of the method in the above embodiments when executing the computer program.

[0146] In one embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps of the method in the above embodiments.

[0147] ​Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, databases, or other media in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0148] The technical features of the above embodiments can be combined in any way. In order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0149] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the patent of the present application should be subject to the appended claims.

Claims

1. A spatio-temporal fusion multi-target tracking method, characterized in that, The method comprises the following steps: obtaining continuous frame images and performing detection processing to obtain targets in historical frame images and targets in a current frame image; for the targets in the historical frame images, performing feature extraction according to the positions of the same target in different frame images to establish a trajectory time sequence diagram of each target; for the targets in the current frame image, when the detection box of the target is high confidence, performing feature extraction according to the positions of different targets in the current frame image to establish a target space diagram; adopting a graph convolutional neural network to perform node feature propagation and fusion on the trajectory time sequence diagram and the target space diagram to obtain trajectory features and space features; adopting a time encoder to perform feature encoding on the trajectory features and adopting a space encoder to perform feature encoding on the space features, and simultaneously inputting a decoder to perform space-time fusion to obtain target features; performing prediction according to the target features to obtain position changes and template features, and matching the position changes and the template features with the targets in the current frame image to realize multi-target tracking; when the time encoder is adopted to perform feature encoding on the trajectory features and the space encoder is adopted to perform feature encoding on the space features, the time encoder and the space encoder introduce uncertainty noise to generate multiple attention distributions, and sampling is performed according to the probability distribution of each category; introducing uncertainty noise to generate multiple attention distributions and sampling according to the probability distribution of each category comprises: wherein, is a random attention distribution, is the query vector for the i th attention head, is the transpose of the key vector for the i th attention head, is a random vector sampled from Gumbel a distribution, is a temperature parameter, is a random number from a [0, 1] uniform distribution; adding an additional random attention to the query vector and the key vector to obtain a new key vector: wherein is the classification distribution of the samples, is Gumbel-softmax is the function, 1 is the temperature coefficient, is the cluster center, is the new key vector obtained by weighted average according to the weight of each center; performing prediction according to the target features to obtain position changes and template features, and matching the position changes and the template features with the targets in the current frame image to realize multi-target tracking comprises: performing prediction according to the target features to obtain position changes and template features; for the position changes, calculating an IOU distance with a current frame detection box as a position similarity; for the template features, calculating a feature similarity with a high-confidence detection box in the current frame; according to the position similarity and the feature similarity, utilizing a Hungarian algorithm to match the position changes and the template features with the targets in the current frame image to realize multi-target tracking.

2. The spatio-temporally fused multi-target tracking method of claim 1, wherein, The time encoder and the space encoder each comprise multiple Transformer encoding layers, and each Transformer encoding layer comprises, in sequence, a multi-head self-attention layer and a fully connected feedforward neural network layer.

3. The spatio-temporally fused multi-target tracking method of claim 1 or 2, wherein, for the targets in the current frame image, when the detection box of the target is low confidence, performing feature extraction according to the positions of different targets in the current frame image by adopting a Kalman filtering algorithm to predict position changes, and matching the position changes with the targets in the current frame image to realize multi-target tracking.

4. A spatio-temporal fusion multi-target tracking device, characterized by, The method for multi-target tracking comprises the following steps: obtaining continuous frame images and performing detection processing to obtain targets in historical frame images and targets in a current frame image; The composition module is configured to perform feature extraction on a target in a historical frame image according to a position of the same target in different frame images, establish a track time sequence diagram of each target, and perform feature extraction on a target in a current frame image according to a position of different targets in the current frame image when a detection box of the target is of high confidence, and establish a target space diagram. The convolution module is configured to perform node feature propagation and fusion on the track time sequence diagram and the target space diagram respectively by using a graph convolutional neural network, and obtain track features and space features. The fusion module is configured to perform feature coding on the track features by using a time encoder, perform feature coding on the space features by using a space encoder, and simultaneously input a decoder for spatio-temporal fusion to obtain target features. The tracking module is configured to perform prediction according to the target features, obtain position changes and template features, and match the position changes and the template features with targets in a current frame image to achieve multi-target tracking. 5.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-4 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method in any one of claims 1 to 3.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 3.

Images