A multi-object tracking method and system based on transformer and graph embedding

Abstract

This invention discloses a multi-target tracking method based on Transformer and graph embedding, comprising: acquiring a video sequence; reading the video sequence frame by frame to obtain all frames; setting a counter cnt1 = 1; determining whether cnt1 is equal to the total number of frames in the video sequence; if not, inputting the cnt1-th frame and the cnt1+1-th frame of the video sequence into a pre-trained multi-target tracking model to obtain an allocation matrix between the target in the cnt1-th frame and the corresponding target in the cnt1+1-th frame, where the cnt-th frame is the previous frame and the cnt+1-th frame is the next frame; obtaining all targets associated with each target in the cnt+1-th frame based on the obtained allocation matrix between the target in the cnt1-th frame and the corresponding target in the cnt1+1-th frame; and constructing the tracking trajectory of the target in the cnt-th frame together with all the obtained targets. This invention can solve the technical problem that the global correlation learning used in existing Transformer-based multi-target tracking methods not only increases computational complexity but also suffers from excessive redundant computation.

Description

Technical Field

[0001] This invention belongs to the field of pattern recognition technology, and more specifically, relates to a multi-target tracking method and system based on Transformer and graph embedding. Background Technology

[0002] Multi-object tracking (MOT) is a hot research topic in the field of computer vision and has wide applications in autonomous driving, video surveillance, intelligent robots and other fields. MOT aims to identify the location of the target of interest in a given video sequence and retrieve the motion trajectory of the same person, involving two tasks: target detection and identity authentication.

[0003] Currently, with the rise of Transformer networks, Transformer-based multi-target tracking methods have been widely applied. These methods employ sparse object queries in a Transformer-based Detection Transformer (DETR) architecture to detect new objects and initialize tracking trajectories. Tracking queries are used to maintain information about different trajectories across frames, enabling trajectory correlation between multiple frames. The common idea behind these works is to use the object query mechanism in DETR to extend existing trajectories frame-by-frame, achieving temporal migration and propagation of the target.

[0004] However, the existing Transformer-based multi-object tracking methods mentioned above have some significant drawbacks:

[0005] First, the success of the Transformer model in the field of vision is attributed to its global relevance modeling feature based on the attention mechanism, but this also leads to the inherent defects of the Transformer architecture itself. Global relevance learning not only increases computational complexity, but also involves excessive redundant computation.

[0006] Second, the Transformer integrated tracking model originates from the detection task, emphasizes modeling the correlation of scene structure over a long time and space span, the model learns freely, the learning process lacks interpretability, and there is no clear priority for learning information in the neighborhood.

[0007] Third, the Transformer integrated MOT framework indirectly solves the identity association problem between the tracking target and the detection response by using a query-based ranking method. The association output of the query target does not consider the similarity results between other query targets and the detection target. In other words, there is no clear optimal reasoning for multi-target data association during the testing phase. Therefore, the association results are often suboptimal, which leads to the distortion of target information in difficult and complex scenarios. The output results based on encoding query ranking are unreliable, which in turn leads to poor identity switching (IDS) metrics during the tracking process. Summary of the Invention

[0008] To address the aforementioned shortcomings or improvement needs of existing technologies, this invention provides a multi-target tracking method and system based on Transformer and graph embedding. The aim is to solve the technical problems of high computational complexity and excessive redundant computation in existing Transformer-based multi-target tracking methods, as well as the lack of interpretability in the learning process, the absence of clear priority for learning local neighborhood information, and poor IDS (Index Difference of Targets) performance during tracking.

[0009] To achieve the above objectives, according to one aspect of the present invention, a multi-target tracking method based on Transformer and graph embedding is provided, comprising:

[0010] (1) Obtain the video sequence by reading each frame of the video sequence to obtain all the frames;

[0011] (2) Set the counter cnt1 = 1;

[0012] (3) Determine whether cnt1 is equal to the total number of frames in the video sequence. If it is, the process ends; otherwise, proceed to step (4).

[0013] (4) Input the first frame cnt1 and the first frame cnt1+1 in the video sequence into the pre-trained multi-target tracking model to obtain the allocation matrix between the target in the first frame cnt1 and the corresponding target in the first frame cnt1+1, where the first frame cnt is the previous frame and the first frame cnt+1 is the next frame.

[0014] (5) Based on the allocation matrix between the target in the cnt1 frame and the corresponding target in the cnt1+1 frame obtained in step (4), obtain all targets associated with each target in the cnt1 frame in the cnt+1 frame, and combine the target in the cnt frame with all the obtained targets to form the tracking trajectory of the target.

[0015] (6) Set the counter cnt1 = cnt1 + 1 and return to step (3).

[0016] Preferably, the multi-target tracking model comprises a baseline depth visual feature extraction network, a Transformer encoder, a Transformer decoder, a graph convolutional network (GCN), and an association network, which are connected in sequence.

[0017] Preferably, the first layer is a baseline depth visual feature extraction network, which includes one convolutional layer, 16 building blocks and one fully connected layer. The input of the baseline depth visual feature extraction network is the previous frame and the next frame, and the output is the baseline depth visual feature with dimensions h×w×C corresponding to the previous frame and the baseline depth with dimensions h×w×C corresponding to the next frame, where h represents the height of the baseline depth visual feature, w represents the height of the baseline depth visual feature, and C represents the number of channels of the baseline depth visual feature, and C = 2048.

[0018] The second layer is the Transformer encoder, whose input is the baseline depth visual features obtained by the baseline depth visual feature extraction network from the previous and next frames. After passing through the L-layer Transformer encoding, the output is the global features with a dimension of 256 corresponding to the previous frame and the global features with a dimension of 256 corresponding to the next frame, where the value of L ranges from 3 to 10.

[0019] The third layer is the Transformer decoder. For the previous frame, its input is the object query and the global features obtained by the Transformer encoder. First, it obtains N output embeddings through a mutual attention mechanism, and then obtains N bounding box coordinates and class labels through a feedforward neural network. For the next frame, its input is the result of concatenating the object query with the N output embeddings obtained from the previous frame and the global features obtained from the second layer. First, it obtains M output embeddings through a mutual attention mechanism, and then obtains M bounding box coordinates and class labels through a feedforward neural network.

[0020] The fourth layer is a graph convolutional network. Its input is the N output embeddings of the previous frame and the M output embeddings of the next frame obtained from the third layer. First, by determining the nearest neighbor relationship, the concatenated output embeddings of the previous frame and the concatenated output embeddings of the next frame are obtained respectively. Then, the graph convolutional network iteratively processes the concatenated output embeddings of the previous frame and the concatenated output embeddings of the next frame respectively, and outputs the local correlation enhancement features of the previous frame and the local correlation enhancement features of the next frame respectively.

[0021] The fifth layer is an association network, whose input is the local correlation enhancement features of the previous frame and the local correlation enhancement features of the next frame obtained from the fourth layer. The two are processed based on the correlation between features to obtain the allocation matrix between the corresponding targets in the previous frame and the next frame.

[0022] Preferably, the multi-target tracking model is trained through the following steps:

[0023] (4-1) Obtain the MOT17 dataset and the CrowdHuman image database. Divide the obtained MOT17 dataset into the first half and the second half in equal proportions. Use the first half of the obtained MOT17 dataset and the entire CrowdHuman image database as the training set, and use the second half of the obtained MOT17 dataset as the test set.

[0024] (4-2) Set the counter cnt3 = 1;

[0025] (4-3) Set counter cnt2 = 1, and determine whether cnt3 is equal to the total number of video sequences in the training set. If yes, proceed to step (4-16); otherwise, proceed to step (4-4).

[0026] (4-4) Determine whether cnt2 is equal to the total number of frames in the cnt3 video sequence in the training set. If yes, proceed to step (4-14); otherwise, proceed to step (4-5).

[0027] (4-5) Initialize N obj N learnable object queries are used as object queries for the previous and next frames, respectively, where N obj =500;

[0028] (4-6) Take the cnt2th frame in the cnt3rd video sequence in the training set as the previous frame and the cnt2+1th frame as the next frame. Input the previous frame and the next frame into the baseline depth visual feature extraction network to obtain the baseline depth visual feature I1 with a dimension of 2048 corresponding to the previous frame and the baseline depth visual feature I2 with a dimension of 2048 corresponding to the next frame.

[0029] (4-7) For the previous frame and the next frame in the cnt3 video sequence in the training set obtained in step (4-6), the reference depth visual features I1 corresponding to the previous frame and the reference depth visual features I2 corresponding to the next frame obtained in step (4-6) are used as the query, key and value of the Transformer encoder, respectively, and input into the multi-head self-attention layer of the Transformer encoder to obtain feature vectors of dimension 256. The 256-dimensional feature vectors are normalized by the layer to obtain normalized feature vectors. The normalized features are then input into the feedforward neural network of the Transformer encoder and normalized to obtain the 512-dimensional global features F1 corresponding to the previous frame and the 512-dimensional global features F2 corresponding to the next frame.

[0030] (4-8) For the previous frame in the cnt3 video sequence in the training set obtained in step (4-6), the object query of the previous frame is input into the multi-head self-attention layer and the layer normalization layer to obtain the object query with a dimension of 256. The global feature F1 corresponding to the previous frame obtained in step (4-7) is used as the key and value of the Transformer decoder, and the 256-dimensional object query is used as the query. It is input into the multi-head mutual attention layer of the Transformer decoder and normalized to obtain N output embeddings of the previous frame. The output embedding is input into the feedforward neural network and normalized to obtain a 512-dimensional feature vector. The 512-dimensional feature vector is input into the feedforward neural network to obtain N tracking trajectories of the previous frame. Each tracking trajectory includes bounding box coordinates and class label.

[0031] (4-9) For the next frame in the cnt3 video sequence in the training set obtained in step (4-6), the tracking query and object query of the next frame are concatenated and input into the multi-head self-attention layer and the layer normalization layer to obtain a concatenated query with a dimension of 256. The concatenated query is used as the query and the global feature F2 obtained in step (4-7) is used as the key and value to be input into the multi-head mutual attention layer of the Transformer decoder and normalized to obtain M output embeddings of the next frame. The output embeddings are input into the feedforward neural network and normalized to obtain a 512-dimensional feature vector. The 512-dimensional feature vector is input into the feedforward neural network to obtain M detection targets of the next frame. Each detection target includes bounding box coordinates and class label.

[0032] (4-10) Establish graph models G1 and G2 for the N tracking trajectories obtained in step (4-8) and the M detection targets obtained in step (4-9), respectively;

[0033] (4-11) Input the graph models G1 and G2 of the previous frame and the next frame obtained in step (4-10) into the graph convolutional network respectively. By increasing the number of iterations of the graph convolutional network, the local correlation enhancement feature E1 of the previous frame and the local correlation enhancement feature E2 of the next frame are obtained.

[0034] (4-12) Input the local correlation enhancement features E1 of the previous frame and the local correlation enhancement features E2 of the next frame obtained in step (4-11) into the correlation network (e.g., Figure 8 In the process, feature-based correlation processing is performed to obtain the allocation matrix between the tracking trajectory of the previous frame and the detected target of the next frame.

[0035] (4-13) Using the N bounding box coordinates and class labels obtained in step (4-8) and the M bounding box coordinates and class labels obtained in step (4-9), train the Transformer encoder, Transformer decoder, graph convolutional network and association network. Based on the Transformer encoder, Transformer decoder and GCN as the previous and next frames in the training set obtained in step (4-6), obtain the bounding box coordinates, class labels and local correlation enhancement features of the tracking trajectory in the previous frame and the bounding box coordinates, class labels and local correlation enhancement features of the detected target in the next frame. Based on the association network, obtain the allocation matrix between the tracking trajectory in the previous frame and the detected target in the next frame. Input the obtained bounding box coordinates, class labels and allocation matrix into the defined loss function to obtain the detection loss, tracking loss and focal loss.

[0036] (4-14) cnt2 = cnt2 + 1, and return to step (4-4);

[0037] (4-15) cnt3 = cnt3 + 1, and return to step (4-3);

[0038] (4-16) Based on the detection loss, tracking loss and focal loss obtained in step (4-13), the multi-target tracking model is iteratively trained using the backpropagation method until the multi-target tracking model converges, thereby obtaining a pre-trained multi-target tracking model.

[0039] (4-17) Use the test set obtained in step (4-1) to verify the multi-target tracking model initially trained in step (4-16) until the tracking accuracy reaches the optimal level, thereby obtaining the trained multi-target tracking model.

[0040] Preferably, step (4-10) specifically involves first determining the vertex sets, where vertex set V1 = (v1, v2, ... vN) consists of all tracking trajectories in the previous frame, and vertex set V2 = (v1, v2, ... vM) consists of all detected targets in the next frame. The vertex values ​​are the output embeddings of the tracking trajectories in the previous frame and the output embeddings of the detected targets in the next frame, respectively. Then, the neighbor nodes of all vertices in vertex sets V1 and V2 are determined to form edges, and finally, graph models G1 and G2 are obtained.

[0041] Preferably, step (4-10) of determining the neighbor nodes of all vertices in V1 and V2 is as follows: First, obtain the bounding box coordinates and output embeddings of all vertices in set V1. Then, based on the bounding box coordinates determined in V1, obtain the intersection-over-union (IoU) between the bounding box coordinates of two vertices and the cosine similarity between the features of two vertices based on the output embeddings. Finally, select the weighted clustering coefficients between each vertex and all vertices except itself. The top K largest nodes are selected as its neighbors to form an edge, and... As the weight of the edges between vertices and their neighbors, the process of determining the neighbors of all vertices in V2 is the same as that in V1.

[0042] The weight of the edges between vertices is the clustering coefficient. The correlation between vertices is represented by the similarity of features and spatial distance between vertices i and j:

[0043]

[0044]

[0045] cos(·,·) represents the cosine similarity between feature vectors, IoU(·,·) represents the intersection-union ratio (IoU) of the coordinates of two target locations, and l represents the number of GCN iteration layers. b represents the vertex features of vertex i in the l-th layer GCN. i Let i represent the bounding box coordinates of vertex i, i∈[1,M], j∈[1,N].

[0046] Preferably, step (4-11) specifically involves first constructing the feature matrix of each vertex in G1 and G2 obtained in step (4-10), where each vertex has a total of Num = K+1 neighboring nodes. The feature matrix Ft is obtained by concatenating the output embeddings of vertices and their neighboring nodes, where Ft ∈ R. Num×d This is the feature matrix of the vertex, where d is the feature dimension of each node, with a dimension of 256. Then, the two-dimensional adjacency matrix Mt∈R of each vertex in G1 and G2 with all its neighboring nodes is obtained respectively. Nu×Num If the target node index is 1, then the initialization definition of the two-dimensional adjacency matrix is:

[0047]

[0048] Where i,j∈{1,2,…,Num}, Mt=1 indicates that there is an edge connecting vertex i and vertex j, and Num represents the total number of the target node and its neighboring nodes. Subsequently, each vertex in G1 and G2 is input into the graph convolutional network to obtain aggregated features. Finally, the aggregated features are input into the feedforward neural network to obtain the local correlation enhancement feature E1 of the previous frame and the local correlation enhancement feature E2 of the next frame.

[0049] Preferably, step (4-12) specifically involves first constructing a feature matrix before using the association network for data association. Since the appearance features of each target contain high-dimensional information, in order to make full use of these rich feature information, the present invention adopts a method of concatenating the local correlation enhancement feature E1 of the previous frame and the local correlation enhancement feature E2 of the next frame obtained in step (4-11) in pairs to obtain the feature matrix E.

[0050] The feature matrix E is then fed into three different 1x1 convolutional layers and compressed to MN×2C dimensions using a reshape operation, yielding Q∈R. MN×2C , K∈R MN×2C , V∈R MN×2C Where C = 256, matrix multiplication of Q and K is performed to obtain the correlation between each pair of tracked targets, and then E is obtained after a softmax operation. Q,K Then E Q,K Perform a matrix multiplication with V to obtain a further incidence matrix.

[0051] Finally, this matrix is ​​fed into the feedforward network layer and passed through two residual connections to obtain deeper correlation features between the tracking trajectory of the previous frame and the detected target in the next frame, ultimately yielding the allocation matrix A∈R. M×N .

[0052] Preferably, the steps for obtaining the detection loss, tracking loss, and focal loss in step (4-13) are as follows: First, obtain the newly detected targets in the next frame and the targets associated with the previous frame in the next frame, and then... t-1 and T t Representing the tracked trajectory in the previous frame and the detected target in the next frame, respectively, when any trajectory trk∈T t / T t-1 When, that is, corresponding to T t The new target in, but not belonging to T t-1 Then, the bounding boxes of newly detected targets in the next frame are input into the detection loss function along with their coordinates and class labels to obtain the detection loss. The bounding box coordinates and class labels of targets in the next frame that are associated with the previous frame are input into the tracking loss function to obtain the tracking loss. Subsequently, the assignment matrix between the tracking trajectory of the previous frame and the detected targets in the next frame obtained in step (4-12) is input into the focal loss function to obtain the focal loss.

[0053] The detection loss function is defined as follows:

[0054] L det =λ cls L cls +λ L1 L box +λgiou L giou

[0055] Where L cls It is based on the focal loss between the class label of the newly detected target and the ground truth class label in the training set, L box and L giou It calculates the L1 distance and generalized IoU between the normalized bounding box coordinates and the ground truth bounding box coordinates in the training set, while λ cls , λ L1 , λ giou These are the weight parameters corresponding to each part;

[0056] Tracking loss function L trk as follows:

[0057] L trk =λ cls L cls +λ L1 L box +λ giou L giou

[0058] The focal loss function is equal to:

[0059]

[0060] The focal loss function adds a modulation coefficient γ and a balance factor α to the cross-entropy loss function. The modulation coefficient γ ranges from 1 to 5, preferably 2, and the α ranges from 0.1 to 0.7, preferably 0.5. This can reduce the loss of negative sample classification and make the loss weight pay more attention to the classification of positive samples. y = 1 indicates a successful match and y = 0 indicates a no match.

[0061] According to another aspect of the present invention, a multi-target tracking system based on Transformer and graph embedding is provided, comprising:

[0062] The first module is used to acquire video sequences by reading each frame of the video sequence to obtain all the frames.

[0063] The second module is used to set the counter cnt1 = 1;

[0064] The third module is used to determine whether cnt1 is equal to the total number of frames in the video sequence. If it is, the process ends; otherwise, it proceeds to the fourth module.

[0065] The fourth module is used to input the cnt1 and cnt1+1 frames in the video sequence into a pre-trained multi-target tracking model to obtain the allocation matrix between the target in the cnt1 frame and the corresponding target in the cnt1+1 frame, where the cnt frame is the previous frame and the cnt+1 frame is the next frame.

[0066] The fifth module is used to obtain all targets associated with each target in the cnt+1 frame based on the allocation matrix between the target in the cnt1 frame and the corresponding target in the cnt1+1 frame obtained by the fourth module, and to form the tracking trajectory of the target in the cnt frame together with all the obtained targets.

[0067] The sixth module is used to set the counter cnt1 = cnt1 + 1 and return to the third module.

[0068] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0069] (1) Because this invention employs steps (4-7) to (4-11), embedding GCN into the Transformer integrated MOT architecture enhances the effectiveness of learning spatiotemporal topological information of neighboring multiple targets in the coding space. Simultaneously, it constructs an association network suitable for linear programming solutions, addressing the deficiency of the Transformer integrated architecture in lacking global reasoning for multi-target identity matching during testing. The model proposed in this invention combines the advantages of Transformer and GCN models, embedding target detection, feature representation, and data association reasoning from MOT into a unified deep network model. This greatly leverages the advantages of multi-task joint optimization learning while improving model convergence. Therefore, this invention can solve the inherent defects of existing Transformer models, where global correlation learning not only increases computational complexity but also suffers from slow convergence due to excessive redundant computation.

[0070] (2) Because this invention employs steps (4-10) to (4-11), there are specific group motion patterns among multiple targets in adjacent frames. On the one hand, the target groups that constitute close neighbors have both consistent motion patterns and fixed neighbor relationships. This social group characteristic reflects the local correlation between moving targets. As a kind of prior information, it can provide an effective basis for target detection and tracking in complex scenes. Therefore, this invention can solve the problem that the Transformer model has free learning, lacks interpretability in the learning process, and has no clear priority for learning local neighborhood information.

[0071] (3) Because this invention employs steps (4-12), inspired by the Transformer architecture's "attention is all your need" model, it proposes an association network for the multi-target tracking data association problem. This network implements a linear programming network model based on a full attention mechanism, embedding the formal neural network implementation of data association into the integrated MOT learning model. This addresses the feature representation and measurement issues applicable to data association optimization problems in autonomous learning scenarios, further enhancing the model's effectiveness. This solves the problem of distorted target information leading to suboptimal association results in difficult and complex scenarios.

[0072] (4) Since the association network in step (4-12) of this invention can also be embedded in the neural network, the powerful learning ability of deep learning can be utilized to further improve the performance of the data association model and the state reasoning model by improving the intrinsic connection between the data association model and the state reasoning model.

[0073] (5) This invention proposes a framework based on Transformer model and graph embedding, which combines the advantages of Transformer and GCN for multi-target tracking problems. Therefore, the parameters in the Transformer model can be automatically solved using the backpropagation algorithm. It can also be combined with the neural network in steps (4-11) and (4-12) for end-to-end learning to obtain more suitable parameters and further improve the performance of the entire model.

[0074] (6) The present invention has a wide range of applications. It can be used not only for pedestrian tracking, but also for tracking the trajectory of any known type of moving target. Attached Figure Description

[0075] Figure 1 This is a flowchart of the multi-target tracking method based on Transformer and graph embedding of the present invention;

[0076] Figure 2 It is the frame extracted in step (4) of the method of the present invention, wherein Figure 2 (a) is the previous frame. Figure 2 (b) is the next frame;

[0077] Figure 3 This is a schematic diagram of the Transformer encoder structure in steps (4-7) of the method of the present invention;

[0078] Figure 4 This is a schematic diagram of the Transformer decoder structure in steps (4-8) and (4-9) of the method of the present invention;

[0079] Figure 5It is the target in the frame extracted in steps (4-8) and (4-9) of the method of the present invention, wherein Figure 5 (a) is the target in the previous frame. Figure 5 (b) is the target in the next frame;

[0080] Figure 6 This is a schematic diagram of the construction diagram in step (4-10) of the method of the present invention;

[0081] Figure 7 This is a schematic diagram of the structure of GCN in step (4-11) of the method of the present invention;

[0082] Figure 8 This is a schematic diagram of the structure of the associated network in step (4-12) of the method of the present invention. Detailed Implementation

[0083] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0084] The basic idea of ​​this invention is to propose an end-to-end integrated multi-target tracking framework of "Transformer + GCN" suitable for multi-target tracking tasks. On the one hand, the GCN model is formalized into the Transformer decoder, and the vertex propagation characteristics of GCN are used to explicitly guide the modeling of local correlations between neighboring target groups, thereby improving the effectiveness of Transformer in learning the characteristics of multi-target groups in a scene. On the other hand, inspired by the "attention is all your need" model of the Transformer architecture, an association network is proposed to address the data association problem in multi-target tracking. This implements a linear programming network model based on a full attention mechanism, thereby embedding the formal neural network realization of data association into the integrated MOT learning model. This addresses the feature representation and measurement problems applicable to data association optimization in autonomous learning scenarios, further improving the effectiveness of the model.

[0085] like Figure 1 As shown, this invention provides a multi-target tracking method based on Transformer and graph embedding, comprising the following steps:

[0086] (1) Obtain the video sequence by reading each frame of the video sequence to obtain all the frames;

[0087] Specifically, this step involves obtaining a video sequence from the MOT17 dataset.

[0088] In this step, the video frame duration is 450 to 1500 frames.

[0089] (2) Set the counter cnt1 = 1;

[0090] (3) Determine whether cnt1 is equal to the total number of frames in the video sequence. If it is, the process ends; otherwise, proceed to step (4).

[0091] (4) Input the first frame (previous frame) and the next frame (cnt1+1 frame) in the video sequence into the pre-trained multi-target tracking model to obtain the allocation matrix between the target in the first frame and the corresponding target in the next frame.

[0092] For example, the allocation matrix obtained in this step is as follows, where the rows represent the cnt1-th frame and the columns represent the cnt1+1-th frame:

[0093]

[0094] As can be seen from the allocation matrix above, the first target in frame cnt1 is associated with the second target in frame cnt1+1; the second target in frame cnt1 is associated with the first target in frame cnt1+1; the third target in frame cnt1 is associated with the fourth target in frame cnt1+1; and the fourth target in frame cnt1 is associated with the third target in frame cnt1+1.

[0095] like Figure 1 As shown, the multi-target tracking model of the present invention includes a baseline depth visual feature extraction network (which is a ResNet-50 network), a Transformer encoder, a Transformer decoder, a graph convolutional network (GCN), and an association network, which are connected in sequence.

[0096] The first layer is the baseline depth visual feature extraction network, which contains one convolutional layer, 16 building block structures, and one fully connected layer. The input of the baseline depth visual feature extraction network is the previous frame and the next frame (which is an RGB image with a width W of 750, a height H of 1333, and 3 channels). The output is the baseline depth visual feature with dimensions h×w×C corresponding to the previous frame and the baseline depth with dimensions h×w×C corresponding to the next frame, where h represents the height of the baseline depth visual feature, w represents the height of the baseline depth visual feature, and C represents the number of channels of the baseline depth visual feature, and C = 2048.

[0097] The second layer is a Transformer encoder. Its input is the baseline depth visual features obtained by the baseline depth visual feature extraction network for the previous and next frames. After passing through the L-layer Transformer encoding, the output is the global features with a dimension of 256 corresponding to the previous frame and the global features with a dimension of 256 corresponding to the next frame. The value of L ranges from 3 to 10, preferably 6.

[0098] The third layer is the Transformer decoder. For the previous frame, its input consists of object queries and global features obtained from the Transformer encoder. First, it obtains N output embeddings (256 dimensions) through a mutual attention mechanism. Then, it uses a feedforward neural network to obtain N bounding box coordinates and class labels, such as... Figure 5 As shown in (a); for the next frame, its input is the result of concatenating object queries with the N output embeddings obtained from the previous frame (which serve as tracking queries) and the global features obtained from the second layer. First, through a mutual attention mechanism, M output embeddings (dimension 256) are obtained, and then M bounding box coordinates and class labels are obtained through a feedforward neural network, as shown in (a). Figure 5 As shown in (b);

[0099] The fourth layer is a graph convolutional network (such as...) Figure 7 The input to the concatenated output embeddings of the previous frame and the M output embeddings of the next frame are obtained from the third layer. First, the nearest neighbor relationship is determined to obtain the concatenated output embeddings of the previous frame and the concatenated output embeddings of the next frame. Then, the concatenated output embeddings of the previous frame and the concatenated output embeddings of the next frame are iteratively processed by the graph convolutional network to output the local correlation enhancement features of the previous frame and the local correlation enhancement features of the next frame, respectively.

[0100] The fifth layer is the interconnected network (e.g., Figure 8 The input is the local correlation enhancement features of the previous frame and the local correlation enhancement features of the next frame obtained from the fourth layer. The two are processed based on the correlation between features to obtain the allocation matrix between the corresponding targets in the previous frame and the next frame.

[0101] The multi-target tracking model of this invention is trained through the following steps:

[0102] (4-1) Obtain the MOT17 dataset and the CrowdHuman image database. Divide the obtained MOT17 dataset into the first half and the second half in equal proportions. Use the first half of the obtained MOT17 dataset and the entire CrowdHuman image database as the training set, and use the second half of the obtained MOT17 dataset as the test set.

[0103] It is important to note that in this step, the main labeled targets in the CrowdHuman image database are moving pedestrians and vehicles. The MOT17 dataset contains a total of 14 video sequences, of which 7 video sequences are the training set with labeled information, and the other 7 video sequences are the test set. The 7 test video sequences come from 7 different scenes, with different shooting angles and camera movements. The total length of the test set is 5919 frames, containing 188076 detection responses and 785 trajectories. The CrowdHuman image database is a pedestrian detection database for crowded environments, containing 15,000 images with a total of 450K targets.

[0104] Furthermore, given the relatively small size of the MOT17 dataset, in deep learning, limited training data leads to convolutional neural networks learning more one-sided features, resulting in models with poor generalization ability and a tendency to overfit. To avoid altering the features and morphology of pedestrian images, this invention expands the dataset through data augmentation. Specifically, it uses another portion from the CrowHuman image database, artificially obtaining adjacent frames by randomly scaling and shifting individual images. This effectively reduces the model's generalization error and increases its robustness.

[0105] Furthermore, in this invention, the training set is used to adjust parameters such as trainable weights and biases in the multi-object tracking model, while the test set is used to adjust hyperparameters such as the learning rate of the multi-object tracking model. The test set does not participate in the training of the model and is used to statistically test the final prediction effect of the multi-object tracking model.

[0106] The advantage of this step is that it can simulate more video scene information, thereby uncovering more potential information about the target in the video.

[0107] (4-2) Set the counter cnt3 = 1 (which is used as a pointer for the entire video sequence in the training set);

[0108] (4-3) Set counter cnt2 = 1 (which is used as a pointer to different frames in the video sequence), and determine whether cnt3 is equal to the total number of video sequences in the training set. If yes, proceed to step (4-16); otherwise, proceed to step (4-4).

[0109] (4-4) Determine whether cnt2 is equal to the total number of frames in the cnt3 video sequence in the training set. If yes, proceed to step (4-14); otherwise, proceed to step (4-5).

[0110] (4-5) Initialize N obj N learnable object queries are used as object queries for the previous and next frames, respectively, where N obj =500;

[0111] Specifically, the number of object queries is far greater than the total number of targets in a single frame.

[0112] (4-6) Take the cnt2th frame in the cnt3rd video sequence in the training set as the previous frame and the cnt2+1th frame as the next frame. Input the previous frame and the next frame into the baseline depth visual feature extraction network to obtain the baseline depth visual feature I1 with a dimension of 2048 corresponding to the previous frame and the baseline depth visual feature I2 with a dimension of 2048 corresponding to the next frame.

[0113] (4-7) For the previous frame and the next frame in the cnt3 video sequence in the training set obtained in step (4-6), the reference depth visual features I1 corresponding to the previous frame and the reference depth visual features I2 corresponding to the next frame obtained in step (4-6) are used as the query, key and value of the Transformer encoder, respectively, and input into the multi-head self-attention layer of the Transformer encoder to obtain feature vectors of dimension 256. The 256-dimensional feature vectors are then normalized to obtain normalized feature vectors. The normalized features are then input into the feedforward neural network of the Transformer encoder and normalized to obtain the 512-dimensional global features F1 corresponding to the previous frame and the 512-dimensional global features F2 corresponding to the next frame.

[0114] Furthermore, Figure 3 The structure of the Transformer encoder used in this step is shown, and the network structure is as follows: the whole consists of L Transformer layers, and each Transformer layer consists of three parts: a multi-head self-attention layer, a feedforward neural network, and a layer normalization layer.

[0115] The advantage of this step is that the Transformer encoder can acquire global information about the target throughout the entire input frame. Utilizing this global information helps the model better extract the features and location information of the target across the entire input frame.

[0116] (4-8) For the previous frame in the cnt3 video sequence in the training set obtained in step (4-6), the object query of the previous frame is input into the multi-head self-attention layer and the layer normalization layer to obtain the object query with a dimension of 256. The global feature F1 corresponding to the previous frame obtained in step (4-7) is used as the key and value of the Transformer decoder, and the 256-dimensional object query is used as the query. It is input into the multi-head mutual attention layer of the Transformer decoder and normalized to obtain N output embeddings (dimension 256) of the previous frame. The output embedding is input into the feedforward neural network and normalized to obtain a 512-dimensional feature vector. The 512-dimensional feature vector is input into the feedforward neural network to obtain N tracking trajectories of the previous frame. Each tracking trajectory includes bounding box coordinates and class label.

[0117] like Figure 5 As shown in (a), the N output embeddings are used as track queries for the next frame;

[0118] Furthermore, in the previous frame, the Transformer decoder only accepts object queries as input, and in this case, the Transformer decoder performs the same function as the detector.

[0119] Furthermore, Figure 4 This step demonstrates the use of a Transformer decoder structure, with the network structure consisting of L Transformer layers. Each Transformer layer comprises four parts: a multi-head self-attention layer, a multi-head mutual attention layer, a feedforward neural network, and a layer normalization layer. The multi-head self-attention layer, feedforward neural network, and layer normalization layer are identical to those in the Transformer encoder. The difference between the multi-head mutual attention layer and the multi-head self-attention layer lies in the source of the input features F. The input features in the Transformer decoder come from object queries or track queries, as well as the global features obtained in steps (4-7).

[0120] (4-9) For the next frame in the cnt3 video sequence obtained in step (4-6), the tracking query and object query of the next frame are concatenated and input into the multi-head self-attention layer and the layer normalization layer to obtain a 256-dimensional concatenated query. The concatenated query is used as the query, and the global feature F2 obtained in step (4-7) is used as the key and value to be input into the multi-head mutual attention layer of the Transformer decoder and normalized to obtain M output embeddings (256 dimensions) of the next frame. The output embeddings are input into the feedforward neural network and normalized to obtain a 512-dimensional feature vector. The 512-dimensional feature vector is then input into the feedforward neural network to obtain M detection targets of the next frame. Each detection target includes bounding box coordinates and class label, such as... Figure 5 As shown in (b);

[0121] Furthermore, in the next frame, the tracking trajectory will be... Figure 5 (a) Track queries serve as the basis for tracking queries, while object queries serve as the basis for labeling new targets, enabling the detection, labeling, and feature representation of new targets. Track queries carry the identity information of targets throughout the video sequence and match their constantly changing positions in an autoregressive manner. To this end, a tracking query is initialized each time a new target is generated, which comes from the output embedding of the previous frame.

[0122] The advantage of steps (4-7) to (4-9) above is that the Transformer's encoder and decoder are composed of multiple identical modules stacked together, each with the same structure and parameters. This modular design makes the Transformer highly flexible and easy to extend and modify. Modules can be freely added or removed according to task requirements, making the Transformer suitable for a variety of different processing tasks.

[0123] (4-10) Establish graph models G1 and G2 for the N tracking trajectories obtained in step (4-8) and the M detection targets obtained in step (4-9), respectively;

[0124] The process for this step is as follows: Figure 6 As shown, specifically, the vertex sets are first determined. Vertex set V1 = (v1, v2, ... vN) consists of all tracking trajectories in the previous frame, and vertex set V2 = (v1, v2, ... vM) consists of all detected targets in the next frame. The vertex values ​​(features) are the output embeddings of the tracking trajectories in the previous frame and the output embeddings of the detected targets in the next frame, respectively. Then, the neighbor nodes of all vertices in vertex sets V1 and V2 are determined to form edges, and finally, graph models G1 and G2 are obtained.

[0125] Furthermore, the process of determining the neighbor nodes of all vertices in V1 and V2 is as follows: First, obtain the bounding box coordinates and output embeddings of all vertices in set V1. Then, based on the bounding box coordinates determined in V1, obtain the Intersection over Union (IoU) between the bounding box coordinates of two vertices and the cosine similarity between the features of two vertices using the output embeddings. Finally, select the weighted clustering coefficients between each vertex and all vertices except itself. The top K largest nodes are selected as its neighbors, where K ranges from 1 to 3, preferably 2, to form an edge. As the weight of the edges between vertices and their neighbors, the process of determining the neighbor nodes of all vertices in V2 is the same as that in V1, and will not be repeated here.

[0126] Furthermore, the weight of the edges between vertices is called the aggregation weight coefficient (AWC). The correlation between vertices is represented by the similarity of features and spatial distance between vertices i and j:

[0127]

[0128]

[0129] cos(·,·) represents the cosine similarity between feature vectors, IoU(·,·) represents the intersection over union (IoU) ratio of the coordinates of two target locations, and l represents the number of GCN iteration layers. b represents the vertex features of vertex i in the l-th layer GCN. i Let i represent the bounding box coordinates of vertex i, i∈[1,M], j∈[1,N].

[0130] (4-11) Input the graph models G1 and G2 obtained in step (4-10) for the previous and next frames into a graph convolutional network (e.g., ...). Figure 7 As shown, by increasing the number of iterations of the graph convolutional network, the local correlation enhancement feature E1 of the previous frame and the local correlation enhancement feature E2 of the next frame are obtained.

[0131] Specifically, this step involves first constructing the feature matrix for each vertex in G1 and G2 obtained in step (4-10). Since each vertex (target node) is connected to its K neighboring nodes, the total number of neighboring nodes for each vertex is Num = K+1. The feature matrix Ft is obtained by concatenating the output embeddings of vertices and their neighbors, where Ft ∈ R. Num×dThis is the feature matrix of the vertex, where d is the feature dimension of each node, with a dimension of 256. Then, the two-dimensional adjacency matrix Mt∈R of each vertex in G1 and G2 with all its neighboring nodes is obtained respectively. Num×Num If the target node index is 1, then the initialization definition of the two-dimensional adjacency matrix is:

[0132]

[0133] Where i,j∈{1,2,…,Num}, Mt=1 indicates that there is an edge connecting vertex i and vertex j, and Num represents the total number of the target node and its neighboring nodes. Then, each vertex in G1 and G2 is input into the graph convolutional network to obtain aggregated features, where each vertex feature update updates the nearest neighbor relationships, as shown in the formula: That is, the correlation between vertices within the nearest neighbor is measured by their similarity. Let... Let the normalization of the adjacency matrix be denoted as , then the feature information update of a GCN model is defined as:

[0134]

[0135] When expanded, it means:

[0136]

[0137] in Let be the feature matrix corresponding to each vertex i in the l-th layer, MLP stands for Multi-Layer Perceptron, and l is the number of layers in the GCN iteration, such as... Figure 7 As shown, as the number of iterations increases, nodes in the neighborhood that contribute consistently to the target node receive positive feedback. This consistent contribution reflects the unique correlation between neighboring nodes and the target node. Finally, the aggregated features are input into the feedforward neural network to obtain the local correlation enhancement features E1 of the previous frame and the local correlation enhancement features E2 of the next frame.

[0138] The advantage of steps (4-10) to (4-11) above is that the GCN model is formalized into the Transformer decoder, and the local correlation between neighboring target groups is explicitly guided by the characteristics of graph vertex propagation, thereby improving the effectiveness of learning the characteristics of multi-target groups in the scene.

[0139] (4-12) Input the local correlation enhancement features E1 of the previous frame and the local correlation enhancement features E2 of the next frame obtained in step (4-11) into the correlation network (e.g., Figure 8 In the process, feature-based correlation processing is performed to obtain the allocation matrix between the tracking trajectory of the previous frame and the detected target of the next frame.

[0140] Specifically, firstly, before using the association network to associate data, a feature matrix needs to be constructed. Since the appearance features of each target contain high-dimensional information, in order to make full use of these rich feature information, this invention adopts a method of concatenating the local correlation enhancement feature E1 of the previous frame and the local correlation enhancement feature E2 of the next frame obtained in step (4-11) in pairs to obtain the feature matrix E.

[0141] The feature matrix E is then fed into three different 1x1 convolutional layers and compressed to MN×2C dimensions using a reshape operation, yielding Q∈R. MN×2C , K∈R MN×2C , V∈R MN×2C Where C = 256, matrix multiplication of Q and K is performed to obtain the correlation between each pair of tracked targets, and then E is obtained after a softmax operation. Q,K Then E Q,K Perform a matrix multiplication with V to obtain a further incidence matrix.

[0142] Finally, this matrix is ​​fed into the feedforward network layer and passed through two residual connections to obtain deeper correlation features between the tracking trajectory of the previous frame and the detected target in the next frame, ultimately yielding the allocation matrix A∈R. M×N .

[0143] Furthermore, if the tracking trajectory in the previous frame is [R1, R2, ..., R...] n The detection target in the next frame is [r1, r2, ..., r]. m ], where n represents the total number of tracking trajectories in the previous frame and m represents the total number of detected targets in the next frame; then the allocation matrix constructed in this step is:

[0144]

[0145] Wherein, the element P in the matrix yz This represents the association confidence between the y-th tracking trajectory in the previous frame and the z-th detected target in the next frame. In this example, y∈[1, n] and z∈[1, m].

[0146] The advantage of this step is that by implementing a linear programming network model based on a self-attention mechanism, the formal neural network realization form of data association is embedded into the integrated MOT learning model. This autonomous learning scenario is suitable for data association optimization problems and measurement problems based on feature representation, further improving the effectiveness of the model.

[0147] (4-13) Using the N bounding box coordinates and class labels obtained in step (4-8) and the M bounding box coordinates and class labels obtained in step (4-9), train the Transformer encoder, Transformer decoder, graph convolutional network and association network. Based on the Transformer encoder, Transformer decoder and GCN as the previous and next frames in the training set obtained in step (4-6), obtain the bounding box coordinates, class labels and local correlation enhancement features of the tracking trajectory in the previous frame and the bounding box coordinates, class labels and local correlation enhancement features of the detected target in the next frame. Based on the association network, obtain the allocation matrix between the tracking trajectory in the previous frame and the detected target in the next frame. Input the obtained bounding box coordinates, class labels and allocation matrix into the defined loss function to obtain the detection loss, tracking loss and focal loss.

[0148] Specifically, the steps for obtaining detection loss, tracking loss, and focal loss are as follows: First, obtain the newly detected targets in the next frame and the targets associated with the previous frame in the next frame. Then, T... t-1 and T t Representing the tracked trajectory in the previous frame and the detected target in the next frame, respectively, when any trajectory trk∈T t / T t-1 When, that is, corresponding to T t The new target in, but not belonging to T t-1 This invention considers the trajectory trk as a newly detected target in the next frame, and a detection loss is used here; when any trajectory trk∈T k ∩T k-1 When the target is shared by the previous and next frames, this invention considers the trajectory trk to be the target associated with the previous frame in the next frame. Then, the bounding boxes of newly detected targets in the next frame are input into the detection loss function along with their coordinates and class labels to obtain the detection loss. Similarly, the bounding box coordinates and class labels of targets associated with the previous frame in the next frame are input into the tracking loss function to obtain the tracking loss. Subsequently, the allocation matrix between the tracking trajectory of the previous frame and the detected target in the next frame, obtained in step (4-12), is input into the focal loss function to obtain the focal loss.

[0149] Furthermore, the detection loss function is defined as follows:

[0150] L det =λ cls L cls +λ L1 L box +λ giou L giou

[0151] Where L clsIt is based on the focal loss between the class label of the newly detected target and the ground truth class label in the training set, L box and L giou It calculates the L1 distance and generalized IoU between the normalized bounding box coordinates and the ground truth bounding box coordinates in the training set, while λ cls , λ L1 , λ giou The weight parameters corresponding to each part are 2, 2, and 3, respectively. Similarly, the tracking loss function L... trk as follows:

[0152] L trk =λ cls L cls +λ L1 L box +λ giou L giou

[0153] When the association network obtains the assignment matrix, the number of successfully matched targets is far less than the number of unmatched targets, resulting in a large number of negative samples. If the cross-entropy loss function is used, the imbalance between positive and negative samples will cause the loss function to fail to converge. Therefore, this invention uses focal loss as the loss function for calculating matching relationships in the association network. Focal loss is an improved version of the cross-entropy loss function, specifically designed to solve the problem of imbalanced positive and negative samples, and its definition is as follows:

[0154]

[0155] The focal loss function adds a modulation coefficient γ and a balance factor α to the cross-entropy loss function. The modulation coefficient γ ranges from 1 to 5, preferably 2, and the α ranges from 0.1 to 0.7, preferably 0.5. This can reduce the loss of negative sample classification and make the loss weight pay more attention to the classification of positive samples. y = 1 indicates a successful match and y = 0 indicates a no match.

[0156] Furthermore, the Transformer network training parameters were set. The model used a ResNet-50 pre-trained on ImageNet as the CNN backbone, employing the Encoder and Decoder architecture from Deformable-DETR. Fine-tuning was performed jointly on MOT17 and CrowHuan, based on a CrowHuan-based pre-trained model. This model adopted the Trackformer training strategy, directly fine-tuning on MOT17 and CrowHuan, using the AdamW optimizer. This is due to the advantages of this optimizer in terms of momentum and adaptive learning rate. The initial learning rate was 2×10⁻⁶. -4The batch size is 2; the training parameters for the graph convolutional network are set, the algorithm is trained from scratch, the number of layers in the graph convolutional network is 3, the model weights are initialized using a ResNet-50 model pre-trained on ImageNet, the optimizer is SGD, and the initial learning rate is 1×10⁻⁶. -3 And after 40 rounds, it was reduced to 1×10 -4 The training parameters for the association network were set, and the network model was trained according to the method created by DeepMOT. This method uses an assignment matrix to represent the ground truth labels of the dataset. The assignment matrix is ​​a two-dimensional matrix composed of 0s and 1s, representing the true matching results between the labeled targets. Therefore, this dataset is suitable for the association network designed in this invention to train the data association process. The initial learning rate was set to 0.005, and the learning rate decreased by a factor of 10 every 10 epochs. The optimizer was set to SGD.

[0157] The advantage of this step is that it integrates target detection, feature representation, and data association reasoning in multi-target tracking into a deep network model, which greatly leverages the advantages of multi-task joint optimization learning.

[0158] (4-14) cnt2 = cnt2 + 1, and return to step (4-4);

[0159] (4-15) cnt3 = cnt3 + 1, and return to step (4-3);

[0160] (4-16) Based on the detection loss, tracking loss and focal loss obtained in step (4-13), the multi-target tracking model is iteratively trained using the backpropagation method until the multi-target tracking model converges, thereby obtaining a pre-trained multi-target tracking model.

[0161] (4-17) Use the test set obtained in step (4-1) to verify the multi-target tracking model initially trained in step (4-16) until the tracking accuracy reaches the optimal level, thereby obtaining the trained multi-target tracking model.

[0162] (5) Based on the allocation matrix between the target in the cnt1 frame and the corresponding target in the cnt1+1 frame obtained in step (4), obtain all targets associated with each target in the cnt1 frame in the cnt+1 frame, and combine the target in the cnt frame with all the obtained targets to form the tracking trajectory of the target.

[0163] (6) Set the counter cnt1 = cnt1 + 1 and return to step (3).

[0164] In summary, this invention proposes a multi-target tracking algorithm based on Transformer and graph embedding. On one hand, the GCN model is formalized into the Transformer decoder, utilizing the vertex propagation characteristics of GCN to explicitly guide the modeling of local correlations among neighboring target groups, thereby improving the effectiveness of Transformer in learning the characteristics of multi-target groups in a scene. On the other hand, inspired by the Transformer architecture's "attention is all your need" model, an association network is proposed to address the data association problem in multi-target tracking. This implements a linear programming network model based on a full attention mechanism, embedding the formalized neural network form of data association into an integrated MOT learning model. This addresses the feature representation and measurement issues applicable to data association optimization problems in autonomous learning scenarios, further enhancing the model's effectiveness. Therefore, this invention effectively reflects the correlation of real data during multi-target tracking, resulting in high accuracy of the tracking results.

[0165] Experimental results

[0166] The actual effectiveness of this invention is illustrated here using test results on the MOT17 test set. The tracking results of the proposed multi-object tracking algorithm on the MOT17 dataset are evaluated using the following standard evaluation metrics: Multiple Object Tracking Accuracy (MOTA), Multiple Object Tracking Precision (MOTP), IDF1 Score, False Positives (FP), False Negatives (FN), Higher Order Tracking Accuracy (HOTA), Association Accuracy (AssA), Detection Accuracy (DetA), and ID Switches (IDS). "↑" indicates higher is better, and "↓" indicates lower is better. Table 1 below shows a detailed comparison of the test results of the present invention with the existing high-performance TransCenter algorithm, TransTrack algorithm and TrackFormer algorithm on the MOT17 test set.

[0167]

[0168] Table 1

[0169] As can be seen from Table 1 above: (1) The method of this invention ranks first in HOTA, and surpasses the other three algorithms in IDF1 and AssA. Among them, HOTA is the main indicator for evaluating the overall performance of the algorithm. Compared with the other three tracking algorithms, the tracking algorithm proposed in this invention has achieved the best results, which shows that the tracking algorithm proposed in this invention is superior to the other three algorithms in overall performance;

[0170] (2) Higher IDF1 and higher AssA indicate that the method proposed in this invention can significantly improve the performance of Re-ID in MOT by guiding Transformer to explicitly mine local context information of the scene by passing the advantages of neighborhood characteristics through graph model, thereby improving tracking performance.

[0171] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-target tracking method based on Transformer and graph embedding, characterized in that, Includes the following steps: (1) Obtain the video sequence by reading each frame of the video sequence to obtain all the frames; (2) Set the counter cnt1=1; (3) Determine whether cnt1 is equal to the total number of frames in the video sequence. If it is, the process ends; otherwise, proceed to step (4). (4) Input the cnt1 and cnt1+1 frames from the video sequence into the pre-trained multi-target tracking model to obtain the allocation matrix between the target in the cnt1 frame and the corresponding target in the cnt1+1 frame, where the cnt1 frame is the previous frame and the cnt1+1 frame is the next frame; the multi-target tracking model is trained through the following steps: (4-1) Obtain the MOT17 dataset and the CrowdHuman image database. Divide the obtained MOT17 dataset into the first half and the second half in equal proportions. Use the first half of the obtained MOT17 dataset and the entire CrowdHuman image database as the training set, and use the second half of the obtained MOT17 dataset as the test set. (4-2) Set the counter cnt3=1; (4-3) Set counter cnt2=1, and determine whether cnt3 is equal to the total number of frames in the video sequence in the training set. If yes, proceed to step (4-16); otherwise, proceed to step (4-4). (4-4) Determine whether cnt2 is equal to the total number of frames in the cnt3 video sequence in the training set. If yes, proceed to step (4-14); otherwise, proceed to step (4-5). (4-5) Initialization Each learnable object query serves as an object query for the previous and next frames, respectively. ; (4-6) Take the cnt2th frame in the cnt3th video sequence in the training set as the previous frame and the cnt2+1th frame as the next frame. Input the previous frame and the next frame into the baseline depth visual feature extraction network to obtain the baseline depth visual features with a dimension of 2048 corresponding to the previous frame. And the baseline depth visual features of the next frame with a dimension of 2048. ; (4-7) For the previous and next frames in the cnt3 video sequence in the training set obtained in step (4-6), the baseline depth visual features corresponding to the previous frame obtained in step (4-6) are used. and the reference depth visual features corresponding to the next frame. As the query, key, and value of the Transformer encoder, respectively, they are input into the multi-head self-attention layer of the Transformer encoder to obtain 256-dimensional feature vectors. These 256-dimensional feature vectors are then normalized to obtain normalized feature vectors. These normalized feature vectors are then input into the feedforward neural network of the Transformer encoder and normalized again to obtain 512-dimensional global features corresponding to the previous frame. And the 512-dimensional global features corresponding to the next frame. ; (4-8) For the previous frame in the cnt3 video sequence in the training set obtained in step (4-6), the object query of the previous frame is input into the multi-head self-attention layer and the layer normalization layer to obtain an object query with a dimension of 256. The global features corresponding to the previous frame obtained in step (4-7) are then used. As the keys and values ​​of the Transformer decoder, and the 256-dimensional object query as the query, they are input into the multi-head mutual attention layer of the Transformer decoder and normalized to obtain N output embeddings from the previous frame. The outputs are then embedded into the feedforward neural network and normalized to obtain a 512-dimensional feature vector. The 512-dimensional feature vector is then input into the feedforward neural network to obtain N tracking trajectories from the previous frame. Each tracking trajectory includes bounding box coordinates and class label. (4-9) For the next frame in the cnt3 video sequence obtained in the training set in step (4-6), the tracking query and object query of the next frame are concatenated and then input into the multi-head self-attention layer and the layer normalization layer to obtain a concatenated query with a dimension of 256. The concatenated query is used as the query and the global feature obtained in step (4-7). The keys and values ​​are input into the multi-head mutual attention layer of the Transformer decoder and normalized to obtain M output embeddings for the next frame. The output embeddings are then input into the feedforward neural network and normalized to obtain a 512-dimensional feature vector. The 512-dimensional feature vector is then input into the feedforward neural network to obtain M detection targets for the next frame. Each detection target includes bounding box coordinates and class label. (4-10) Establish graph models for the N tracking trajectories obtained in step (4-8) and the M detection targets obtained in step (4-9). ; (4-11) Combine the previous and next frame graph models obtained in step (4-10) The data are input into a graph convolutional network, and by increasing the number of iterations in the graph convolutional network, the local correlation enhancement features of the previous frame are obtained. Local correlation enhancement features of the next frame ; (4-12) Obtain the local correlation enhancement features of the previous frame from step (4-11). Local correlation enhancement features in the next frame The input is fed into the correlation network and subjected to feature-based correlation processing to obtain the allocation matrix between the tracking trajectory of the previous frame and the detected target of the next frame. (4-13) Using the N bounding box coordinates and class labels obtained in step (4-8) and the M bounding box coordinates and class labels obtained in step (4-9), train the Transformer encoder, Transformer decoder, graph convolutional network, and association network. Based on the Transformer encoder, Transformer decoder, and GCN, obtain the bounding box coordinates, class labels, and local correlation enhancement features of the tracking trajectory in the previous frame and the bounding box coordinates, class labels, and local correlation enhancement features of the detected target in the next frame from the previous frame and the detected target in the next frame from the association network. Obtain the allocation matrix between the tracking trajectory in the previous frame and the detected target in the next frame. Input the obtained bounding box coordinates, class labels, and allocation matrix into the defined loss function to obtain the detection loss, tracking loss, and focal loss. (4-14) cnt2 = cnt2 + 1, and return to step (4-4); (4-15) cnt3 = cnt3 + 1, and return to step (4-3); (4-16) Based on the detection loss, tracking loss and focal loss obtained in step (4-13), the multi-target tracking model is iteratively trained using the backpropagation method until the multi-target tracking model converges, thereby obtaining a pre-trained multi-target tracking model. (4-17) Use the test set obtained in step (4-1) to verify the multi-target tracking model initially trained in step (4-16) until the tracking accuracy reaches the optimal level, thereby obtaining the trained multi-target tracking model. (5) Based on the allocation matrix between the target in the cnt1 frame and the corresponding target in the cnt1+1 frame obtained in step (4), obtain all targets associated with each target in the cnt1 frame in the cnt1+1 frame, and combine the target in the cnt1 frame with all the obtained targets to form the tracking trajectory of the target; (6) Set the counter cnt1=cnt1+1 and return to step (3).

2. The multi-target tracking method based on Transformer and graph embedding according to claim 1, characterized in that, The multi-object tracking model consists of a baseline depth visual feature extraction network, a Transformer encoder, a Transformer decoder, a graph convolutional network (GCN), and an association network, connected in sequence.

3. The multi-target tracking method based on Transformer and graph embedding according to claim 2, characterized in that, The first layer is the baseline depth visual feature extraction network, which contains one convolutional layer, 16 building blocks, and one fully connected layer. The input to the baseline depth visual feature extraction network is the previous frame and the next frame, and the output is the corresponding element of the previous frame with dimensions of [missing information]. The baseline depth visual features and the corresponding dimension of the next frame. The reference depth is defined as h, w, and C, where h represents the height of the reference depth visual feature, w represents the height of the reference depth visual feature, and C represents the number of channels of the reference depth visual feature, with C=2048. The second layer is the Transformer encoder, whose input is the baseline depth visual features obtained by the baseline depth visual feature extraction network in the previous and next frames. Through the Transformer encoding of the L layer, the output is the global feature with a dimension of 256 corresponding to the previous frame and the global feature with a dimension of 256 corresponding to the next frame, where the value of L ranges from 3 to 10. The third layer is the Transformer decoder. For the previous frame, its input is the object query and the global features obtained by the Transformer encoder. First, it obtains N output embeddings through a mutual attention mechanism, and then obtains N bounding box coordinates and class labels through a feedforward neural network. For the next frame, its input is the result of concatenating the object query with the N output embeddings obtained from the previous frame and the global features obtained from the second layer. First, it obtains M output embeddings through a mutual attention mechanism, and then obtains M bounding box coordinates and class labels through a feedforward neural network. The fourth layer is a graph convolutional network. Its input is the N output embeddings of the previous frame and the M output embeddings of the next frame obtained from the third layer. First, by determining the nearest neighbor relationship, the concatenated output embeddings of the previous frame and the concatenated output embeddings of the next frame are obtained respectively. Then, the graph convolutional network iteratively processes the concatenated output embeddings of the previous frame and the concatenated output embeddings of the next frame respectively, and outputs the local correlation enhancement features of the previous frame and the local correlation enhancement features of the next frame respectively. The fifth layer is an association network, whose input is the local correlation enhancement features of the previous frame and the local correlation enhancement features of the next frame obtained from the fourth layer. The two are processed based on the correlation between features to obtain the allocation matrix between the corresponding targets in the previous frame and the next frame.

4. The multi-target tracking method based on Transformer and graph embedding according to claim 3, characterized in that, Step (4-10) specifically involves first determining the vertex set. It consists of all the tracking trajectories in the previous frame, a vertex set. It consists of all detected targets in the next frame, with vertex values ​​representing the output embedding of the tracking trajectory in the previous frame and the output embedding of the detected target in the next frame, respectively. Then, the vertex set is determined accordingly. By connecting all the neighboring nodes of each vertex to form edges, we finally obtain the graph model. .

5. The multi-target tracking method based on Transformer and graph embedding according to claim 4, characterized in that, Step (4-10) Determined The process of finding the neighbor nodes of all vertices is as follows: First, obtain... The bounding box coordinates and output embeddings for all vertices in the set are then determined based on... Given the bounding box coordinates, the Intersection over Union (IoU) between the bounding box coordinates of two vertices is obtained, and the output embedding is used to obtain the cosine similarity between the features of the two vertices. Finally, the weighted clustering coefficients between each vertex and all other vertices except itself are selected. The top K largest nodes are selected as its neighbors to form an edge, and... As the weight of the edge between a vertex and its neighboring nodes, determine The process of finding the neighbor nodes of all vertices in the equation and Same; Weighted clustering coefficient The correlation between vertices is expressed as: ， The cosine similarity between feature vectors is represented by... The intersection-union ratio (IoU) represents the ratio of the coordinates of two target locations. Indicates the number of GCN iteration layers. It means that vertex i is at the 1st dpole. Vertex features of layer GCN Let i represent the bounding box coordinates of vertex i, i∈[1,M], j∈[1,N].

6. The multi-target tracking method based on Transformer and graph embedding according to claim 5, characterized in that, Step (4-11) specifically involves first constructing the results obtained in step (4-10). and The feature matrix of each vertex, and the total number of neighboring nodes of each vertex. The feature matrix is ​​obtained by embedding the outputs of concatenated vertices and their neighboring nodes. This is the feature matrix of the vertex, where d is the feature dimension of each node, with a dimension of 256; then, we obtain... and A two-dimensional adjacency matrix of each vertex and all its neighboring nodes. If the target node index is 1, then the initialization definition of the two-dimensional adjacency matrix is: ， in, Mt =1 indicates that there is an edge connecting vertex i and vertex j, Num represents the total number of nodes between the target node and its neighbors, and then... and Each vertex in the graph is input into a graph convolutional network to obtain aggregated features. Each vertex feature update updates the nearest neighbor relationships, as shown in the formula: That is, the correlation between vertices within the nearest neighbor is measured by their similarity. Let... Let denoted as the normalization of the adjacency matrix, then the feature information update of a GCN model is defined as follows: ， When expanded, it means: ， in For the first The layer corresponds to the feature matrix of each vertex i, and MLP is a multilayer perceptron MLP. The number of layers in the GCN iteration is [number]. As the number of iterations increases, nodes in the neighborhood that consistently contribute to the target node receive positive feedback. This consistent contribution reflects the unique correlation between neighboring nodes and the target node. Finally, the aggregated features are input into the feedforward neural network to obtain the local correlation enhancement features of the previous frame. Local correlation enhancement features of the next frame .

7. The multi-target tracking method based on Transformer and graph embedding according to claim 6, characterized in that, Step (4-12) specifically involves first constructing a feature matrix before using the association network for data association. Since the appearance features of each target contain high-dimensional information, in order to fully utilize this rich feature information, the local correlation enhancement feature obtained in step (4-11) from the previous frame is used. The feature matrix is ​​obtained by pairwise concatenation. ; Then the feature matrix It is fed into three different 1x1 convolutional layers and compressed to [size missing] by a reshape operation. Dimensions, respectively obtained , , ,in Will and Perform matrix multiplication to obtain the correlation between each pair of tracked targets, and then apply the softmax operation to obtain the final result. Then and Perform a matrix multiplication to obtain a further incidence matrix; Finally, this matrix is ​​fed into the feedforward network layer and passed through two residual connections to obtain deeper correlation features between the tracking trajectory of the previous frame and the detected target in the next frame, ultimately yielding the allocation matrix. .

8. The multi-target tracking method based on Transformer and graph embedding according to claim 7, characterized in that, The process of obtaining detection loss, tracking loss, and focal loss in step (4-13) is as follows: First, obtain the newly detected targets in the next frame and the targets associated with the previous frame in the next frame. and These represent the tracking trajectory in the previous frame and the detected target in the next frame, respectively. When any trajectory... Time, that is, corresponding New goals in but not belonging to Then, the bounding boxes of newly detected targets in the next frame are input into the coordinates and class labels, which are then input into the detection loss function to obtain the detection loss; the bounding box coordinates and class labels of targets in the next frame that are associated with the previous frame are input into the tracking loss function to obtain the tracking loss; subsequently, the allocation matrix between the tracking trajectory of the previous frame and the detected targets in the next frame obtained in step (4-12) is input into the focal loss function to obtain the focal loss. The focal loss function is equal to: ， Modulation coefficient The value range is from 1 to 5. The value range is 0.1-0.

7. This indicates a successful match. This indicates no match.

9. A multi-target tracking system based on Transformer and graph embedding, implemented by the multi-target tracking method based on Transformer and graph embedding as described in claims 1 to 8, characterized in that, The multi-target tracking system includes: The first module is used to acquire video sequences by reading each frame of the video sequence to obtain all the frames. The second module is used to set the counter cnt1=1; The third module is used to determine whether cnt1 is equal to the total number of frames in the video sequence. If it is, the process ends; otherwise, it proceeds to the fourth module. The fourth module is used to input the cnt1 and cnt1+1 frames in the video sequence into a pre-trained multi-target tracking model to obtain the allocation matrix between the target in the cnt1 frame and the corresponding target in the cnt1+1 frame, where the cnt1 frame is the previous frame and the cnt1+1 frame is the next frame. The fifth module is used to obtain all targets associated with each target in the cnt1+1 frame based on the allocation matrix between the target in the cnt1 frame and the corresponding target in the cnt1+1 frame obtained by the fourth module, and to construct the tracking trajectory of the target together with all the obtained targets. The sixth module is used to set the counter cnt1 = cnt1 + 1 and return to the third module.

Images