End-to-end multi-target tracking method and device

Abstract

The present invention discloses an end-to-end multi-target tracking method and device, belonging to the field of computer vision technology. The present invention encodes the position of the feature map of the input image to obtain a position encoding vector, inputs the feature map and the position encoding vector into the encoder to obtain a comprehensive encoding vector, sets a detection query vector, inputs the detection query vector and the comprehensive encoding vector into the decoder to obtain a detection decoding vector, inputs the detection decoding vector into the detection head network to obtain a target detection result, then inputs the target detection result and the detection decoding vector into the associator to obtain a tracking decoding vector, inputs the tracking decoding vector into the associator head network to obtain an associator output result, sets a tracking sequence for the object appearing in the associator output result, and obtains a tracking result through the tracking sequence, thereby decoupling the detection task and the association task in the end-to-end multi-target tracking model, effectively reducing the conflict between the detection task and the association task, and improving the tracking efficiency.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more specifically, to an end-to-end multi-target tracking method and device. Background Art

[0002] With the remarkable development of deep learning models, computer vision plays an increasingly important role in actual production and life through a wide range of visual tasks. As an important task, multi-target tracking connects underlying tasks such as target tracking and data association, and supports cutting-edge practical scene requirements such as autonomous driving, safety monitoring, and robot navigation. Generally speaking, multi-target tracking refers to tracking objects of the category that appear in the scene given the category of the object to be tracked. The tracking results are represented by the object position and ID, and the same object that appears in different frames has the same unique ID. In actual scenes, factors such as occlusion, lighting, and shooting angle pose huge challenges to the multi-target tracking task. Therefore, designing a multi-target tracker that can cover different scenes, has excellent performance, and has a simple structure has become a major task in the development of this field.

[0003] At present, multi-target tracking models can be divided into detection-based two-stage trackers and transformer-based end-to-end trackers. The detection-based two-stage tracker first detects the objects to be tracked of the specified category through the target detection model, and then associates the detection results in different frames through the data association algorithm to assign a unique ID to the same object. Since the two tasks of target detection and data association can be developed independently, and the target detection model has made substantial progress, this type of method has dominated the field of multi-target tracking for some time. However, its disadvantage is that it requires manual setting of hyperparameters and has poor association performance. Compared with the detection-based two-stage tracker, the transformer-based end-to-end tracker can directly output the tracking results. Since this type of method does not require manual setting of tracking hyperparameters and can output tracking results simply and directly, it has begun to receive more and more attention and gradually become a new paradigm for multi-target tracking tasks. The existing transformer-based end-to-end multi-target tracking model is still based on the transformer-based detector, and tracks objects by introducing the tracking query vector. After the video is input, in the feature extraction and encoding stages, the existing transformer-based multi-target tracker still uses the same processing method as the detector, while in the decoding stage, the detection query vector and the tracking query vector are fused to achieve better association performance. However, in this framework, the query vectors representing the two types of tasks, detection and association, interact in all decoding stages, and are severely coupled, which in turn brings two disadvantages: first, the two tasks compete severely, increasing the difficulty of model optimization; second, the model must be trained from scratch, which places high demands on both data and computing power, and cannot use the existing transformer-based pre-trained detection large model. Therefore, it is urgent to design a new end-to-end multi-target tracking framework to decouple the two tasks of detection and tracking and reduce the conflict between the two tasks.

[0004] After searching, the Chinese patent application, application number 202310349047.8, published on September 1, 2023, discloses an end-to-end multi-target tracking method and system based on YOLOX. The method is based on the end-to-end multi-target tracking method of YOLOX. In view of the fact that the existing target tracking method requires the use of two steps, detection and tracking, it is proposed to implement an end-to-end target tracking method by using a target detection algorithm based on YOLOX. Only one YOLOX network is needed to implement the multi-target tracking method. The algorithm first improves the input end of the YOLOX target detection algorithm to adapt to the target tracking of the video stream; and in view of the problem of slow speed caused by the separation of detection and tracking in the existing target tracking method, the appearance features and position prediction of the target are integrated into the target detection network in the YOLOX network to implement an end-to-end target tracking method to improve the accuracy and speed of target tracking. However, the scheme can only consider the content of the previous and next two frames, and the performance in complex motion scenes that require long-term tracking cannot be guaranteed, and the scheme requires post-processing of the prediction results of the model to obtain the final tracking result. Summary of the invention

[0005] 1. Technical problems to be solved

[0006] In view of the problems existing in the prior art that the query vectors of the tracker detection task and the associated task in the end-to-end multi-target tracker have serious interactions and coupling in all decoding stages, the present invention provides an end-to-end multi-target tracking method and device, which decouples the detection task and the associated task in the tracker, reduces the conflict between the detection task and the associated task, and thus effectively improves the tracking efficiency.

[0007] 2. Technical solution

[0008] The purpose of the present invention is achieved through the following technical solutions.

[0009] An end-to-end multi-target tracking method, the steps of which include:

[0010] Acquire images and preprocess the images;

[0011] Constructing an end-to-end multi-target tracking model, the end-to-end multi-target tracking model includes a target detection module and a target association module; the target detection module includes a backbone network, an encoder, a decoder and a detection head network, and the target association module includes an associator and an association head network;

[0012] In the target detection module, the image is input into the backbone network for feature extraction to obtain a feature map, the feature map is position-encoded to obtain a position encoding vector, the feature map and the position encoding vector are serialized and converted and input into the encoder to obtain a comprehensive encoding vector, a detection query vector is set, the detection query vector and the comprehensive encoding vector are input into the decoder to obtain a detection decoding vector, and the detection decoding vector is input into the detection head network to obtain the target detection result;

[0013] In the target association module, the target detection result and the detection decoding vector are input into the associator to obtain the tracking decoding vector, and the tracking decoding vector is input into the association head network to obtain the associator output result. The tracking sequence is set according to the objects appearing in the associator output result, and the tracking result is obtained through the tracking sequence.

[0014] Furthermore, in the target association module, a tracking sequence is set according to the objects appearing in the output result of the associator, and the step of obtaining the tracking result through the tracking sequence includes:

[0015] S11: Create a new tracking sequence and set the ID number of the objects in the tracking sequence;

[0016] S12: for the first frame image in the image sequence, input the target detection result and the detection decoding vector of the first frame image into the association head network to obtain the tracking decoding vector, and input the tracking decoding vector into the association head network to obtain the associator output result, wherein the associator output result includes the object category, the classification confidence and the position information;

[0017] Set the classification confidence threshold α. If the classification confidence of the associator output result corresponding to the detection decoding vector of the first frame image is not less than the classification confidence threshold α, then mark the associator output result as a newly appeared object, add the object to the tracking sequence and assign an ID number, and record the tracking decoding vector of the object in the tracking sequence as the tracking query vector for tracking the next frame image;

[0018] S13: for the second frame image in the image sequence, the target detection result, the detection decoding vector, the comprehensive encoding vector, the tracking query vectors and the position information of all objects in the tracking sequence of the second frame image are input into the associator to obtain the tracking decoding vector, and the tracking decoding vector is input into the association head network to obtain the associator output result of the second frame image;

[0019] If the classification confidence of the associator output result corresponding to the detection decoding vector of the second frame image is not less than the classification confidence threshold α, the associator output result is marked as a newly appeared object, the object is added to the tracking sequence and assigned an ID number, and the tracking decoding vector of the object in the tracking sequence is recorded as the tracking query vector for tracking the next frame image;

[0020] If the classification confidence of the associator output result corresponding to the tracking query vector of the second frame image is lower than the classification confidence threshold α, in the training phase, the object corresponding to the associator output result is removed from the tracking sequence. In the inference phase, the object corresponding to the associator output result is marked as lost. For the object corresponding to the associator output result that is lost for more than L frames in a row, it is removed from the tracking sequence.

[0021] S14: For other frame images in the image sequence, repeat step S13 to obtain a final tracking sequence, which is the tracking result.

[0022] Furthermore, the associator includes a content interaction module, a position update module and a decoding layer network, and the association head network includes a classification subnetwork and a regression subnetwork, and the classification subnetwork and the regression subnetwork are both composed of multiple linear mapping layers, nonlinear mapping layers and standardized operation modules.

[0023] Furthermore, in the target association module, a classification confidence threshold β is set, and the detection decoding vector is screened by the classification confidence threshold β, and the detection decoding vector whose classification confidence is not less than the classification confidence threshold β is recorded as a valid detection decoding vector, and the detection decoding vector whose classification confidence is less than the classification confidence threshold β is recorded as an invalid detection decoding vector;

[0024] In the content interaction module, the effective detection decoding vector is used to update the tracking query vector, the attention weights of the effective detection decoding vector and the tracking query vector are calculated, and the tracking query vector is updated using the effective detection decoding vector according to the attention weights;

[0025] In the position update module, the position information corresponding to the effective detection decoding vector and the position information corresponding to the tracking query vector are matched using the optimal bipartite graph matching strategy, and the position information of the tracking query vector is updated using the position information of the effective detection decoding vector according to the matching result;

[0026] The updated tracking query vector, the unmatched valid detection decoding vector, the invalid detection decoding vector and their corresponding position information and the comprehensive coding vector are input into the decoding layer network to obtain the tracking decoding vector.

[0027] Furthermore, the tracking decode vector generated by the detection decoding vector is respectively input into the classification subnetwork and regression subnetwork of the association head network to obtain the classification result and regression result, and the detection loss function is calculated; the tracking decode vector generated by the tracking query vector is respectively input into the classification subnetwork and regression subnetwork of the association head network to obtain the classification result and regression result, and the tracking loss function is calculated.

[0028] Furthermore, in the training phase, for newly appeared objects, the local optimal bipartite graph matching strategy is used to calculate the detection loss function and train the end-to-end multi-target tracking model. The calculation formula is:

[0029]

[0030] in, represents the detection loss function, y new Represents the true value label of the newly appeared object, y new =(c new ,b new ), c new Indicates the true category information of the newly appeared object, b new Represents the true position information of the newly appeared object, represents the predicted value of the detection decoding vector, Indicates that the predicted category is c new The probability value of Represents the predicted position information of the newly appeared object, i represents a natural number, M represents the number of detection decoding vectors, represents the classification loss function in the detection task, represents the regression loss function in the detection task, Indicates that the calculation scope of the loss function is the positive sample object, Indicates the number of steps from the i-th true value to the i-th A map of predicted values.

[0031] Furthermore, in the training phase, for objects that have appeared in the tracking sequence, the tracking loss function is calculated by the <true value label-prediction> matching relationship when the object first appears. The calculation formula is:

[0032]

[0033] in, represents the tracking loss function, N represents the number of predictions for the tracking query vector, and y old Indicates the true value label of the object that has appeared, y old =(c old ,b old ), c old Indicates the true value category information of the object that has appeared, b old Indicates the true position information of the object that has appeared, represents the predicted value of the tracking query vector, Indicates that the predicted category is c old The probability value of Indicates the predicted position information of the object that has appeared. represents the classification loss function in the tracking task, represents the regression loss function in the tracking task, Indicates the number of steps from the i-th true value to the i-th true value in the tracking task. A map of predicted values.

[0034] An end-to-end multi-target tracking device, comprising:

[0035] Input module, obtains images and preprocesses them;

[0036] A construction module is used to construct an end-to-end multi-target tracking model, wherein the end-to-end multi-target tracking model includes a target detection module and a target association module; the target detection module includes a backbone network, an encoder, a decoder and a detection head network, and the target association module includes an associator and an association head network;

[0037] Detection module: In the target detection module, the image is input into the backbone network for feature extraction to obtain a feature map, the feature map is position-encoded to obtain a position encoding vector, the feature map and the position encoding vector are serialized and converted and input into the encoder to obtain a comprehensive encoding vector, a detection query vector is set, the detection query vector and the comprehensive encoding vector are input into the decoder to obtain a detection decoding vector, and the detection decoding vector is input into the detection head network to obtain the target detection result;

[0038] Tracking module, in the target association module, the target detection result and the detection decoding vector are input into the associator to obtain the tracking decoding vector, the tracking decoding vector is input into the association head network to obtain the associator output result, the tracking sequence is set according to the object appearing in the associator output result, and the tracking result is obtained through the tracking sequence.

[0039] A computer device comprises a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the above-mentioned method when executing the computer program.

[0040] A computer-readable storage medium stores a computer program, and the computer program executes the above method when executed by a processor.

[0041] 3. Beneficial effects

[0042] Compared with the prior art, the advantages of the present invention are:

[0043] An end-to-end multi-target tracking method and device of the present invention, based on the end-to-end characteristics of the multi-target tracker, decouples the detection task and the associated task in tracking, thereby reducing the conflict between the detection task and the associated task, and reducing the difficulty of optimizing the end-to-end multi-target tracking model, making the end-to-end multi-target tracking model more open, and only needs to train the associator module, thereby significantly reducing training parameters, reducing the demand for data and computing power, reducing scientific research costs, and improving scientific research efficiency. It has strong practicality and wide applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flow chart of a method according to an embodiment of the present invention;

[0045] Figure 2 It is an overall framework diagram of an embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of the structure of an encoder according to an embodiment of the present invention;

[0047] Figure 4 A schematic diagram of the structure of a decoder according to an embodiment of the present invention;

[0048] Figure 5 Schematic diagram of the structure of the associator according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0050] Example

[0051] like Figure 1As shown, an end-to-end multi-target tracking method provided by this embodiment. Specifically in this embodiment, first, an image is acquired and preprocessed. In this embodiment, image preprocessing includes preprocessing the image in the training phase and preprocessing the image in the test phase. Specifically, in the training phase, image preprocessing includes data augmentation of the acquired image to avoid overfitting of the data, indirectly increase the amount of training data, and thus improve the training accuracy of the end-to-end multi-target tracking model provided by this embodiment. In the test phase, image preprocessing includes operations such as resizing the image and standardizing the image pixel values, so as to ensure the prediction accuracy of the end-to-end multi-target tracking model provided by this embodiment. Then, after image preprocessing, image I is obtained, and the size of image I is W×H×3, where W represents the width of the preprocessed image I, H represents the height of the preprocessed image I, and 3 represents the three channels of RGB. In this embodiment, the height H of image I is set to 800 pixels, and the width W of image I is set proportionally according to the aspect ratio of image I. If the width W of image I after adjustment exceeds 1500 pixels, the width W of image I is set to 1500 pixels, and the height H of image I is set according to the aspect ratio.

[0052] Furthermore, an end-to-end multi-target tracking model is constructed. The end-to-end multi-target tracking model constructed in this embodiment includes a target detection module and a target association module.

[0053] like Figure 2 As shown, in this embodiment, the target detection module includes a backbone network, an encoder, a decoder, and a detection head network. For the backbone network, in this embodiment, the backbone network can select an existing feature extraction basic network, such as a residual series network (ResNet18, ResNet50, ResNet101, ...), DenseNet, and Swin Transformer networks. In order to improve the prediction speed and prediction accuracy of the end-to-end multi-target tracking model, ResNet50 is preferably selected as the backbone network in this embodiment.

[0054] For encoders, such as Figure 3 As shown, the encoder is composed of one or more encoding layer networks, and the encoding layer network includes an encoding layer multi-head self-attention module, a forward mapping network, and a standardization processing module. In this embodiment, the encoder is composed of six encoding layer networks, and the encoding layer network includes an encoding layer multi-head self-attention module, a forward mapping network, and a standardization processing module. Specifically, each encoding layer network includes an encoding layer multi-head self-attention module, a forward mapping network, and a standardization processing module. In particular, in this embodiment, the encoding layer multi-head self-attention module is expressed as:

[0055]

[0056] Among them, Q e represents the query vector of the multi-head self-attention module in the encoding layer, K e represents the key vector of the multi-head self-attention module in the encoding layer, V e Represents the value vector of the multi-head self-attention module in the coding layer. It is worth noting that, in this embodiment, i is set to be a natural number, then in the i-th coding layer network, V e =W ev EE i-1 , where W eq , W ek , W ev Represents different mapping matrices in the coding layer network, PE represents position encoding information, represents the i-1th encoding layer output vector with position information, d k represents the dimension of the key vector of the multi-head self-attention module in the encoding layer, softmax represents attention normalization, and T represents the transpose operation.

[0057] In this embodiment, the forward mapping network is composed of a fully connected network, a nonlinear mapping layer, and a random dropout layer connected in sequence. The standardization processing module refers to mapping the image pixel values ​​to a unified space, for example, mapping the image pixel values ​​to a standardized space with 0 as the mean and 1 as the variance through standardization processing.

[0058] For the decoder, such as Figure 4 As shown, the decoder is composed of one or more decoding layer networks, and the decoding layer network includes a decoding layer multi-head self-attention module, a multi-head cross-attention module, a forward mapping network, and a standardization processing module. In this embodiment, the decoder is composed of six decoding layer networks, and the decoding layer network includes a decoding layer multi-head self-attention module, a multi-head cross-attention module, a forward mapping network, and a standardization processing module. Specifically, in this embodiment, each decoding layer network includes a decoding layer multi-head self-attention module, a multi-head cross-attention module, a forward mapping network, and a standardization processing module. In this embodiment, the decoding layer network is a decoding layer network in the detector DAB-DETR, and thus the decoding layer multi-head self-attention module is expressed as:

[0059]

[0060] Among them, Q d represents the query vector of the multi-head self-attention module in the decoding layer, K d represents the key vector of the multi-head self-attention module in the decoding layer, V d Represents the value vector of the multi-head self-attention module in the decoding layer. It is worth noting that in this embodiment, in the i-th decoding layer network, V d =W dv ·DE i-1 , where W dq , W dk , W dv Represents different mapping matrices in the decoding layer network, DE i-1 is the output vector of the i-1th decoding layer network, D query represents the detection query vector, represents the i-1th decoding layer output vector combined with the detection query vector, d k Represents the dimension of the key vector of the multi-head self-attention module in the decoding layer.

[0061] It should be noted that, in this embodiment, the forward mapping network and the standardization processing module in the decoding layer network are the same as the forward mapping network and the standardization processing module in the encoding layer network.

[0062] For the detection head network, in this embodiment, the detection head network includes a classification subnetwork and a regression subnetwork, and the classification subnetwork and the regression subnetwork are both composed of multiple linear mapping layers, nonlinear mapping layers and standardized operation modules.

[0063] Furthermore, the target association module includes an associator and an association head network. For the associator, Figure 5 As shown, in this embodiment, the associator includes a content interaction module, a position update module and a decoding layer network. The interaction module updates the detection query vector content, and the position update module updates the sampling position of the detection query vector through distance matching. For the association head network, in this embodiment, the association head network includes a classification subnetwork and a regression subnetwork, and the classification subnetwork and the regression subnetwork are both composed of multiple linear mapping layers, nonlinear mapping layers and standardized operation modules.

[0064] Therefore, in the target detection module, the image is input into the backbone network for feature extraction to obtain a feature map, the feature map is position-encoded to obtain a position encoding vector, the feature map and the position encoding vector are serialized and converted and input into the encoder to obtain a comprehensive encoding vector, a detection query vector is set, the detection query vector and the comprehensive encoding vector are input into the decoder to obtain a detection decoding vector, and the detection decoding vector is input into the detection head network to obtain the target detection result.

[0065] Specifically, the image I is input into the backbone network, and feature extraction is performed through two-dimensional convolution, nonlinear mapping, and pooling operations to obtain the feature map feat of the image I in the high-dimensional space. In this embodiment, the size of the feature map feat is W0×H0×c, where W0 represents the width of the feature map feat, W0=W / s, s represents the downsampling rate of the backbone network, H0 represents the height of the feature map feat, H0=H / s, and c represents the number of channels of the feature map feat. It should be noted that in this embodiment, the two-dimensional convolution, nonlinear mapping, and pooling operations used are prior art. In order to enable the encoder and decoder to have position recognition capabilities, it is necessary to position encode the feature map feat extracted by the backbone network, that is, to establish a pixel coordinate system, and map the horizontal coordinate (from 0 to W0) and the vertical coordinate (from 0 to H0) of each feature point in the feature map feat to obtain a position encoding vector PE(feat). The size of the position encoding vector PE(feat) is the same as the size of the feature map vector, both of which are W0×H0×c. The calculation formula for mapping the horizontal and vertical coordinates of each feature point in the feature map feat to obtain the position encoding vector is:

[0066]

[0067]

[0068]

[0069]

[0070] Among them, pos represents the feature point, x represents the horizontal coordinate, y represents the vertical coordinate, pos x Indicates the horizontal coordinate of the feature point, pos y represents the ordinate of the feature point, d represents the longitudinal dimension of each feature point after encoding. In this embodiment, d=2 / c, i represents the index of the position encoding in the longitudinal dimension. In this embodiment, the range of i is [0, d / 2]. Represents the sinusoidal encoding vector of the horizontal coordinate of the feature point, Represents the cosine encoding vector of the horizontal coordinate of the feature point, The sinusoidal encoding vector representing the ordinate of the feature point, The cosine encoding vector representing the ordinate of the feature point.

[0071] Furthermore, the feature map feat and the position encoding vector PE(feat) are input into the encoder for serialization conversion to obtain a comprehensive encoding vector (Encoded Embedding, EE). It should be noted that, in this embodiment, the feature map feat and the position encoding vector PE(feat) are input into the encoder for serialization conversion to obtain the comprehensive encoding vector EE, which is a prior art. In this embodiment, N groups of learnable parameters are set as detection query vectors (Detection query, Dquery), and the comprehensive encoding vector EE and the detection query vector Dquery are input into the decoder to obtain N detection decoding vectors (DecodedEmbedding, DE), and then the detection decoding vector DE is input into the detection head network to obtain the target detection result. In this embodiment, the target detection result includes object category, classification confidence and location information.

[0072] In the target association module, the target detection result and the detection decoding vector are input into the associator to obtain the tracking decoding vector (Track Embedding, TE), and the tracking decoding vector TE is input into the association head network to obtain the associator output result. The tracking sequence is set according to the object appearing in the associator output result, and the tracking result is obtained through the tracking sequence. Specifically, in the target association module, the tracking sequence is set according to the object appearing in the associator output result, and the step of obtaining the tracking result through the tracking sequence includes:

[0073] S11: Create a new tracking sequence and set the ID number of the objects in the tracking sequence;

[0074] S12: for the first frame image in the image, input the target detection result of the first frame image and the detection decoding vector DE into the associator to obtain the tracking decoding vector TE, and input the tracking decoding vector TE into the detection head network to obtain the associator output result, and the associator output result includes the object category, classification confidence and location information;

[0075] A classification confidence threshold α is set. If the classification confidence of the associator output result corresponding to the detection decoding vector DE of the first frame image is not lower than the classification confidence threshold α, the associator output result is marked as a newly appeared object, the object is added to the tracking sequence and assigned an ID number, and the tracking decoding vector TE of the object in the tracking sequence is recorded as a tracking query vector (Track Query, Tquery) for tracking the next frame image. In this embodiment, the classification confidence threshold α is set to 0.5.

[0076] S13: For the second frame image in the image, the target detection result of the second frame image, the detection decoding vector DE, the integrated encoding vector EE and the tracking query vector Tquery and position information of all objects in the tracking sequence are input into the associator to obtain the tracking decoding vector TE, and the tracking decoding vector TE is input into the association head network to obtain the associator output result of the second frame image.

[0077] If the classification confidence of the associator output result corresponding to the detection decoding vector DE of the second frame image is not less than the classification confidence threshold α, the associator output result is marked as a newly appeared object, the object is added to the tracking sequence and assigned an ID number, and the tracking decoding vector TE of the object in the tracking sequence is recorded as the tracking query vector Tquery, which is used for tracking the next frame image;

[0078] If the classification confidence of the associator output result corresponding to the tracking query vector Tquery of the second frame image is less than the classification confidence threshold α, in the training stage, the object corresponding to the associator output result is removed from the tracking sequence, and in the inference stage, the object corresponding to the associator output result is marked as lost. For the object corresponding to the associator output result that is continuously lost for more than L frames, it is removed from the tracking sequence. In this embodiment, L is set to 20 frames;

[0079] S14: Repeat step S13 for other frame images in the image to obtain a final tracking sequence, which is the tracking result.

[0080] It is worth noting that, in this embodiment, the detection decoding vector DE is input into the classification sub-network to obtain the classification result, and the detection decoding vector DE is input into the regression sub-network to obtain the regression result. The classification result includes the category information and the corresponding confidence, and the regression result includes the minimum rectangular frame coordinates of the detected object. In this embodiment, the minimum rectangular frame coordinates are (c x , c y , w, h) or (x1, y1, x2, y2), where c x , c y They represent the horizontal and vertical coordinates of the center point of the minimum rectangular box, w and h represent the width and height of the minimum rectangular box, x1 and y1 represent the horizontal and vertical coordinates of the upper left corner of the minimum rectangular box, and x2 and y2 represent the horizontal and vertical coordinates of the lower right corner of the minimum rectangular box.

[0081] Furthermore, in the target association module, a classification confidence threshold β is set, and the classification results and regression results of the detection decoding vector DE are screened by the classification confidence threshold β, and the detection decoding vector DE with a classification confidence not less than the classification confidence threshold β is recorded as a valid detection decoding vector DE c, the detection decoding vector DE whose classification confidence is lower than the classification confidence threshold β is recorded as an invalid detection decoding vector DE uc In this embodiment, the classification confidence threshold β is set to 0.3.

[0082] In the content interaction module, the effective detection decoding vector DE is used c Update the tracking query vector Tquery and calculate the effective detection decoding vector DE c And the attention weight of the tracking query vector Tquery, using the effective detection decoding DE according to the attention weight c The vector updates the tracking query vector Tquery.

[0083] In the position update module, the optimal bipartite graph matching strategy is used to effectively detect the decoding vector DE c The corresponding position information is matched with the position information corresponding to the tracking query vector Tquery, and the effective detection decoding vector DE is used according to the matching result. c The location information of the tracking query vector Tquery is updated.

[0084] Then, the updated tracking query vector Tquery and the unmatched valid detection decoding vector DE c , invalid detection decoding vector DE uc The corresponding position information and the integrated coding vector EE are input into the decoding layer network to obtain the tracking decoding vector TE. Specifically, in this embodiment, starting from the second frame image of each video (the object is detected in the previous frame image), the tracking query vector Tquery and the effective detection decoding vector DE of the current frame are input into the decoding layer network to obtain the tracking decoding vector TE. c The tracking query vector Tquery and the effective detection decoding vector DE entering the associator are input together with their position information to obtain the tracking decoding vector TE, and then the tracking decoding vector TE is input into the classification sub-network of the association head network to obtain the classification result, and input into the regression sub-network to obtain the regression result. c To interact, the interaction formula is:

[0085]

[0086] Among them, T query represents the tracking query vector, represents a tracking query vector containing location information, DE c represents the effective detection decoding vector, represents a valid detection decoding vector containing position information, T ref Represents the location information corresponding to the tracking query vector, Dref Indicates the position information corresponding to the effective detection decoding vector, d k Denotes the effective detection decoding vector DE c The dimension of , T represents the transpose operation.

[0087] The location update module calculates the location information T corresponding to the tracking query vector Tquery ref and effective detection decoding vector DE c Corresponding location information D ref The mutual distance between them matches the T corresponding to the same object. ref and D ref , update the sampling point coordinates of the object at the next moment according to the matching results. Specifically, calculate the position information T corresponding to the tracking query vector Tquery ref and the position information D corresponding to the effective detection decoding vector ref The calculation formula of the mutual distance between them is:

[0088] D = Dist(T ref ,D ref )

[0089] Where D represents the location information T corresponding to the tracking query vector Tquery ref and effective detection decoding vector DE c Corresponding location information D ref The mutual distance between them, Dist represents the distance measurement algorithm, and GIOU is used as the distance measurement in this embodiment.

[0090] According to the location information T corresponding to the tracking query vector Tquery ref and effective detection decoding vector DE c Corresponding location information D ref The mutual distance D between them is used to perform optimal bipartite graph matching. In this embodiment, the Hungarian algorithm is used to achieve matching, and the matching formula is:

[0091] ω=linear_assign(T ref ,D ref )

[0092] Among them, ω represents the matching result, and linear_assign represents the Hungarian matching algorithm.

[0093] By matching the valid detection decoding vector DE c Corresponding location information D ref Track the location information T corresponding to the query vector Tquery ref Update, the update formula is:

[0094]

[0095] in, represents the location information corresponding to the i-th tracking query vector, Indicates that the number obtained after the optimal bipartite graph matching is ω i The position information corresponding to the valid detection query vector, F represents an update operation. In this embodiment,

[0096] Then, delete the matching location information and the corresponding detection decoding vector DE, the interactive tracking query vector Tquery, the unmatched valid detection decoding vector DE c and invalid detection decoding vector DE uc and its position information are input into the associator to obtain the tracking decoding vector TE, and the tracking decoding vector TE is respectively input into the classification subnetwork and regression subnetwork of the association head network to obtain the classification result and regression result in turn. In this embodiment, the tracking decoding vector TE generated by the detection decoding vector DE is respectively input into the classification subnetwork and regression subnetwork of the association head network to obtain the classification result and regression result, and the detection loss function is calculated. The tracking decoding vector TE generated by the tracking query vector Tquery is respectively input into the classification subnetwork and regression subnetwork of the association head network to obtain the classification result and regression result, and the tracking loss function is calculated. In the training stage, for newly appeared objects, the local optimal bipartite graph matching strategy is used to calculate the detection loss function, and the end-to-end multi-target tracking model is trained. The calculation formula is:

[0097]

[0098] in, represents the detection loss function, y new Represents the true value label of the newly appeared object, y new =(c new ,b new ), c new Indicates the true category information of the newly appeared object, b new Represents the true position information of the newly appeared object, represents the predicted value of the detection decoding vector, Indicates that the predicted category is c new The probability value of represents the predicted position information of the newly appeared object, i represents a natural number, and M represents the number of detection decoding vectors. In this embodiment, M=300 is taken, and each detection decoding vector corresponds to a prediction value. represents the classification loss function in the detection task, represents the regression loss function in the detection task, Indicates that the calculation scope of the loss function is the positive sample object, that is, the object whose label is the foreground category. Indicates the number of steps from the i-th true value to the i-th The mapping of the predicted values ​​is obtained by matching the predicted values ​​corresponding to all detection decoding vectors with the true value labels of all newly appeared objects by the optimal bipartite graph matching strategy. If the number of newly appeared objects is less than M, the background categories are used to fill them so that the number of true value labels is equal to M.

[0099] In the training phase, for objects that have appeared in the tracking sequence, the tracking loss function is calculated by the <true value label-prediction> matching relationship when the object first appears. The calculation formula is:

[0100]

[0101] in, represents the tracking loss function, N represents the number of predictions of the tracking query vector, N is equal to the number of objects in the tracking sequence, y old Indicates the true value label of the object that has appeared, y old =(c old ,b old ), c old Indicates the true value category information of the object that has appeared, b old Indicates the true position information of the object that has appeared, represents the predicted value of the tracking query vector, Indicates that the predicted category is c old The probability value of Indicates the predicted position information of the object that has appeared. represents the classification loss function in the tracking task, represents the regression loss function in the tracking task, Indicates the number of steps from the i-th true value to the i-th true value in the tracking task. In this embodiment, the mapping of each object in the tracking sequence That is, the <true value label-prediction> matching relationship does not change over time until the object is removed from the tracking sequence.

[0102] Therefore, the end-to-end multi-target tracking method provided in this embodiment decouples the detection task and the associated task in the tracking based on the end-to-end characteristics of the multi-target tracker, thereby reducing the conflict between the detection task and the associated task, and reducing the difficulty of optimizing the end-to-end multi-target tracking model, making the end-to-end multi-target tracking model more open, and only needs to train the associator module, significantly reducing the training parameters, reducing the demand for data and computing power, reducing scientific research costs, and improving scientific research efficiency. It has strong practicality and wide applicability.

[0103] This embodiment also provides an end-to-end multi-target tracking device, which includes an input module, a construction module, a detection module and a tracking module. The input module is used to acquire an image and pre-process the image. The construction module is used to construct an end-to-end multi-target tracking model, which includes a target detection module and a target association module. The target detection module includes a backbone network, an encoder, a decoder and a detection head network, and the target association module includes an associator and an association head network. In the detection module, the image is input into the backbone network for feature extraction to obtain a feature map, the feature map is position-encoded to obtain a position encoding vector, the feature map and the position encoding vector are serialized and converted and input into the encoder to obtain a comprehensive encoding vector, a detection query vector is set, the detection query vector and the comprehensive encoding vector are input into the decoder to obtain a detection decoding vector, and the detection decoding vector is input into the detection head network to obtain a target detection result. Tracking module, in the target association module, the target detection result and the detection decoding vector are input into the associator to obtain the tracking decoding vector, the tracking decoding vector is input into the association head network to obtain the associator output result, the object appearing in the associator output result is set to a tracking sequence, and the tracking result is obtained through the tracking sequence. The end-to-end multi-target tracking device provided in this embodiment can implement any method of the end-to-end multi-target tracking method, and the specific working process of an end-to-end multi-target tracking device can refer to the corresponding process in the embodiment of the end-to-end multi-target tracking method. The method and device provided in this embodiment can be implemented in other ways. For example, the device embodiment described above is only schematic; for example, the division of a module is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the connection or communication connection between each other shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0104] This embodiment also provides a computer device. A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the end-to-end multi-target tracking method when executing the computer program.

[0105] This embodiment also provides a computer-readable storage medium. A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, an end-to-end multi-target tracking method described in this embodiment is executed. Among them, the computer-readable storage medium can be any tangible medium containing or storing a program, and the program can be used by or in combination with an instruction execution system, device or device; the program code contained on the computer-readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0106] The above schematically describes the invention and its implementation methods, which is not restrictive. Without departing from the spirit or basic features of the invention, the invention can be implemented in other specific forms. What is shown in the accompanying drawings is only one of the implementation methods of the invention. The actual structure is not limited thereto, and any figure mark in the claims should not limit the claims involved. Therefore, if a person of ordinary skill in the art is inspired by it, without departing from the purpose of the invention, a structural method and an embodiment similar to the technical solution are designed without creativity, which should all belong to the protection scope of the present invention. In addition, the word "including" does not exclude other elements or steps, and the word "one" before the element does not exclude the inclusion of "multiple" elements. The multiple elements stated in the product claim can also be implemented by one element through software or hardware. The words first, second, etc. are used to indicate the name, and do not indicate any specific order.

Claims

1. An end-to-end multi-target tracking method, the steps of which include: Acquire images and preprocess the images; Constructing an end-to-end multi-target tracking model, wherein the end-to-end multi-target tracking model includes a target detection module and a target association module; The target detection module includes a backbone network, an encoder, a decoder and a detection head network, the target association module includes an associator and an association head network; the associator includes a content interaction module, a position update module and a decoding layer network, the association head network includes a classification subnetwork and a regression subnetwork, and the classification subnetwork and the regression subnetwork are both composed of multiple linear mapping layers, nonlinear mapping layers and standardized operation modules; In the target detection module, the image is input into the backbone network for feature extraction to obtain a feature map, the feature map is position-encoded to obtain a position encoding vector, the feature map and the position encoding vector are serialized and converted and input into the encoder to obtain a comprehensive encoding vector, a detection query vector is set, the detection query vector and the comprehensive encoding vector are input into the decoder to obtain a detection decoding vector, and the detection decoding vector is input into the detection head network to obtain the target detection result; In the target association module, the target detection result and the detection decoding vector are input into the associator to obtain the tracking decoding vector, and the tracking decoding vector is input into the association head network to obtain the associator output result. The tracking sequence is set according to the object appearing in the associator output result, and the tracking result is obtained through the tracking sequence. The steps to obtain tracking results through tracking sequence include: S11: Create a new tracking sequence and set the ID number of the objects in the tracking sequence; S12: for the first frame image in the image sequence, input the target detection result and the detection decoding vector of the first frame image into the association head network to obtain the tracking decoding vector, and input the tracking decoding vector into the association head network to obtain the associator output result, wherein the associator output result includes the object category, the classification confidence and the position information; Set the classification confidence threshold α. If the classification confidence of the associator output result corresponding to the detection decoding vector of the first frame image is not less than the classification confidence threshold α, then mark the associator output result as a newly appeared object, add the object to the tracking sequence and assign an ID number, and record the tracking decoding vector of the object in the tracking sequence as the tracking query vector for tracking the next frame image; S13: for the second frame image in the image sequence, the target detection result, the detection decoding vector, the comprehensive encoding vector, the tracking query vectors and the position information of all objects in the tracking sequence of the second frame image are input into the associator to obtain the tracking decoding vector, and the tracking decoding vector is input into the association head network to obtain the associator output result of the second frame image; If the classification confidence of the associator output result corresponding to the detection decoding vector of the second frame image is not less than the classification confidence threshold α, the associator output result is marked as a newly appeared object, the object is added to the tracking sequence and assigned an ID number, and the tracking decoding vector of the object in the tracking sequence is recorded as the tracking query vector for tracking the next frame image; If the classification confidence of the associator output result corresponding to the tracking query vector of the second frame image is lower than the classification confidence threshold α, in the training phase, the object corresponding to the associator output result is removed from the tracking sequence. In the inference phase, the object corresponding to the associator output result is marked as lost. For the object corresponding to the associator output result that is lost for more than L frames in a row, it is removed from the tracking sequence. S14: for other frame images in the image sequence, repeat step S13 to obtain a final tracking sequence, which is the tracking result; In the training phase, for newly appeared objects, the local optimal bipartite graph matching strategy is used to calculate the detection loss function and train the end-to-end multi-target tracking model. The calculation formula is: in, represents the detection loss function, y new Represents the true value label of the newly appeared object, y new =(c new ,b new ), c new Indicates the true category information of the newly appeared object, b new Represents the true position information of the newly appeared object, represents the predicted value of the detection decoding vector, Indicates that the predicted category is c new The probability value of Represents the predicted position information of the newly appeared object, i represents a natural number, M represents the number of detection decoding vectors, represents the classification loss function in the detection task, represents the regression loss function in the detection task, Indicates that the calculation scope of the loss function is the positive sample object, Indicates the number of steps from the i-th true value to the i-th A mapping of predicted values; In the training phase, for objects that have appeared in the tracking sequence, the tracking loss function is calculated by the <true value label-prediction> matching relationship when the object first appears. The calculation formula is: in, represents the tracking loss function, N represents the number of predictions for the tracking query vector, and y old Indicates the true value label of the object that has appeared, y old =(c old ,b old ), c old Indicates the true value category information of the object that has appeared, b old Indicates the true position information of the object that has appeared, represents the predicted value of the tracking query vector, Indicates that the predicted category is c old The probability value of Indicates the predicted position information of the object that has appeared. represents the classification loss function in the tracking task, represents the regression loss function in the tracking task, Indicates the number of steps from the i-th true value to the i-th true value in the tracking task. A mapping of predicted values.

2. The end-to-end multi-target tracking method according to claim 1, characterized in that: In the target association module, a classification confidence threshold β is set, and the detection decoding vector is screened by the classification confidence threshold β, and the detection decoding vector whose classification confidence is not less than the classification confidence threshold β is recorded as a valid detection decoding vector, and the detection decoding vector whose classification confidence is less than the classification confidence threshold β is recorded as an invalid detection decoding vector; In the content interaction module, the effective detection decoding vector is used to update the tracking query vector, the attention weights of the effective detection decoding vector and the tracking query vector are calculated, and the tracking query vector is updated using the effective detection decoding vector according to the attention weights; In the position update module, the position information corresponding to the effective detection decoding vector and the position information corresponding to the tracking query vector are matched using the optimal bipartite graph matching strategy, and the position information of the tracking query vector is updated using the position information of the effective detection decoding vector according to the matching result; The updated tracking query vector, the unmatched valid detection decoding vector, the invalid detection decoding vector and their corresponding position information and the comprehensive coding vector are input into the decoding layer network to obtain the tracking decoding vector.

3. The end-to-end multi-target tracking method according to claim 2, characterized in that: The tracking decode vector generated by the detection decoding vector is input into the classification subnetwork and regression subnetwork of the association head network respectively to obtain the classification result and regression result, and the detection loss function is calculated. The tracking decode vector generated by the tracking query vector is input into the classification subnetwork and regression subnetwork of the association head network respectively to obtain the classification result and regression result, and the tracking loss function is calculated.

4. An end-to-end multi-target tracking device, characterized in that: include: Input module, obtains images and preprocesses them; A construction module is used to construct an end-to-end multi-target tracking model, wherein the end-to-end multi-target tracking model includes a target detection module and a target association module; the target detection module includes a backbone network, an encoder, a decoder and a detection head network, and the target association module includes an associator and an association head network; the associator includes a content interaction module, a position update module and a decoding layer network, and the association head network includes a classification subnetwork and a regression subnetwork, and the classification subnetwork and the regression subnetwork are both composed of multiple linear mapping layers, nonlinear mapping layers and standardized operation modules; Detection module: In the target detection module, the image is input into the backbone network for feature extraction to obtain a feature map, the feature map is position-encoded to obtain a position encoding vector, the feature map and the position encoding vector are serialized and converted and input into the encoder to obtain a comprehensive encoding vector, a detection query vector is set, the detection query vector and the comprehensive encoding vector are input into the decoder to obtain a detection decoding vector, and the detection decoding vector is input into the detection head network to obtain the target detection result; Tracking module: In the target association module, the target detection result and the detection decoding vector are input into the associator to obtain the tracking decoding vector, and the tracking decoding vector is input into the association head network to obtain the associator output result. The tracking sequence is set according to the object appearing in the associator output result, and the tracking result is obtained through the tracking sequence; The steps to obtain tracking results through tracking sequence include: S11: Create a new tracking sequence and set the ID number of the objects in the tracking sequence; S12: for the first frame image in the image sequence, input the target detection result and the detection decoding vector of the first frame image into the association head network to obtain the tracking decoding vector, and input the tracking decoding vector into the association head network to obtain the associator output result, wherein the associator output result includes the object category, the classification confidence and the position information; Set the classification confidence threshold α. If the classification confidence of the associator output result corresponding to the detection decoding vector of the first frame image is not less than the classification confidence threshold α, then mark the associator output result as a newly appeared object, add the object to the tracking sequence and assign an ID number, and record the tracking decoding vector of the object in the tracking sequence as the tracking query vector for tracking the next frame image; S13: for the second frame image in the image sequence, the target detection result, the detection decoding vector, the comprehensive encoding vector, the tracking query vectors and the position information of all objects in the tracking sequence of the second frame image are input into the associator to obtain the tracking decoding vector, and the tracking decoding vector is input into the association head network to obtain the associator output result of the second frame image; If the classification confidence of the associator output result corresponding to the detection decoding vector of the second frame image is not less than the classification confidence threshold α, the associator output result is marked as a newly appeared object, the object is added to the tracking sequence and assigned an ID number, and the tracking decoding vector of the object in the tracking sequence is recorded as the tracking query vector for tracking the next frame image; If the classification confidence of the associator output result corresponding to the tracking query vector of the second frame image is lower than the classification confidence threshold α, in the training phase, the object corresponding to the associator output result is removed from the tracking sequence. In the inference phase, the object corresponding to the associator output result is marked as lost. For the object corresponding to the associator output result that is lost for more than L frames in a row, it is removed from the tracking sequence. S14: for other frame images in the image sequence, repeat step S13 to obtain a final tracking sequence, which is the tracking result; In the training phase, for newly appeared objects, the local optimal bipartite graph matching strategy is used to calculate the detection loss function and train the end-to-end multi-target tracking model. The calculation formula is: in, represents the detection loss function, y new Represents the true value label of the newly appeared object, y new =(c new ,b new ), c new Indicates the true category information of the newly appeared object, b new Represents the true position information of the newly appeared object, represents the predicted value of the detection decoding vector, Indicates that the predicted category is c new The probability value of Represents the predicted position information of the newly appeared object, i represents a natural number, M represents the number of detection decoding vectors, represents the classification loss function in the detection task, represents the regression loss function in the detection task, Indicates that the calculation scope of the loss function is the positive sample object, Indicates the number of steps from the i-th true value to the i-th A mapping of predicted values; In the training phase, for objects that have appeared in the tracking sequence, the tracking loss function is calculated by the <true value label-prediction> matching relationship when the object first appears. The calculation formula is: in, represents the tracking loss function, N represents the number of predictions for the tracking query vector, and y old Indicates the true value label of the object that has appeared, y old =(c old ,b old ), c old Indicates the true value category information of the object that has appeared, b old Indicates the true position information of the object that has appeared, represents the predicted value of the tracking query vector, Indicates that the predicted category is c old The probability value of Indicates the predicted position information of the object that has appeared. represents the classification loss function in the tracking task, represents the regression loss function in the tracking task, Indicates the number of steps from the i-th true value to the i-th true value in the tracking task. A mapping of predicted values.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is executed.

Images