A video multi-target tracking method based on query selection

Abstract

The present invention relates to the field of video tracking technology, and in particular to a video multi-target tracking method based on query selection, which is used to perform multi-target tracking based on a query coding representation of a current frame video image, thereby realizing real-time tracking of long sequence video data and improving the accuracy and efficiency of multi-target tracking.

Description

Technical Field

[0001] The present invention relates to the technical field of video tracking, and in particular to a video multi-target tracking method, device, computer equipment and storage medium based on query selection. Background Art

[0002] Multi-object tracking (MOT) is an important task in the field of computer vision, which aims to identify and track the motion trajectories of multiple targets from video sequences.

[0003] In recent years, with the widespread application of the Transformer framework in the field of natural language processing, many researchers have begun to try to apply it to multi-target tracking tasks. However, due to its complex structure and a large number of parameters, the Transformer model requires strong computing power to maintain real-time performance during the inference stage. In addition, the Transformer model has a high computational complexity and takes a long time to process each frame of video data. Especially when processing long sequence data, the delay problem is more prominent, making it difficult to meet the needs of real-time tracking. Summary of the invention

[0004] Based on this, the purpose of the present invention is to provide a video multi-target tracking method, device, computer equipment and storage medium based on query selection, which is used to perform multi-target tracking based on the constructed query coding representation of the current frame video image, thereby realizing real-time tracking of long sequence video data and improving the accuracy and efficiency of multi-target tracking.

[0005] In a first aspect, an embodiment of the present application provides a video multi-target tracking method based on query selection, comprising the following steps:

[0006] Obtaining a video sequence to be tracked and a preset multi-target tracking model, wherein the video sequence to be tracked includes a plurality of frames of video images, and the multi-target tracking model includes a feature extraction module, a query selection module, a decoding module, and a tracking module;

[0007] Inputting the video sequence to be tracked into the feature extraction module, obtaining feature maps of several scales of the current frame video image, performing feature fusion on the feature maps of several scales of the current frame video image, and obtaining several multi-scale feature maps of the current frame video image;

[0008] Inputting a plurality of multi-scale feature maps of the current frame video image into the query selection module for query selection, obtaining a tracking query representation and a target query representation of the current frame video image, and splicing the tracking query representation and the target query representation of the current frame video image to obtain a query encoding representation of the current frame video image;

[0009] Inputting the query encoding representation of the current frame video image and a plurality of multi-scale feature maps into the decoding module for decoding processing to obtain a query decoding representation of the current frame video image, wherein the query decoding representation includes a plurality of decoding query vectors corresponding to the queries;

[0010] The query decoding representation of the current frame video image is input into the tracking module, and multi-target tracking is performed according to the decoded query vectors corresponding to several queries in the query decoding representation to obtain the predicted multi-target tracking result of the current frame video image, and the multi-target tracking is repeated for the next frame video image until the predicted multi-target tracking results of several frames of video images in the video sequence to be tracked are obtained.

[0011] In a second aspect, an embodiment of the present application provides a video multi-target tracking device based on query selection, comprising:

[0012] A data acquisition module, used to obtain a video sequence to be tracked and a preset multi-target tracking model, wherein the video sequence to be tracked includes a plurality of frames of video images, and the multi-target tracking model includes a feature extraction module, a query selection module, a decoding module and a tracking module;

[0013] A feature processing module, used for inputting the video sequence to be tracked into the feature extraction module, obtaining feature maps of several scales of the current frame video image, performing feature fusion on the feature maps of several scales of the current frame video image, and obtaining several multi-scale feature maps of the current frame video image;

[0014] A query initialization selection module, used to input a plurality of multi-scale feature maps of the current frame video image into the query selection module for query selection, obtain a tracking query representation and a target query representation of the current frame video image, and splice the tracking query representation and the target query representation of the current frame video image to obtain a query encoding representation of the current frame video image;

[0015] A query decoding module, used for inputting the query encoding representation of the current frame video image and a plurality of multi-scale feature maps into the decoding module for decoding processing, and obtaining a query decoding representation of the current frame video image, wherein the query decoding representation includes a plurality of decoding query vectors corresponding to the queries;

[0016] A multi-target tracking module is used to input the query decoding representation of the current frame video image into the tracking module, perform multi-target tracking according to the decoded query vectors corresponding to several queries in the query decoding representation, obtain the predicted multi-target tracking result of the current frame video image, and repeat the multi-target tracking for the next frame video image until the predicted multi-target tracking results of several frames of video images in the video sequence to be tracked are obtained.

[0017] In a third aspect, an embodiment of the present application provides a computer device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the video multi-target tracking method based on query selection as described in the first aspect are implemented.

[0018] In a fourth aspect, an embodiment of the present application provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the video multi-target tracking method based on query selection as described in the first aspect.

[0019] In an embodiment of the present application, a query-selection-based video multi-target tracking method, apparatus, computer equipment and storage medium are provided, which are used to perform multi-target tracking based on a query coding representation of a current frame video image, thereby achieving real-time tracking of long sequence video data and improving the accuracy and efficiency of multi-target tracking.

[0020] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flowchart of a video multi-target tracking method based on query selection provided by an embodiment of the present application;

[0022] Figure 2 A schematic diagram of S2 in the process of a video multi-target tracking method based on query selection provided by an embodiment of the present application;

[0023] Figure 3 A schematic diagram of S3 in the process of a video multi-target tracking method based on query selection provided by an embodiment of the present application;

[0024] Figure 4 A schematic diagram of S31 in the process of a video multi-target tracking method based on query selection provided by an embodiment of the present application;

[0025] Figure 5 A schematic diagram of S4 in the process of a video multi-target tracking method based on query selection provided by an embodiment of the present application;

[0026] Figure 6 A schematic diagram of S5 in the process of a video multi-target tracking method based on query selection provided by an embodiment of the present application;

[0027] Figure 7 A schematic diagram of S6 in the process of a video multi-target tracking method based on query selection provided by another embodiment of the present application;

[0028] Figure 8A schematic diagram of the structure of a video multi-target tracking device based on query selection provided by an embodiment of the present application;

[0029] Fig. 9 A schematic diagram of the structure of a computer device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0030] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0031] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0032] It should be understood that, although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" / "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".

[0033] See also Figure 1 , Figure 1 A flowchart of a video multi-target tracking method based on query selection is provided in one embodiment of the present application. The method comprises the following steps:

[0034] S1: Obtain a video sequence to be tracked and a preset multi-target tracking model.

[0035] The executor of the query selection-based video multi-target tracking method is a tracking device of the query selection-based video multi-target tracking method (hereinafter referred to as the tracking device). In an optional embodiment, the tracking device can be a computer device, a server, or a server cluster composed of multiple computer devices.

[0036] In this embodiment, the tracking device obtains the video sequence to be tracked by querying a preset database, wherein the video sequence to be tracked includes a plurality of frames of video images.

[0037] The tracking device obtains a preset multi-target tracking model, wherein the multi-target tracking model includes a feature extraction module, a query selection module, a decoding module and a tracking module.

[0038] S2: Input the video sequence to be tracked into the feature extraction module to obtain feature maps of several scales of the current frame video image, perform feature fusion on the feature maps of several scales of the current frame video image, and obtain several multi-scale feature maps of the current frame video image.

[0039] The tracking device adopts the Resnet50 neural network as the feature extraction module. In this embodiment, the tracking device inputs the video sequence to be tracked into the feature extraction module, obtains feature maps of several scales of the current frame video image, performs feature fusion on the feature maps of several scales of the current frame video image, and obtains several multi-scale feature maps of the current frame video image.

[0040] Specifically, the Resnet50 neural network structure includes several stacked convolutional units, which are used to calculate feature maps of different scales of video images. Figure 2 , Figure 2 The schematic diagram of S2 in the process of the video multi-target tracking method based on query selection provided by an embodiment of the present application includes steps S21 to S23, which are specifically as follows:

[0041] S21: Use the current frame video image as the input image of the first convolution unit, perform convolution processing on the input image, and obtain a convolution feature map corresponding to the input image; down-sample the convolution feature map, and input the convolution feature map after the down-sampling processing to the next convolution unit until the convolution feature map output by the last convolution unit is obtained, and use the convolution feature maps output by the last several convolution units as the feature maps to obtain feature maps of several scales of the current frame video image.

[0042] In this embodiment, the tracking device uses the current frame video image as the input image of the first convolution unit, performs convolution processing on the input image, obtains a convolution feature map corresponding to the input image, downsamples the convolution feature map, and inputs the convolution feature map after the downsampling processing to the next convolution unit to reduce the size of the convolution feature map, until the convolution feature map output by the last convolution unit is obtained.

[0043] The tracking device uses the convolution feature maps output by the last several convolution units as the feature maps to obtain feature maps of several scales of the current frame video image. Specifically, the convolution feature maps output by the last three convolution units of the tracking device, the F3 convolution feature map, the F4 convolution feature map and the F5 convolution feature map are used as the final output of the Resnet50 neural network to obtain feature maps of several scales of the current frame video image.

[0044] S22: extracting a feature map of a target scale from feature maps of several scales of the current frame video image, and performing self-attention processing on the feature map of the target scale using a self-attention mechanism to obtain a self-attention feature map of the current frame video image.

[0045] In order to reduce delay time and computational complexity, in this embodiment, the tracking device extracts a feature map of the target scale from feature maps of several scales of the current frame video image. Specifically, the tracking device uses the feature map of the smallest scale as the feature map of the target scale.

[0046] The tracking device uses a self-attention mechanism to perform self-attention processing on the feature map of the target scale to obtain the self-attention feature map of the current frame video image. Specifically, the tracking device maps the feature map of the target scale to a query matrix (Q), a key matrix (K), and a value matrix (V), and obtains an attention score matrix by calculating the dot product of Q and K according to a preset self-attention algorithm, and then processes it with a normalized exponential function to obtain an attention weight matrix, and finally applies the attention weight matrix to V to obtain the self-attention feature map of the current frame video image, so as to establish a global dependency of the feature map so that each feature vector has a rich context representation, wherein the self-attention algorithm is:

[0047]

[0048] In the formula, is the self-attention feature map, is the normalized exponential function, is the dimension parameter, T is the bias sign.

[0049] S23: Perform feature fusion processing according to the self-attention feature map of the current frame video image and feature maps of other scales to obtain several multi-scale feature maps of the current frame video image.

[0050] In this embodiment, the tracking device performs feature fusion processing based on the self-attention feature map of the current frame video image and feature maps of other scales to obtain several multi-scale feature maps of the current frame video image. Specifically, the tracking device upsamples the self-attention feature map of the current frame video image from top to bottom through bilinear interpolation, expands it to the same scale as the F4 convolution feature map, convolutionally fuses the upsampled self-attention feature map with the F4 convolution feature map, obtains the P4 fusion feature map, upsamples the P4 fusion feature map again to align it with the F3 convolution feature map of the largest scale, and convolutionally fuses the P4 fusion feature map with the F3 convolution feature map of the largest scale to obtain the P3 fusion feature map.

[0051] Subsequently, the tracking device uses the P3 fusion feature map as the N3 fusion feature map, downsamples the N3 fusion feature map through the convolution layer from bottom to top, reduces it to the same scale as the P4 fusion feature map, and convolutionally fuses the N3 fusion feature map with the largest scale P4 fusion feature map to obtain the N4 fusion feature map; then, the tracking device downsamples the N4 fusion feature map through the convolution layer, reduces it to the same scale as the P5 fusion feature map, and convolutionally fuses the N4 fusion feature map with the largest scale P5 fusion feature map to obtain the N5 fusion feature map, thereby obtaining several multi-scale feature maps of the current frame video image.

[0052] S3: Inputting several multi-scale feature maps of the current frame video image into the query selection module for query selection, obtaining the tracking query representation and the target query representation of the current frame video image, splicing the tracking query representation and the target query representation of the current frame video image, and obtaining the query coding representation of the current frame video image.

[0053] In this embodiment, the tracking device inputs several multi-scale feature maps of the current frame video image into the query selection module for query selection, obtains the tracking query representation and the target query representation of the current frame video image, splices the tracking query representation and the target query representation of the current frame video image, and obtains the query coding representation of the current frame video image.

[0054] See also Figure 3 , Figure 3 The schematic diagram of S3 in the process of the video multi-target tracking method based on query selection provided by an embodiment of the present application includes steps S31 to S33, which are specifically as follows:

[0055] S31: Flatten and splice several multi-scale feature maps of the current frame video image to construct an initial query representation of the current frame video image; confirm several tracking queries and identity tags corresponding to the tracking queries from several queries in the initial query representation to construct an initial tracking query representation of the current frame video image.

[0056] In this embodiment, the tracking device flattens and splices several multi-scale feature maps of the current frame video image to construct an initial query representation of the current frame video image. The tracking device confirms several tracking queries and identity tags corresponding to the tracking queries from several queries in the initial query representation to construct an initial tracking query representation of the current frame video image, wherein the initial tracking query representation includes initial query vectors corresponding to several tracking queries.

[0057] See also Figure 4 , Figure 4 The schematic diagram of S31 in the process of the video multi-target tracking method based on query selection provided by an embodiment of the present application includes steps S311 to S312, which are specifically as follows:

[0058] S311: Calculate the confidence of the initial query representation of the current frame video image to obtain confidence scores corresponding to several queries of the current frame video image; extract query vectors corresponding to several candidate queries from the initial query representation based on the confidence scores, and construct a candidate query representation of the current frame video image.

[0059] In this embodiment, the tracking device performs confidence calculation on the initial query representation of the current frame video image according to a preset confidence calculation algorithm to obtain confidence scores corresponding to several queries of the current frame video image, wherein the confidence calculation algorithm is:

[0060]

[0061] In the formula, is the confidence score, is the confidence weight parameter, is the confidence bias parameter, is the initial query representation.

[0062] The tracking device selects several candidate queries with the highest confidence from several queries using the Top-K method according to the confidence score to represent the target to be tracked in the current frame video image, and extracts query vectors corresponding to several candidate queries from the initial query representation according to the candidate queries to construct a candidate query representation of the current frame video image.

[0063] S312: Obtain an identity tag query representation of the previous frame of video image; perform similarity calculation based on the candidate query representation of the current frame of video image and the identity tag query representation of the previous frame of video image to obtain similarity scores between several candidate queries of the current frame of video image and several queries corresponding to identity tags of the previous frame of video image; based on the similarity scores, confirm several tracking queries and the identity tags corresponding to the tracking queries.

[0064] In this embodiment, the tracking device obtains an identity tag query representation of the previous frame of video image, wherein the identity tag query representation includes a decoded query vector corresponding to a query corresponding to a number of identity tags, and the identity tag is used to indicate the ID of the target frame corresponding to the query.

[0065] The tracking device performs similarity calculation based on the candidate query representation of the current frame video image, the identity tag query representation of the previous frame video image, and a preset similarity calculation algorithm to obtain similarity scores between several candidate queries of the current frame video image and queries corresponding to several identity tags of the previous frame video image, wherein the similarity calculation algorithm is:

[0066]

[0067] In the formula, is the i-th candidate query of the current frame video image and the i-th candidate query of the previous frame video image. j The similarity score between the queries corresponding to the identity tags, The current frame of the video image i The query vector corresponding to the candidate query is is the first frame of the previous video image. j The decoded query vector corresponding to the query corresponding to the identity label.

[0068] The tracking device uses the similarity scores between several candidate queries of the current frame video image and the queries corresponding to several identity tags of the previous frame video image and a preset similarity threshold. If the similarity score is greater than the similarity threshold, the candidate query of the current frame video image is used as the tracking query, and the identity tag associated with the candidate query is used as the identity tag of the tracking query, and confirms several tracking queries and the identity tags corresponding to the tracking queries.

[0069] S32: According to the embedding vectors corresponding to several preset identity tags, the values ​​of the initial query vectors corresponding to several tracking queries in the initial tracking query representation of the current frame video image are added to the values ​​of the embedding vectors of the corresponding identity tags to obtain the added query vectors corresponding to the several tracking queries of the current frame video image, and construct the tracking query representation of the current frame video image.

[0070] Since there may be tracking queries corresponding to multiple identity tags in the current frame video image, that is, multiple ID targets may need to be tracked, in order to distinguish several of the tracking queries, in this embodiment, the tracking device adds the values ​​of the initial query vectors corresponding to several tracking queries in the initial tracking query representation of the current frame video image to the values ​​of the embedding vectors of the corresponding identity tags based on the embedding vectors corresponding to several preset identity tags, and obtains the added query vectors corresponding to the several tracking queries of the current frame video image.

[0071] S33: Obtain the target detection result of the current frame video image, perform sine and cosine encoding on the detection results of several target boxes in the target detection result of the current frame video image, and obtain the embedding vectors of several target boxes of the current frame video image; respectively add the values ​​of the embedding vectors of several target boxes of the current frame video image to the values ​​of preset learnable embedding vectors to obtain the added embedding vectors of several target boxes as query vectors corresponding to several target queries, and construct the target query representation of the current frame video image.

[0072] In this embodiment, the tracking device obtains a target query representation of the current frame video image, wherein the target query represents a query vector corresponding to a plurality of target queries. Specifically, the tracking device inputs the current frame video image into a preset YOLOx target detector to obtain a target detection result of the current frame video image, wherein the target detection result is:

[0073]

[0074] In the formula, z is the target detection result, is the detection result of the nth target box, , x and y are the horizontal and vertical coordinates of the center point of the target frame, h And w is the height and width of the target box, and s is the confidence of the target box.

[0075] The tracking device performs sine and cosine encoding on the detection results of several target frames in the target detection results of the current frame video image to obtain the embedding vectors of several target frames of the current frame video image, and respectively adds the values ​​of the embedding vectors of several target frames of the current frame video image to the values ​​of preset learnable embedding vectors to obtain the added embedding vectors of several target frames as query vectors corresponding to several target queries, and constructs the target query representation of the current frame video image, wherein the target query representation is:

[0076]

[0077] In the formula, is the target query representation, is the query vector corresponding to the nth target query.

[0078] S4: Inputting the query encoding representation of the current frame video image and a plurality of multi-scale feature maps into the decoding module for decoding processing to obtain the query decoding representation of the current frame video image.

[0079] In this embodiment, the tracking device inputs the query encoding representation of the current frame video image and a plurality of multi-scale feature maps into the decoding module for decoding processing to obtain the query decoding representation of the current frame video image.

[0080] Specifically, the decoding module includes a plurality of stacked attention units, each of which includes a self-attention layer and a cross-attention layer connected in sequence, see Figure 5 , Figure 5 The schematic diagram of S4 in the process of the video multi-target tracking method based on query selection provided by an embodiment of the present application includes steps S41 to S43, which are specifically as follows:

[0081] S41: Using the query encoding representation of the current frame video image as the input representation of the self-attention layer of the first attention unit, performing self-attention processing on the input representation of the current frame video image to obtain the self-attention feature representation of the current frame video image.

[0082] In this embodiment, the tracking device uses the query encoding representation of the current frame video image as the input representation of the self-attention layer of the first attention unit, and performs self-attention processing on the input representation of the current frame video image according to a preset self-attention algorithm to obtain the self-attention feature representation of the current frame video image, wherein the self-attention feature representation includes attention feature vectors corresponding to several queries. For the specific embodiment, please refer to step S22, which will not be repeated here.

[0083] S42: Inputting the self-attention feature representation of the current frame video image into the cross-attention layer for linear transformation to obtain decoding parameters of the current frame video image; performing cross-attention processing according to the decoding parameters of the current frame video image and a plurality of multi-scale feature maps to obtain the cross-attention feature representation of the current frame video image.

[0084] In this embodiment, the tracking device inputs the self-attention feature representation of the current frame video image into the cross attention layer, performs linear transformation according to a preset linear transformation algorithm, and obtains the decoding parameters of the current frame video image, wherein the decoding parameters include a cross attention weight parameter and a cross attention offset parameter, and the linear transformation algorithm is:

[0085]

[0086] In the formula, is the cross-attention weight parameter of the k-th sampling point of the l-th multi-scale feature map corresponding to the q-th query of the m-th cross-attention head, is the cross-attention bias parameter of the k sampling points of the l-th multi-scale feature map corresponding to the q-th query of the m-th cross-attention head, is the self-attention feature representation, as well as are the first weight parameter and the second weight parameter respectively preset, as well as They are respectively a preset first bias parameter and a second bias parameter.

[0087] The tracking device performs cross-attention processing according to the decoding parameters of the current frame video image, a plurality of multi-scale feature maps and a preset multi-head cross-attention algorithm to obtain a cross-attention feature representation of the current frame video image, wherein the cross-attention algorithm is:

[0088]

[0089] In the formula, is the cross attention feature representation, as well as are the first weight parameters of the preset mth cross attention head, M is the number of cross attention heads, K is the number of sampling points, L is the number of multi-scale feature maps, is the lth multi-scale feature map, is the normalized reference point coordinate parameter of the qth query on the multi-scale feature map, .

[0090] S43: Input the cross-attention feature representation of the current frame video image to the next attention unit until the cross-attention feature representation of the current frame video image output by the last attention unit is obtained as the query decoding representation, thereby obtaining the query decoding representation of the current frame video image.

[0091] In this embodiment, the tracking device inputs the cross-attention feature representation of the current frame video image to the next attention unit until the cross-attention feature representation of the current frame video image output by the last attention unit is obtained as the query decoding representation, thereby obtaining the query decoding representation of the current frame video image.

[0092] S5: Input the query decoding representation of the current frame video image into the tracking module, perform multi-target tracking according to the decoded query vectors corresponding to several queries in the query decoding representation, obtain the predicted multi-target tracking result of the current frame video image, and repeat the multi-target tracking for the next frame video image until the predicted multi-target tracking results of several frames of video images in the video sequence to be tracked are obtained.

[0093] In this embodiment, the tracking device inputs the query decoding representation of the current frame video image into the tracking module, performs multi-target tracking according to the decoded query vectors corresponding to several queries in the query decoding representation, and obtains the predicted multi-target tracking result of the current frame video image, wherein the predicted multi-target tracking result includes several tracked successful queries and predicted tracking results of tracking newly added queries, and the predicted tracking result includes predicted positioning parameters and identity tags.

[0094] The tracking device repeatedly performs multi-target tracking on the next frame of video image until the predicted multi-target tracking results of several frames of video images in the video sequence to be tracked are obtained. The query coding representation of the current frame of video image is constructed to perform multi-target tracking, thereby realizing real-time tracking of long sequence video data and improving the accuracy and efficiency of multi-target tracking.

[0095] Specifically, the tracking module is a fully connected network (FFN) Figure 6 , Figure 6 The schematic diagram of S5 in the process of the video multi-target tracking method based on query selection provided by an embodiment of the present application includes steps S51 to S54, which are specifically as follows:

[0096] S51: Calculate category scores according to decoded query vectors corresponding to several tracking queries and target queries in the query decoding representation of the current frame video image to obtain predicted category scores of the several tracking queries and target queries of the current frame video image.

[0097] In this embodiment, the tracking device performs category score calculation according to the decoded query vectors corresponding to the tracking queries and the target query in the query decoded representation of the current frame video image and a preset category score calculation algorithm to obtain the predicted category scores of the tracking queries and the target query of the current frame video image, wherein the category score calculation algorithm is:

[0098]

[0099] In the formula, is the predicted category score for the nth query, is the category score weight parameter, is the decoded query vector corresponding to the nth query, is the class score bias parameter.

[0100] S52: Based on the predicted category score of the tracking query and the corresponding identity tag, obtain a tracking query with the maximum predicted category score corresponding to several identity tags; based on the tracking query with the maximum category score and a preset tracking category score threshold, obtain a tracking query with the maximum predicted category score corresponding to several identity tags that meet the tracking category score threshold as a successful tracking query, and obtain several identity tags of the successful tracking queries.

[0101] For multi-target tracking tasks, there are only two categories: foreground and background. The predicted category score can be regarded as the confidence that the query can track the target.

[0102] In this embodiment, the tracking device obtains a tracking query with a maximum predicted category score corresponding to several identity tags based on the predicted category score of the tracking query and the corresponding identity tag. Based on the tracking query with the maximum category score and a preset tracking category score threshold, if the predicted category score of the tracking query is greater than the tracking category score threshold, it is determined that the tracking query satisfies the tracking category score threshold, and a tracking query with a maximum predicted category score corresponding to several identity tags that satisfy the tracking category score threshold is obtained as a successful tracking query, and several identity tags of the successful tracking queries are obtained.

[0103] S53: According to the predicted category score of the target query and a preset target category score threshold, a target query that meets the target category score threshold is obtained as a tracked new query, and several identity tags of the tracked new queries are obtained.

[0104] In this embodiment, the tracking device determines that the target query satisfies the target category score threshold based on the predicted category score of the target query and a preset target category score threshold. If the predicted category score of the target query is greater than the target category score threshold, the tracking device obtains the target query that satisfies the target category score threshold as a newly added query for tracking, uses the corresponding identity tag as the identity tag, and obtains several identity tags for the newly added queries for tracking.

[0105] S54: Perform positioning processing according to the decoded query vectors corresponding to the multiple tracking successful queries and the tracking newly added queries of the current frame video image, and obtain the predicted positioning parameters of the multiple tracking successful queries and the tracking newly added queries of the current frame video image.

[0106] In this embodiment, the tracking device performs positioning processing according to the decoded query vectors corresponding to the multiple tracking successful queries and the newly added tracking queries of the current frame video image and the preset positioning algorithm to obtain the predicted positioning parameters of the multiple tracking successful queries and the newly added tracking queries of the current frame video image, wherein the positioning algorithm is:

[0107]

[0108] In the formula, is the predicted positioning parameter for the nth query, is the weight parameter for target box positioning, Position the offset parameters for the target box.

[0109] In an optional embodiment, the step S6 is further included: training the multi-target tracking model, see Figure 7 , Figure 7 A flowchart of S6 in a video multi-target tracking method based on query selection provided in another embodiment of the present application includes steps S61 to S65, which are specifically as follows:

[0110] S61: Obtain a sample video sequence and actual multi-target tracking results of several frames of sample images in the sample video sequence.

[0111] In this embodiment, the tracking device obtains a sample video sequence and real multi-target tracking results of several frames of sample images in the sample video sequence, wherein the real multi-target tracking results include real tracking results of several queries, the queries include tracking queries and target queries, and the real tracking results include real positioning parameters and real category scores.

[0112] S62: Input the sample video sequence into the multi-target tracking model to be trained to obtain predicted tracking results of several queries of several frames of sample images; adopt a binary matching method to construct a matching strategy based on several tracking queries of several frames of sample images and actual tracking results of target queries and predicted tracking results, and obtain matching relationship results between several queries of several frames of sample images.

[0113] In this embodiment, the tracking device inputs the sample video sequence into the multi-target tracking model to be trained to obtain predicted tracking results of several queries of several frames of sample images. The specific implementation method can refer to steps S2~S5, which will not be repeated here.

[0114] The tracking device adopts a binary matching method, constructs a matching strategy according to several tracking queries of several frame sample images and actual tracking results of target queries and predicted tracking results, and obtains matching relationship results between several queries of several frame sample images.

[0115] Specifically, the tracking device constructs real label parameters of several target queries of several frames of sample images according to the real positioning parameters and real category scores in the real tracking results of several target queries of several frames of sample images; and constructs predicted label parameters of several target queries of several frames of sample images according to the predicted positioning parameters and predicted category scores in the predicted tracking results, as follows:

[0116] ;

[0117] In the formula, is the true label parameter of the target query, is the actual positioning parameter of the target query, is the true category score for the target query, is the predicted label parameter of the target query, is the predicted positioning parameter of the target query, A predicted category score for the target query.

[0118] The tracking device constructs an algorithm based on the real label parameters, predicted label parameters and preset target query label allocation strategy of several target queries of several frame sample images to obtain the target query label allocation strategy, wherein the target query label allocation strategy construction algorithm is:

[0119]

[0120] In the formula, assigning a strategy to the target query label, To obtain the minimum set function, is the preset bipartite matching space, is the binary matching loss function.

[0121] The tracking device constructs the real label parameters of the tracking query corresponding to the several identity labels of the several frame sample images according to the identity labels of the several tracking queries of the several frame sample images and the real positioning parameters and the real category scores in the real tracking results, as follows:

[0122]

[0123] In the formula, is a combined sequence of the real label parameters of the tracking query corresponding to several identity labels of the current frame sample image, is the true label parameter of the tracking query corresponding to the j-th identity label, is the real positioning parameter of the tracking query corresponding to the j-th identity tag, is the true category score of the tracking query corresponding to the j-th identity label.

[0124] The tracking device constructs the predicted label parameters of the tracking query corresponding to the identity labels of the several frames of sample images according to the identity labels of the several tracking queries of the several frames of sample images and the predicted positioning parameters and the predicted category scores in the predicted tracking results, as follows:

[0125]

[0126] In the formula, is a combined sequence of predicted label parameters of tracking queries corresponding to several identity labels of the current frame sample image, is the predicted label parameter of the tracking query corresponding to the jth identity label, is the predicted positioning parameter of the tracking query corresponding to the jth identity tag, is the predicted category score of the tracking query corresponding to the j-th identity label.

[0127] The tracking device constructs an algorithm according to the real label parameters, predicted label parameters and preset tracking query label allocation strategy of the tracking query corresponding to the several identity labels of the several frames of sample images, and obtains the tracking query label allocation strategy. The tracking device uses the target query label allocation strategy and the tracking query label allocation strategy to obtain the matching relationship results between the several queries of the several frames of sample images, wherein the tracking query label allocation strategy construction algorithm is:

[0128]

[0129] In the formula, A policy is assigned to the tracking query tag.

[0130] S63: Constructing several groups of query matching combinations according to the matching relationship results between several queries of several frames of sample images; performing loss calculation according to the real positioning parameters and predicted positioning parameters of the corresponding queries in the query matching combinations to obtain the positioning loss values ​​of the several frames of sample images; performing loss calculation according to the real category scores and predicted category scores of the corresponding queries in the query matching combinations to obtain the category loss values ​​of the several frames of sample images.

[0131] In this embodiment, the tracking device constructs several groups of query matching combinations according to the matching relationship results between several queries of several frames of sample images, and performs loss calculation according to the real positioning parameters, predicted positioning parameters and preset positioning loss algorithm of the corresponding queries in the query matching combinations to obtain the positioning loss values ​​of several frames of sample images, wherein the positioning loss algorithm is:

[0132]

[0133] In the formula, is the positioning loss value, is the predicted positioning parameter of the i-th query, is the actual positioning parameter of the j-th query, It is an overlap calculation function, which is used to measure the degree of overlap between the predicted target frame and the real target frame in space. is the positioning loss weight parameter.

[0134] The tracking device performs loss calculation according to the real category score, the predicted category score and the preset category loss algorithm of the corresponding query in the query matching combination to obtain the category loss value of several frames of sample images, wherein the category loss algorithm is:

[0135]

[0136] In the formula, is the category loss value, is the predicted category score for the i-th query, Score the true category of the j-th query.

[0137] By using the positioning loss value obtained based on the matching relationship results and the category loss value obtained by predicting the category score for model training, the model can focus on new targets, namely target queries, thereby improving the accuracy of multi-target tracking.

[0138] S64: constructing a positive query set and a negative query set for several queries of several frames of sample images according to identity labels of several queries of several frames of sample images; performing loss calculation according to the positive query set and the negative query set for several queries of several frames of sample images to obtain distance loss values ​​of several frames of sample images.

[0139] In this embodiment, the tracking device constructs a positive query set and a negative query set of several queries for several frames of sample images based on the identity tags of several queries for several frames of sample images, wherein the positive query set includes several queries corresponding to several identical identity tags in the previous frame of sample images; and the negative query set includes several queries corresponding to several different identity tags in the current frame of sample images.

[0140] The tracking device calculates the loss based on the positive query set, the negative query set and the preset distance loss algorithm of the several frames of sample images, and obtains the distance loss values ​​of the several frames of sample images for model training, so that the model can better model the same features of the same target and the difference features of different targets, shorten the distance between positive examples and increase the distance between negative examples, and improve the accuracy of target tracking of the model, wherein the distance loss algorithm is:

[0141]

[0142] In the formula, is the distance loss value, A is the number of identity labels of the current frame sample image, B is the number of identity labels of the previous frame sample image, and U is the preset margin parameter. is the Euclidean distance between the query of the identity label j of the current frame sample image and the query of the same identity label of the previous frame sample image in the corresponding positive query set, is the Euclidean distance between the query of the identity label j of the current frame sample image and the query of the different identity label l of the current frame sample image in the corresponding negative query set, is the activation function.

[0143] S65: Training the multi-target tracking model to be trained according to the positioning loss values, category loss values ​​and distance loss values ​​of several frames of sample images.

[0144] In this embodiment, the tracking device obtains a total loss value according to the positioning loss values, category loss values, distance loss values ​​and a preset total loss algorithm of several frames of sample images, and trains the multi-target tracking model to be trained according to the total loss value, wherein the total loss algorithm is:

[0145]

[0146] In the formula, is the total loss value, , as well as In an optional embodiment, in order to enable the model to better learn the motion law of each target, the tracking device counts the total loss value of each frame sample image, but only performs back propagation once after averaging the total loss values ​​of all 5 frames of sample images after every 5 frames to update the hyperparameters, so as to further improve the accuracy of model training and improve the accuracy and efficiency of the model's multi-target tracking.

[0147] Please refer to Figure 8 , Figure 8 A schematic diagram of the structure of a video multi-target tracking device based on query selection provided by an embodiment of the present application. The device can implement all or part of the video multi-target tracking device based on query selection through software, hardware, or a combination of both. The device 8 includes:

[0148] A data acquisition module 81 is used to obtain a video sequence to be tracked and a preset multi-target tracking model, wherein the video sequence to be tracked includes a plurality of video frames, and the multi-target tracking model includes a feature extraction module, a query selection module, a decoding module and a tracking module;

[0149] A feature processing module 82 is used to input the video sequence to be tracked into the feature extraction module, obtain feature maps of several scales of the current frame video image, perform feature fusion on the feature maps of several scales of the current frame video image, and obtain several multi-scale feature maps of the current frame video image;

[0150] A query initialization selection module 83 is used to input a plurality of multi-scale feature maps of the current frame video image into the query selection module for query selection, obtain a tracking query representation and a target query representation of the current frame video image, and splice the tracking query representation and the target query representation of the current frame video image to obtain a query encoding representation of the current frame video image;

[0151] A query decoding module 84 is used to input the query encoding representation of the current frame video image and a plurality of multi-scale feature maps into the decoding module for decoding processing to obtain a query decoding representation of the current frame video image, wherein the query decoding representation includes a plurality of decoding query vectors corresponding to the queries;

[0152] The multi-target tracking module 85 is used to input the query decoding representation of the current frame video image into the tracking module, perform multi-target tracking according to the decoded query vectors corresponding to several queries in the query decoding representation, obtain the predicted multi-target tracking result of the current frame video image, and repeat the multi-target tracking for the next frame video image until the predicted multi-target tracking results of several frames of video images in the video sequence to be tracked are obtained.

[0153] In an embodiment of the present application, a video sequence to be tracked and a preset multi-target tracking model are obtained through a data acquisition module, wherein the video sequence to be tracked includes several frames of video images, and the multi-target tracking model includes a feature extraction module, a query selection module, a decoding module and a tracking module; through a feature processing module, the video sequence to be tracked is input into the feature extraction module to obtain feature maps of several scales of the current frame video image, and feature fusion is performed on the feature maps of several scales of the current frame video image to obtain several multi-scale feature maps of the current frame video image; through a query initialization selection module, several multi-scale feature maps of the current frame video image are input into the query selection module for query selection to obtain a tracking query representation and a target query representation of the current frame video image, and the current frame video is input into the query selection module for query selection to obtain a tracking query representation and a target query representation of the current frame video image. The tracking query representation and the target query representation of the image are spliced ​​to obtain the query coding representation of the current frame video image; through the query decoding module, the query coding representation of the current frame video image and several multi-scale feature maps are input into the decoding module for decoding processing to obtain the query decoding representation of the current frame video image, wherein the query decoding representation includes several decoding query vectors corresponding to the queries; through the multi-target tracking module, the query decoding representation of the current frame video image is input into the tracking module, and multi-target tracking is performed according to the decoding query vectors corresponding to the several queries in the query decoding representation to obtain the predicted multi-target tracking result of the current frame video image, and the multi-target tracking is repeated for the next frame video image until the predicted multi-target tracking results of several frames of the video sequence to be tracked are obtained. Based on the constructed query coding representation of the current frame video image, multi-target tracking is performed to realize real-time tracking of long sequence video data, and improve the accuracy and efficiency of multi-target tracking.

[0154] Please refer to Fig. 9 , Fig. 9 The computer device 9 is a schematic diagram of a structure of a computer device provided in an embodiment of the present application. The computer device 9 includes: a processor 91, a memory 92, and a computer program 93 stored in the memory 92 and executable on the processor 91; the computer device may store a plurality of instructions, which are suitable for being loaded and executed by the processor 91 Figures 1 to 7 The method steps of the embodiment shown in the figure can be found in the specific implementation process. Figures 1 to 7 The specific description of the illustrated embodiment will not be repeated here.

[0155] Among them, the processor 91 may include one or more processing cores. The processor 91 uses various interfaces and lines to connect various parts in the server, and executes various functions and processes data of the video multi-target tracking device 8 based on query selection by running or executing instructions, programs, code sets or instruction sets stored in the memory 92, and calling the data in the memory 92. Optionally, the processor 91 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programble Logic Array, PLA). The processor 91 can integrate one or more combinations of a central processing unit 91 (Central Processing Unit, CPU), an image processor 91 (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the touch display screen; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 91, but may be implemented separately through a chip.

[0156] The memory 92 may include a random access memory 92 (RAM) or a read-only memory 92 (Read-Only Memory). Optionally, the memory 92 includes a non-transitory computer-readable storage medium. The memory 92 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 92 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch instructions, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 92 may also be optionally at least one storage device located away from the aforementioned processor 91.

[0157] The present application also provides a storage medium that can store multiple instructions, which are suitable for the processor to load and execute the above-mentioned Figures 1 to 7 For the method steps of the embodiment, the specific execution process can be found in Figures 1 to 7 The specific description of the embodiment will not be repeated here.

[0158] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0159] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0160] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraint algorithm of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0161] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0162] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0163] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0164] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc.

[0165] The present invention is not limited to the above-mentioned embodiments. If various changes or modifications to the present invention do not depart from the spirit and scope of the present invention, and if these changes and modifications fall within the scope of the claims and equivalent technologies of the present invention, the present invention is also intended to include these changes and modifications.

Claims

1. A video multi-target tracking method based on query selection, characterized in that: The following steps are involved: Obtain a video sequence to be tracked and a preset multi-target tracking model, wherein the video sequence to be tracked includes a plurality of video frames, and the multi-target tracking model includes a feature extraction module, a query selection module, a decoding module, and a tracking module; the decoding module includes a plurality of stacked attention units, and the attention unit includes a self-attention layer and a cross-attention layer connected in sequence; Inputting the video sequence to be tracked into the feature extraction module, obtaining feature maps of several scales of the current frame video image, performing feature fusion on the feature maps of several scales of the current frame video image, and obtaining several multi-scale feature maps of the current frame video image; Inputting a plurality of multi-scale feature maps of the current frame video image into the query selection module, flattening and splicing the plurality of multi-scale feature maps of the current frame video image, and constructing an initial query representation of the current frame video image; Calculating the confidence of the initial query representation of the current frame video image to obtain confidence scores corresponding to several queries of the current frame video image; extracting query vectors corresponding to several candidate queries from the initial query representation according to the confidence scores, and constructing a candidate query representation of the current frame video image; Obtaining an identity tag query representation of a previous frame of video image; performing similarity calculation based on a candidate query representation of a current frame of video image and an identity tag query representation of a previous frame of video image, obtaining similarity scores between several candidate queries of the current frame of video image and several queries corresponding to identity tags of the previous frame of video image; confirming several tracking queries and identity tags corresponding to the tracking queries based on the similarity scores, and constructing an initial tracking query representation of the current frame of video image, wherein the initial tracking query representation includes initial query vectors corresponding to several tracking queries; According to the embedded vectors corresponding to the preset identity tags, the values ​​of the initial query vectors corresponding to the tracking queries in the initial tracking query representation of the current frame video image are added to the values ​​of the embedded vectors of the corresponding identity tags to obtain the added query vectors corresponding to the tracking queries of the current frame video image, and construct the tracking query representation of the current frame video image; Obtaining the target detection result of the current frame video image, performing sine and cosine encoding on the detection results of several target frames in the target detection result of the current frame video image, and obtaining the embedding vectors of several target frames of the current frame video image; adding the values ​​of the embedding vectors of several target frames of the current frame video image to the values ​​of preset learnable embedding vectors, respectively, to obtain the added embedding vectors of several target frames as the query vectors corresponding to several target queries, and constructing the target query representation of the current frame video image; splicing the tracking query representation and the target query representation of the current frame video image, and obtaining the query coding representation of the current frame video image; Inputting the query encoding representation of the current frame video image and a plurality of multi-scale feature maps into the decoding module, using the query encoding representation of the current frame video image as the input representation of the self-attention layer of the first attention unit, performing self-attention processing on the input representation of the current frame video image, and obtaining the self-attention feature representation of the current frame video image; Inputting the self-attention feature representation of the current frame video image into the cross attention layer for linear transformation to obtain the decoding parameters of the current frame video image; performing cross attention processing according to the decoding parameters of the current frame video image and a plurality of multi-scale feature maps to obtain the cross attention feature representation of the current frame video image; Inputting the cross-attention feature representation of the current frame video image to the next attention unit until the cross-attention feature representation of the current frame video image output by the last attention unit is obtained as the query decoding representation, thereby obtaining the query decoding representation of the current frame video image, wherein the query decoding representation includes a number of decoding query vectors corresponding to the queries; The query decoding representation of the current frame video image is input into the tracking module, and multi-target tracking is performed according to the decoded query vectors corresponding to several queries in the query decoding representation to obtain the predicted multi-target tracking result of the current frame video image, and the multi-target tracking is repeated for the next frame video image until the predicted multi-target tracking results of several frames of video images in the video sequence to be tracked are obtained.

2. The video multi-target tracking method based on query selection according to claim 1, characterized in that: The feature extraction module includes a plurality of stacked convolution units; The step of inputting the video sequence to be tracked into the feature extraction module, obtaining feature maps of several scales of the current frame video image, and performing feature fusion on the feature maps of several scales of the current frame video image to obtain several multi-scale feature maps of the current frame video image comprises the following steps: The current frame video image is used as the input image of the first convolution unit, and the input image is convolved to obtain a convolution feature map corresponding to the input image; the convolution feature map is downsampled, and the convolution feature map after the downsampling is input to the next convolution unit until the convolution feature map output by the last convolution unit is obtained, and the convolution feature maps output by the last several convolution units are used as the feature maps to obtain feature maps of several scales of the current frame video image; Extracting a feature map of a target scale from feature maps of several scales of a current frame video image, and performing self-attention processing on the feature map of the target scale using a self-attention mechanism to obtain a self-attention feature map of the current frame video image; Feature fusion processing is performed based on the self-attention feature map of the current frame video image and the feature maps of other scales to obtain several multi-scale feature maps of the current frame video image.

3. The video multi-target tracking method based on query selection according to claim 2 is characterized in that: The predicted multi-target tracking results include predicted tracking results of several successfully tracked queries and newly added queries, and the predicted tracking results include predicted positioning parameters and identity tags; The query decoding representation of the current frame video image is input into the tracking module, and multi-target tracking is performed according to the decoded query vectors corresponding to a plurality of queries in the query decoding representation to obtain the predicted multi-target tracking result of the current frame video image, including the steps of: Calculating category scores based on decoded query vectors corresponding to a plurality of tracking queries and target queries in a query decoding representation of a current frame of video image, and obtaining predicted category scores of a plurality of tracking queries and target queries of the current frame of video image; According to the predicted category score of the tracking query and the corresponding identity tag, a tracking query with the maximum predicted category score corresponding to several identity tags is obtained; according to the tracking query with the maximum category score and a preset tracking category score threshold, a tracking query with the maximum predicted category score corresponding to several identity tags that meet the tracking category score threshold is obtained as a successful tracking query, and several identity tags of the successful tracking queries are obtained; According to the predicted category score of the target query and a preset target category score threshold, a target query that meets the target category score threshold is obtained as a tracked new query, and identity tags of several tracked new queries are obtained; Positioning processing is performed according to the decoded query vectors corresponding to the several successful tracking queries and the newly added tracking queries of the current frame video image to obtain the predicted positioning parameters of the several successful tracking queries and the newly added tracking queries of the current frame video image.

4. The video multi-target tracking method based on query selection according to claim 3 is characterized in that: Also includes the steps: Training the multi-target tracking model; The training of the multi-target tracking model comprises the steps of: Obtaining a sample video sequence and a real multi-target tracking result of several frames of sample images in the sample video sequence, wherein the real multi-target tracking result includes real tracking results of several queries, the queries include tracking queries and target queries, and the real tracking result includes real positioning parameters and real category scores; The sample video sequence is input into the multi-target tracking model to be trained to obtain the predicted tracking results of several queries of several frames of sample images; a matching strategy is constructed according to several tracking queries of several frames of sample images and the actual tracking results and predicted tracking results of the target query by using a bipartite matching method to obtain the matching relationship results between several queries of several frames of sample images; According to the matching relationship results between several queries of several frames of sample images, several groups of query matching combinations are constructed; according to the real positioning parameters and predicted positioning parameters of the corresponding queries in the query matching combinations, loss calculation is performed to obtain the positioning loss values ​​of several frames of sample images; according to the real category scores and predicted category scores of the corresponding queries in the query matching combinations, loss calculation is performed to obtain the category loss values ​​of several frames of sample images; According to the identity labels of several queries of several frames of sample images, construct a positive query set and a negative query set of several queries of several frames of sample images; according to the positive query set and the negative query set of several queries of several frames of sample images, perform loss calculation to obtain distance loss values ​​of several frames of sample images; The multi-target tracking model to be trained is trained according to the positioning loss values, category loss values ​​and distance loss values ​​of several frames of sample images.

5. A device applied to the video multi-target tracking method based on query selection according to any one of claims 1 to 4, characterized in that: include: A data acquisition module, used to obtain a video sequence to be tracked and a preset multi-target tracking model, wherein the video sequence to be tracked includes a plurality of frames of video images, and the multi-target tracking model includes a feature extraction module, a query selection module, a decoding module and a tracking module; A feature processing module, used for inputting the video sequence to be tracked into the feature extraction module, obtaining feature maps of several scales of the current frame video image, performing feature fusion on the feature maps of several scales of the current frame video image, and obtaining several multi-scale feature maps of the current frame video image; A query initialization selection module, used to input a plurality of multi-scale feature maps of the current frame video image into the query selection module for query selection, obtain a tracking query representation and a target query representation of the current frame video image, and splice the tracking query representation and the target query representation of the current frame video image to obtain a query encoding representation of the current frame video image; A query decoding module, used for inputting the query encoding representation of the current frame video image and a plurality of multi-scale feature maps into the decoding module for decoding processing, and obtaining a query decoding representation of the current frame video image, wherein the query decoding representation includes a plurality of decoding query vectors corresponding to the queries; A multi-target tracking module is used to input the query decoding representation of the current frame video image into the tracking module, perform multi-target tracking according to the decoded query vectors corresponding to several queries in the query decoding representation, obtain the predicted multi-target tracking result of the current frame video image, and repeat the multi-target tracking for the next frame video image until the predicted multi-target tracking results of several frames of video images in the video sequence to be tracked are obtained.

6. A computer device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor; when the computer program is executed by the processor, the steps of the video multi-target tracking method based on query selection as described in any one of claims 1 to 4 are implemented.

7. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video multi-target tracking method based on query selection are implemented as described in any one of claims 1 to 4.

Images