Target tracking method and device, computer storage medium, and electronic device

Abstract

The present disclosure relates to the technical field of artificial intelligence, and provides a target tracking method, a target tracking device, a computer storage medium and an electronic device, wherein the target tracking method comprises: acquiring a language instruction to be processed and a video stream; extracting a t-1th visual feature corresponding to a t-1th frame image in the video stream, and extracting a word-level text feature and a sentence-level text feature corresponding to the language instruction; t is an integer greater than or equal to 2; performing cross-modal interweaving fusion on the t-1th visual feature and the word-level text feature to obtain a t-1th cross-modal fusion feature; decoding the t-1th cross-modal fusion feature, the t-1th correlation feature and position marker information of each tracking target in a t-2th frame image to output a t-1th decoding feature; and predicting a tracking result of each tracking target in the t-1th frame image by using the t-1th decoding feature. The present disclosure can improve the accuracy of target positioning in the video stream.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a target tracking method, a target tracking device, a computer storage medium, and an electronic device. Background Technology

[0002] Instruction-based multi-object tracking is an emerging language-visual cross-modal understanding task that aims to locate targets of interest in videos and maintain their identity based on linguistic instructions. Compared to traditional multi-object tracking tasks, instruction-based multi-object tracking requires algorithms to accurately understand the semantic information of linguistic instructions and accurately locate the target of interest in video frames.

[0003] In related technologies, the TransMOT algorithm is generally used to perform the above-mentioned multi-target tracking task. However, the above scheme completes the cross-modal feature fusion process before the encoder-decoder, which fails to make full use of the advantages of the encoder-decoder network, resulting in the inability to accurately identify and locate the target.

[0004] Therefore, there is an urgent need in this field to develop a new target tracking method and device.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure. Summary of the Invention

[0006] The purpose of this disclosure is to provide a target tracking method, target tracking device, computer storage medium, and electronic device, thereby overcoming, to at least a certain extent, the technical problem of inaccurate target identification and positioning caused by limitations in related technologies.

[0007] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0008] According to a first aspect of this disclosure, a target tracking method is provided, comprising:

[0009] Acquire language instructions and video streams to be processed; the language instructions are used to instruct the location of at least one tracked target, and the video stream contains multiple frames of images;

[0010] Extract the (t-1)th visual feature corresponding to the (t-1)th frame image in the video stream, and extract the word-level text features and sentence-level text features corresponding to the language instruction; t is an integer greater than or equal to 2;

[0011] The (t-1)th visual feature and the word-level text feature are fused across modalities to obtain the (t-1)th cross-modal fused feature;

[0012] The (t-1)th cross-modal fusion feature, the (t-1)th associated feature, and the position marker information of each tracked target in the (t-2)th frame image are decoded to output the (t-1)th decoded feature; the (t-1)th associated feature is determined based on the sentence-level text feature, the (t-2)th decoded feature corresponding to the (t-2)th frame image, and the pre-learned decoding auxiliary features;

[0013] The tracking result of each tracked target in the (t-1)th frame image is predicted using the (t-1)th decoding feature; the tracking result includes content recognition information and location marking information of each tracked target in the (t-1)th frame image.

[0014] In an exemplary embodiment of this disclosure, after extracting the word-level text features and sentence-level text features corresponding to the language instructions, the method further includes:

[0015] The (t-1)th visual feature, the word-level text feature, and the sentence-level text feature are processed into the same dimension through a fully connected network layer.

[0016] In an exemplary embodiment of this disclosure, the step of performing cross-modal interleaving and fusion of the (t-1)th visual feature and the word-level text feature to obtain the (t-1)th cross-modal fused feature includes:

[0017] The (t-1)th visual feature is enhanced based on a deformable attention mechanism to obtain the (t-1)th enhanced visual feature.

[0018] The (t-1)th enhanced visual feature and the word-level text feature are fused together using a cross-modal interleaving encoder to obtain the (t-1)th cross-modal fused feature.

[0019] In an exemplary embodiment of this disclosure, the cross-modal interleaved encoder comprises l coding layers, where l is an integer greater than 1;

[0020] The method of performing cross-modal interleaving and fusion of the (t-1)th enhanced visual feature and the word-level text feature based on the cross-modal interleaving encoder to obtain the (t-1)th cross-modal fused feature includes:

[0021] The first cross-modal interleaving and fusion of the (t-1)th enhanced visual feature and the word-level text feature are performed through the first encoding layer to obtain the first visual text fusion feature and the first text visual fusion feature.

[0022] The first visual text fusion feature and the first text visual fusion feature are subjected to a second cross-modal interleaving fusion through a second encoding layer to obtain the second visual text fusion feature and the second text visual fusion feature.

[0023] After obtaining the l-th visual-text fusion feature and the l-th text-visual fusion feature output by the l-th coding layer, the l-th visual-text fusion feature is determined as the (t-1)-th cross-modal fusion feature.

[0024] In an exemplary embodiment of this disclosure, the first cross-modal interleaving and fusion of the (t-1)th enhanced visual feature and the word-level text feature through the first encoding layer to obtain the first visual-text fusion feature and the first text-visual fusion feature includes:

[0025] The first query vector corresponding to the word-level text feature is calculated through the first encoding layer, and the second query vector corresponding to the (t-1)th enhanced visual feature is calculated.

[0026] Calculate the shared attention vector based on the first query vector, the second query vector, and the dimension value of the first query vector or the second query vector;

[0027] Calculate the first value vector corresponding to the (t-1)th enhanced visual feature, and determine the first visual text fusion feature based on the first value vector, the shared attention vector, and the preset first weight matrix;

[0028] Calculate the second value vector corresponding to the word-level text feature, and determine the first text visual fusion feature based on the second value vector, the shared attention vector, and the preset second weight matrix.

[0029] In an exemplary embodiment of this disclosure, when t equals 2, the (t-1)th associated feature is determined in the following manner:

[0030] After repeating the sentence-level text feature a specified number of times, it is superimposed with the pre-learned decoding auxiliary feature to obtain the (t-1)th associated feature.

[0031] In an exemplary embodiment of this disclosure, when t is greater than 2, the (t-1)th associated feature is determined in the following manner:

[0032] The (t-2)th decoded feature corresponding to the (t-2)th frame image is concatenated with the pre-learned decoding auxiliary feature to obtain the concatenated feature;

[0033] The sentence-level text feature is repeated a specified number of times and then superimposed with the concatenated feature to obtain the (t-1)th associated feature.

[0034] In an exemplary embodiment of this disclosure, the position marking information of each tracked target in the (t-2)th frame image includes the position parameters of the position box of each tracked target in the (t-2)th frame image.

[0035] In an exemplary embodiment of this disclosure, after predicting the tracking result of each of the tracked targets in the (t-1)th frame image using the (t-1)th decoded feature, the method further includes:

[0036] Extract the t-th visual feature corresponding to the t-th frame image in the video stream;

[0037] The t-th visual feature and the word-level text feature are interleaved and fused across modally to obtain the t-th cross-modal fused feature;

[0038] The t-th cross-modal fusion feature, the t-th association feature, and the position marker information of each tracked target in the (t-1)-th frame image are decoded to output the t-th decoded feature; the t-th association feature is determined based on the sentence-level text feature, the (t-1)-th decoded feature corresponding to the (t-1)-th frame image, and the pre-learned decoding auxiliary features;

[0039] The tracking result of each tracked target in the t-th frame image is predicted using the t-th decoding feature; the tracking result includes content recognition information and location marking information of each tracked target in the t-th frame image.

[0040] According to a second aspect of this disclosure, a target tracking device is provided, comprising:

[0041] An acquisition module is used to acquire language instructions and video streams to be processed; the language instructions are used to indicate the location of at least one tracking target, and the video stream contains multiple frames of images.

[0042] The feature extraction module is used to extract the (t-1)th visual feature corresponding to the (t-1)th frame image in the video stream, and to extract the word-level text features and sentence-level text features corresponding to the language instructions; t is an integer greater than or equal to 2;

[0043] The cross-modal fusion module is used to perform cross-modal interleaving and fusion of the (t-1)th visual feature and the word-level text feature to obtain the (t-1)th cross-modal fused feature;

[0044] The decoding feature output module is used to decode the (t-1)th cross-modal fusion feature, the (t-1)th associated feature, and the position marker information of each tracked target in the (t-2)th frame image, and output the (t-1)th decoded feature; when t equals 2, the (t-1)th associated feature is determined based on the sentence-level text features and the pre-learned decoding auxiliary features; when t is greater than 2, the (t-1)th associated feature is determined based on the (t-2)th decoded feature and the (t-2)th associated feature of the (t-2)th frame image.

[0045] The result output module is used to predict the tracking result of each tracking target in the (t-1)th frame image using the (t-1)th decoded feature; the tracking result includes content recognition information and location marking information of each tracking target in the (t-1)th frame image.

[0046] According to a third aspect of this disclosure, a computer storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the target tracking method described in the first aspect above.

[0047] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the target tracking method described in the first aspect by executing the executable instructions.

[0048] As can be seen from the above technical solutions, the target tracking method, target tracking device, computer storage medium, and electronic device in the exemplary embodiments of this disclosure have at least the following advantages and positive effects:

[0049] In some embodiments of the present disclosure, the technical solutions provided include, on the one hand, obtaining the (t-1)th cross-modal fusion feature by performing cross-modal interleaving and fusion of the (t-1)th visual feature and the word-level text feature, which can combine linguistic information with visual information to improve the accuracy and robustness of target tracking; furthermore, by decoding the (t-1)th cross-modal fusion feature, the (t-1)th association feature, and the position marker information of each tracked target in the (t-2)th frame image, the (t-1)th decoded feature is output, which can establish a more stable and accurate temporal association relationship between the tracked targets, thereby making the output (t-1)th decoded feature more detailed and having stronger feature representation capabilities, thus providing an effective guarantee for the accuracy of the subsequent output tracking results; on the other hand, predicting the tracking result of each tracked target in the (t-1)th frame image through the above-mentioned (t-1)th decoded feature can output a more accurate tracking result based on the aforementioned better feature representation, thereby achieving effective and accurate tracking of the target.

[0050] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0051] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0052] Figure 1 A flowchart illustrating the target tracking method in an embodiment of this disclosure is shown;

[0053] Figure 2 This diagram illustrates the process of how to perform cross-modal interleaving and fusion of the (t-1)th visual feature and the word-level text feature in an embodiment of this disclosure to obtain the (t-1)th cross-modal fused feature;

[0054] Figure 3 This illustration shows a flowchart of how to perform cross-modal interleaving and fusion of the (t-1)th enhanced visual feature and the word-level text feature through a cross-modal interleaving encoder in an embodiment of this disclosure to obtain the (t-1)th cross-modal fused feature;

[0055] Figure 4 This illustration shows a flowchart of how, in an embodiment of the present disclosure, the first visual text fusion feature and the first text visual fusion feature are obtained by performing a first cross-modal interleaving and fusion of the (t-1)th enhanced visual feature and the word-level text feature through the first encoding layer.

[0056] Figure 5 This illustration shows a flowchart of how to determine the t-th associated feature when t is greater than 2 in an embodiment of this disclosure.

[0057] Figure 6 This diagram illustrates a flowchart of how the tracking result of each tracked target in the t-th frame image is determined in an embodiment of this disclosure.

[0058] Figure 7 This diagram illustrates the input and output information of the decoder between any two frames in an embodiment of this disclosure.

[0059] Figure 8 This diagram illustrates the overall flow of the target tracking method in an embodiment of this disclosure.

[0060] Figure 9 This diagram illustrates the structure of the target tracking device in an exemplary embodiment of the present disclosure.

[0061] Figure 10 A schematic diagram of the structure of an electronic device in an exemplary embodiment of this disclosure is shown. Detailed Implementation

[0062] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0063] The terms “a,” “an,” “the,” and “the” are used in this specification to indicate the presence of one or more elements / components / etc.; the terms “including” and “having” are used to indicate an open-ended inclusion and to mean that there may be other elements / components / etc. in addition to the listed elements / components / etc.; the terms “first” and “second” are used only as markings and are not a limitation on the number of objects.

[0064] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0065] Instruction-based multi-object tracking is an emerging language-visual cross-modal understanding task that aims to locate targets of interest in videos based on verbal instructions and maintain their identity (maintaining identity: in instruction-based multi-object tracking, this means correctly identifying and tracking targets associated with verbal instructions across consecutive frames of the video. This task requires not only that the system can locate targets in the video based on verbal instructions, but also that these targets maintain consistent identity throughout the video). Compared to traditional video multi-object tracking, instruction-based multi-object tracking offers more flexible interaction methods and has broad application prospects in scenarios such as smart logistics, autonomous robot navigation, human-computer interaction, and autonomous driving.

[0066] Compared to traditional multi-object tracking tasks, instruction-based multi-object tracking requires algorithms to accurately understand the semantic information of language instructions and accurately locate the target of interest in video frames. Compared to other existing video instruction tasks, it faces greater challenges, such as dynamic changes in the number of targets and significant temporal dynamic changes, which pose a significant challenge to accurately identifying the target indicated by the language instructions.

[0067] The existing method TransMOT is directly derived from MOTR, which is designed for traditional multi-object tracking tasks. This approach has at least the following drawbacks:

[0068] First, it completes cross-modal feature fusion (such as the fusion of images, language, and text) only before the encoder-decoder, without fully utilizing the structural advantages of Transformer, which can only result in suboptimal feature fusion.

[0069] Second, during the inter-frame update tracking query, the algorithm ignores the use of prior information about the target content context (i.e., the re-identification information of the content) and prior information about the spatial location, which causes the algorithm to fail in complex scenarios such as rapid target movement and occlusion.

[0070] In the embodiments of this disclosure, a target tracking method is first provided, which at least to some extent overcomes the shortcomings of related technologies in accurately identifying and locating targets.

[0071] Figure 1 The diagram shows a flowchart of a target tracking method in an embodiment of this disclosure. The subject executing the target tracking method may be a server that tracks the target.

[0072] refer to Figure 1 A target tracking method according to an embodiment of the present disclosure includes the following steps:

[0073] Step S110: Obtain the language instructions and video stream to be processed; the language instructions are used to indicate the location of at least one tracking target, and the video stream contains multiple frames of images;

[0074] Step S120: Extract the (t-1)th visual feature corresponding to the (t-1)th frame image in the video stream, and extract the word-level text features and sentence-level text features corresponding to the language instructions; t is an integer greater than or equal to 2;

[0075] Step S130: Perform cross-modal interleaving and fusion of the (t-1)th visual feature and the word-level text feature to obtain the (t-1)th cross-modal fused feature;

[0076] Step S140: Decode the (t-1)th cross-modal fusion feature, the (t-1)th associated feature, and the position marker information of each tracked target in the (t-2)th frame image, and output the (t-1)th decoded feature;

[0077] Step S150: Use the (t-1)th decoded feature to predict the tracking result of each tracked target in the (t-1)th frame image; the tracking result includes the content recognition information and location marking information of each tracked target in the (t-1)th frame image.

[0078] exist Figure 1 In the technical solution provided by the illustrated embodiment, on the one hand, by performing cross-modal interleaving and fusion of the (t-1)th visual feature and the word-level text feature, the (t-1)th cross-modal fusion feature is obtained, which can combine linguistic information with visual information to improve the accuracy and robustness of target tracking. Furthermore, by decoding the (t-1)th cross-modal fusion feature, the (t-1)th association feature, and the position marker information of each tracked target in the (t-2)th frame image, the (t-1)th decoded feature is output, which can establish a more stable and accurate temporal association relationship of the tracked target, thereby making the output (t-1)th decoded feature more detailed and having stronger feature representation capabilities, thus providing an effective guarantee for the accuracy of the subsequent output tracking results. On the other hand, by predicting the tracking result of each tracked target in the (t-1)th frame image through the above-mentioned (t-1)th decoded feature, a more accurate tracking result can be output based on the aforementioned better feature representation, thereby achieving effective and accurate tracking of the target.

[0079] The following are Figure 1 The specific implementation process of each step in the process will be explained in detail:

[0080] In step S110, the language instructions and video stream to be processed are obtained.

[0081] In this step, the language instructions to be processed can be obtained. And video stream H.

[0082] The aforementioned language instructions contain N words and are used to indicate the location of at least one tracking target (e.g., a car, a cat, a dog, etc., which can be set according to the actual situation; this disclosure does not impose any special limitations on this), for example, P tracking targets. The aforementioned video stream may contain multiple frames of images.

[0083] In step S120, the (t-1)th visual feature corresponding to the (t-1)th frame image in the video stream is extracted, and the word-level text features and sentence-level text features corresponding to the language instructions are extracted; t is an integer greater than or equal to 2.

[0084] In this step, a visual encoder can be used to extract the (t-1)th visual feature (e.g., F) corresponding to the (t-1)th frame image in the video stream. i ), and, using a text encoder to extract word-level text features corresponding to the above language instructions (e.g., F w ) and sentence-level text features (e.g., F s ).

[0085] A visual encoder's role is to convert visual information into a form that computers can understand and process, typically by converting images or videos into digital signals or vector representations. Encoders convert information such as color, shape, texture, and motion in images or videos into digital feature vectors, enabling computers to analyze and understand visual information. In tasks such as object detection, tracking, and recognition, encoders can convert input image or video frames into fixed-dimensional feature vectors, which can then be used for subsequent tasks such as classification, matching, and recognition.

[0086] The role of a text encoder is to convert text information into a vector representation that a computer can understand and process. Text encoders can extract text features from text descriptions, thus playing an important role in NLP tasks.

[0087] Among them, word-level text features refer to features directly related to words, used to describe the distribution and occurrence of words in the text. Word-level text features focus on the attributes of words; while sentence-level text features refer to the overall features of the sentence in which the words are located. These features can cover the semantic information of the entire sentence, such as: syntactic structure, comment length, and whether it contains emojis, to describe the structure and meaning of sentences in the text. Sentence-level text features focus on the structure and semantic information of sentences.

[0088] It should be noted that after extracting the word-level and sentence-level text features corresponding to the language instructions, this disclosure can use a fully connected network layer to integrate the (t-1)th visual feature and the word-level text feature F. w and sentence-level text features F s Process them to the same dimension d.

[0089] In step S130, the (t-1)th visual feature and the word-level text feature are fused across modally to obtain the (t-1)th cross-modal fused feature.

[0090] In this step, refer to Figure 2 , Figure 2 This illustration shows a flowchart of how to perform cross-modal interleaving and fusion of the (t-1)th visual feature and word-level text features to obtain the (t-1)th cross-modal fused feature in an embodiment of this disclosure, including steps S201-S202:

[0091] In step S201, the (t-1)th visual feature is enhanced based on the deformable attention mechanism to obtain the (t-1)th enhanced visual feature.

[0092] In this step, the (t-1)th visual feature can be enhanced based on a deformable attention mechanism to obtain the (t-1)th enhanced visual feature (e.g.: Feature enhancement processing refers to the use of methods such as feature selection, feature transformation, feature combination, feature dimensionality reduction, and feature generation to improve or enhance the recognition effect of features.

[0093] In step S202, the (t-1)th enhanced visual feature and the word-level text feature are fused across modally using a cross-modal interleaving encoder to obtain the (t-1)th cross-modal fused feature.

[0094] In this step, the aforementioned cross-modal interleaved encoder may include l (l is an integer greater than 1) coding layers, thus, referring to Figure 3 , Figure 3 This illustration shows a flowchart of how to perform cross-modal interleaving and fusion of the (t-1)th enhanced visual feature and the word-level text feature using a cross-modal interleaving encoder in an embodiment of this disclosure to obtain the (t-1)th cross-modal fused feature, including steps S301-S303:

[0095] In step S301, the first cross-modal interleaving and fusion of the (t-1)th enhanced visual feature and the word-level text feature is performed through the first encoding layer to obtain the first visual-text fusion feature and the first text-visual fusion feature.

[0096] In this step, refer to Figure 4 , Figure 4 This illustration shows a flowchart of how, in an embodiment of this disclosure, the (t-1)th enhanced visual feature and the word-level text feature are first interleaved and fused across the first encoding layer to obtain the first visual-text fusion feature and the first text-visual fusion feature, including steps S401-S404:

[0097] In step S401, the first query vector corresponding to the word-level text features is calculated through the first encoding layer, and the second query vector corresponding to the (t-1)th enhanced visual feature is calculated.

[0098] In this step, the word-level text features F can be calculated based on the following formula 1. w The corresponding first query vector

[0099]

[0100] in, W represents the first query vector mentioned above. (Q,L) This represents the weight matrix learned during the training process.

[0101] The enhanced visual feature mentioned above can be calculated based on the following formula 2. The corresponding second query vector

[0102]

[0103] in, W represents the second query vector mentioned above. (Q,I) This represents the weight matrix learned during training.

[0104] In step S402, a shared attention vector is calculated based on the dimension values ​​of the first query vector, the second query vector, or the first query vector or the second query vector.

[0105] In this step, the shared attention vector X can be calculated based on the following formula 2:

[0106]

[0107] Where X represents the shared attention vector mentioned above, and d represents the dimension value of the first query vector / second query vector mentioned above.

[0108] In step S403, the first value vector corresponding to the (t-1)th enhanced visual feature is calculated, and the first visual text fusion feature is determined based on the first value vector, the shared attention vector, and the preset first weight matrix.

[0109] In this step, the first value vector corresponding to the (t-1)th enhanced visual feature can be calculated based on the following formula 4.

[0110]

[0111] Among them, the above W represents the first value vector mentioned above. (V,I) This represents the weight matrix learned during training.

[0112] The first visual text fusion feature F can be calculated based on the following formula 5. iw :

[0113]

[0114] Among them, F iw The first visual text fusion feature mentioned above is represented by X, which represents the shared attention vector mentioned above, and W represents the first visual text fusion feature mentioned above. (()*,I) This represents the weight matrix learned during the training process.

[0115] In step S404, the second value vector corresponding to the word-level text features is calculated, and the first text visual fusion feature is determined based on the second value vector, the shared attention vector, and the preset second weight matrix.

[0116] In this step, the second value vector can be calculated based on the following formula 6.

[0117]

[0118] in, W represents the second value vector mentioned above. (V,L) This represents the weight matrix learned during the training process.

[0119] The first text visual fusion feature F can be calculated based on the following formula 7. wi :

[0120]

[0121] Among them, F wi Representing the first text visual fusion feature mentioned above, while W (()*,L) This represents the weight matrix learned during the training process.

[0122] Next, refer to Figure 3 In step S302, the first visual text fusion feature and the first text visual fusion feature are subjected to a second cross-modal interleaving fusion through the second encoding layer to obtain the second visual text fusion feature and the second text visual fusion feature.

[0123] In this step, the first visual-text fusion feature and the first text-visual fusion feature output from the first encoding layer can be used as the input to the second encoding layer. Then, the second encoding layer can be based on the above... Figure 4 A similar processing procedure is used to process the first visual text fusion feature and the first text visual fusion feature, and output the second visual text fusion feature and the second text visual fusion feature.

[0124] In step S303, after obtaining the l-th visual-text fusion feature and the l-th text-visual fusion feature output by the l-th coding layer, the l-th visual-text fusion feature is determined as the (t-1)-th cross-modal fusion feature.

[0125] In this step, after the l-th encoding layer outputs the l-th visual-text fusion feature and the l-th text-visual fusion feature, the l-th text-visual fusion feature can be filtered out, and only the l-th visual-text fusion feature is determined as the (t-1)-th cross-modal fusion feature (e.g., M). *-1 ).

[0126] Next, refer to Figure 1 In step S140, the (t-1)th cross-modal fusion feature, the (t-1)th associated feature, and the position marker information of each tracked target in the (t-2)th frame image are decoded, and the (t-1)th decoded feature is output.

[0127] In this step, for example, the position marker information of each tracked target in the (t-2)th frame image refers to the position parameter b of the bounding box of each tracked target in the (t-2)th frame image. *- / For example, the decoder can be constructed using a two-layer feedforward neural network F. 0(s The above position parameters are processed to obtain the position embedding features of each tracked target in the (t-2)th frame image. Then, the (t-1)th cross-modal fusion feature, the (t-1)th associated feature, and the position embedding feature of each tracked target in the (t-2)th frame image are decoded to obtain the aforementioned (t-1)th decoded feature.

[0128] The following describes the (t-1)th associated feature Z. *-1 The process of determining [the value] will be explained as follows:

[0129] When t equals 2, t-1 = 1, then the (t-1)th frame is the first frame of the aforementioned video stream. Therefore, this disclosure can determine the first associated feature in the following way: [The sentence-level text feature F is then used in the original text]. s Repeat a specified number of times (e.g., P+N) d After (times), it is compared with the pre-learned decoding auxiliary features (e.g., Z). det The features are superimposed to obtain the (t-1)th associated feature (Z). *-1 ).

[0130] When t is greater than 2, for example, taking t as 3, then t-1 = 2, and the (t-1)th frame is the second frame of the video stream. Therefore, this disclosure can determine the (t-1)th (i.e., 2nd) associated feature in the following way:

[0131] refer to Figure 5 , Figure 5 This illustration shows a flowchart of how to determine the t-th associated feature when t is greater than 2 in an embodiment of this disclosure, including steps S501-S502:

[0132] In step S501, the (t-2)th decoded feature corresponding to the (t-2)th frame image is concatenated with the pre-learned decoded auxiliary feature to obtain the concatenated feature.

[0133] In this step, the (t-2)th decoded feature corresponding to the (t-2)th frame image above and the pre-learned decoded auxiliary features can be concatenated based on the following formula 8 to obtain the concatenated feature:

[0134] Z = concat 6Z det ,e *- / 8Formula 8

[0135] Where represents the (t-2)th decoding feature mentioned above, and Z represents the splicing feature mentioned above.

[0136] In step S502, the sentence-level text features are repeated a specified number of times and then superimposed with the concatenated features to obtain the (t-1)th associated feature.

[0137] In this step, the (t-1)th associated feature can be determined based on the following formula 9:

[0138] Z *-1 = repeat(F s Formula 9 (+Z)

[0139] Among them, Z *-1 This is the (t-1)th associated feature mentioned above.

[0140] After obtaining the (t-1)th cross-modal fusion feature mentioned above, the (t-1)th cross-modal fusion feature M can be... *-1 The (t-1)th associated feature Z *-1 and the location marker information b of each tracked target in the (t-2)th frame image. *- / The input is to the decoder, and the decoder outputs the (t-1)th decoded feature e. *-1 This process can be expressed as the following formula 10:

[0141] e *-1 =Decoder 6M *-1 Z *-1 b *- / 8 formula 10

[0142] Next, refer to Figure 1 In step S150, the tracking result of each tracked target in the (t-1)th frame image is predicted using the (t-1)th decoded feature.

[0143] In this step, the aforementioned (t-1)th decoded feature can be input into the prediction head to predict the tracking result of each tracked target in the (t-1)th frame image using the aforementioned (t-1)th decoded feature.

[0144] The tracking results include content recognition information and confidence level of each tracked target in the (t-1)th frame image, as well as location marking information and confidence level of each tracked target in the (t-1)th frame image.

[0145] It should be noted that when predicting the position parameters of the bounding box of each tracked target in frame t-1, the position parameters b of the bounding box of each tracked target in frame t-2 can be used. *- / As initial value Furthermore, the offset of the bounding box of each tracked target is predicted using the (t-1)th decoded feature, and then, based on the above initial values... Based on this, position offset is performed to determine the position parameters of the location box of each tracked target in the (t-1)th frame.

[0146] The aforementioned content recognition information can be the re-identification information of each tracked target in the (t-1)th frame image. Re-identification information usually refers to information that can help the system re-identify the same target in different frames or different scenes, such as the target's appearance, texture, and shape. Simply put, even if the target's position and angle change, as long as its re-identification information remains unchanged, the system can still identify that it is the same target.

[0147] The aforementioned location marker information can be the parameter information of the bounding box of each tracked target in the (t-1)th frame image (b *-1 Specifically, the location box can contain four dimensions: x, y, w, and h, where (x, y) are the coordinates of the top-left corner of the location box, w is the width of the location box, and h is the height of the location box.

[0148] After determining the tracking result of each tracked target in the (t-1)th frame image, refer to Figure 6 , Figure 6 This illustration shows a flowchart of how to determine the tracking result of each tracked target in the t-th frame image in an embodiment of this disclosure, including steps S601-S604:

[0149] In step S601, the t-th visual feature corresponding to the t-th frame image in the video stream is extracted.

[0150] In this step, you can refer to the relevant explanation of step S120 to use the visual encoder to extract the t-th visual feature corresponding to the (t-1)-th frame image in the video stream.

[0151] In step S602, the t-th visual feature and the word-level text feature are cross-modal interleaved and fused to obtain the t-th cross-modal fused feature.

[0152] In this step, referring to the explanation in step S130, the t-th visual feature and the aforementioned word-level text features are cross-modal interleaved and fused to obtain the t-th cross-modal fused feature (e.g., M). * ).

[0153] In step S603, the t-th cross-modal fusion feature, the t-th associated feature, and the position marker information of each tracked target in the (t-1)-th frame image are decoded to output the t-th decoded feature.

[0154] In this step, referring to the relevant explanation of step S140 above, the t-th cross-modal fusion feature M can be... * The t-th associated feature Z * and the position marker information b of each tracked target in the (t-1)th frame image. *-1 The input to the decoder is the t-th decoded feature e, which is output by the decoder. * .

[0155] Among them, the t-th associated feature Z * It can be based on the (t-1)th decoded feature e corresponding to the (t-1)th frame image. *-1 and the (t-1)th associated feature Z *-1 The jointly determined, exemplary, can be calculated based on the following formula 11 for the t-th associated feature Z. * :

[0156] Z * =ψ(e *-1 Z *-1 ) Formula 11

[0157] It should be noted that ψ represents the update module, and the function of ψ is to update the decoded feature e corresponding to the (t-1)th frame of the image. *-1 and the pre-learned decoding auxiliary features Z det The features are concatenated to obtain the second concatenation feature, and the sentence-level text features F are then combined. s After repeating the process a specified number of times, it is superimposed on the second concatenation feature mentioned above to obtain the t-th associated feature Z. * .

[0158] In step S604, the tracking result of each tracked target in the t-th frame image is predicted using the t-th decoded feature.

[0159] In this step, the relevant explanation of step S150 above can be referred to, and the tracking result of each tracked target in the t-th frame image can be predicted using the above-mentioned t-th decoding feature. The tracking result includes the content recognition information and position marking information of each tracked target in the t-th frame image.

[0160] refer to Figure 7 , Figure 7 This diagram illustrates the input and output information of the decoder between any two frames in an embodiment of this disclosure, such as... Figure 7 As shown:

[0161] The left side shows the decoder corresponding to the (t-1)th frame, which contains L decoding layers. The input of the decoder includes: the (t-1)th cross-modal fusion feature M. *-1 b) Location marker information for each tracked target in the (t-2)th frame. *- / (Right now ) and the (t-1)th associated feature Z *-1 After passing through the self-attention layer and the deformable attention layer in the decoder, the output of the decoder is the (t-1)th decoded feature e. *-1 ;

[0162] The right side shows the decoder corresponding to frame t, which contains L decoding layers. The decoder's input includes: the t-th cross-modal fusion feature M. * b) is the location marker information of each tracked target in the (t-1)th frame. *-1 (Right now ) and the t-th associated feature Z * (Z * It is based on Z *-1 and e *-1 After the update, the decoder outputs the t-th decoded feature e after passing through the self-attention layer and deformable attention layer in the decoder. * .

[0163] refer to Figure 8 , Figure 8 This diagram illustrates the overall flow of the target tracking method in an embodiment of this disclosure, as follows: Figure 8 As shown:

[0164] When the given video stream is a road surveillance video of a certain road, and the given language instruction is "Right cars which are parking", this disclosure can extract the visual features corresponding to each frame of the image through a video encoder, and at the same time, extract the word-level text features and sentence-level text features corresponding to the above language instruction through a text encoder;

[0165] Furthermore, for each frame of image, visual features and word-level text features can be input into a cross-modal interleaved encoder to output cross-modal fused features;

[0166] Then, the cross-modal fusion features, the associated features determined based on the tracking results of the previous frame, and the position marker information of each tracked object in the previous frame can be input into the language-guided decoder to output the decoded features.

[0167] Then, the tracking result of the target in each frame of the image can be predicted based on the decoded features (i.e., content recognition information and...). Figure 8 The target bounding box shown in the figure can be used to calculate the next associated feature through the QUM module, so as to serve as the input feature of the decoder when processing the next frame of the image, and to assist in better target tracking in the next frame, so as to better adapt to complex scenes such as fast target movement and occlusion.

[0168] Based on the above technical solutions, this disclosure has at least the following technical effects:

[0169] First, cross-modal feature fusion: Feature fusion is performed in both the encoder and decoder: visual and word-level text features are fused through cross-modal interleaving encoder in the encoding stage, and sentence-level text features are used to guide the decoding process in the decoder stage to achieve more accurate target localization;

[0170] Second, during the inter-frame propagation of track query, the associated features of the input part of the next frame decoder are updated based on the input and output information of the previous frame decoder. This enables better inter-frame connection to adapt to changes in target shape and position, improves the consistency of predicted trajectory, and achieves more accurate tracking.

[0171] This disclosure also provides a target tracking device. Figure 9 This diagram illustrates the structure of a target tracking device in an exemplary embodiment of this disclosure; as shown below. Figure 9 As shown, the target tracking device 900 may include an acquisition module 910, a feature extraction module 920, a cross-modal fusion module 930, a decoded feature output module 940, and a result output module 950. Wherein:

[0172] The acquisition module 910 is used to acquire language instructions and video streams to be processed; the language instructions are used to indicate the location of at least one tracking target, and the video stream contains multiple frames of images.

[0173] The feature extraction module 920 is used to extract the (t-1)th visual feature corresponding to the (t-1)th frame image in the video stream, and to extract the word-level text features and sentence-level text features corresponding to the language instructions; t is an integer greater than or equal to 2;

[0174] The cross-modal fusion module 930 is used to perform cross-modal interleaving and fusion of the (t-1)th visual feature and the word-level text feature to obtain the (t-1)th cross-modal fused feature;

[0175] The decoding feature output module 940 is used to decode the (t-1)th cross-modal fusion feature, the (t-1)th associated feature, and the position marker information of each tracked target in the (t-2)th frame image, and output the (t-1)th decoded feature; when t equals 2, the (t-1)th associated feature is determined based on the sentence-level text feature and the pre-learned decoding auxiliary feature; when t is greater than 2, the (t-1)th associated feature is determined based on the (t-2)th decoded feature and the (t-2)th associated feature of the (t-2)th frame image;

[0176] The result output module 950 is used to predict the tracking result of each tracked target in the (t-1)th frame image using the (t-1)th decoded feature; the tracking result includes content recognition information and location marker information of each tracked target in the (t-1)th frame image. In an exemplary embodiment of this disclosure, after extracting the word-level text features and sentence-level text features corresponding to the language instruction, the feature extraction module 920 is configured to:

[0177] The (t-1)th visual feature, the word-level text feature, and the sentence-level text feature are processed into the same dimension through a fully connected network layer.

[0178] In an exemplary embodiment of this disclosure, the cross-modal fusion module 930 performs cross-modal interleaving fusion on the (t-1)th visual feature and the word-level text feature to obtain the (t-1)th cross-modal fused feature, including:

[0179] The (t-1)th visual feature is enhanced based on a deformable attention mechanism to obtain the (t-1)th enhanced visual feature.

[0180] The (t-1)th enhanced visual feature and the word-level text feature are fused together using a cross-modal interleaving encoder to obtain the (t-1)th cross-modal fused feature.

[0181] In an exemplary embodiment of this disclosure, the cross-modal interleaved encoder comprises l coding layers, where l is an integer greater than 1;

[0182] The cross-modal fusion module 930 performs cross-modal interleaving fusion of the (t-1)th enhanced visual feature and the word-level text feature based on the cross-modal interleaving encoder to obtain the (t-1)th cross-modal fusion feature, including:

[0183] The first cross-modal interleaving and fusion of the (t-1)th enhanced visual feature and the word-level text feature are performed through the first encoding layer to obtain the first visual text fusion feature and the first text visual fusion feature.

[0184] The first visual text fusion feature and the first text visual fusion feature are subjected to a second cross-modal interleaving fusion through a second encoding layer to obtain the second visual text fusion feature and the second text visual fusion feature.

[0185] After obtaining the l-th visual-text fusion feature and the l-th text-visual fusion feature output by the l-th coding layer, the l-th visual-text fusion feature is determined as the (t-1)-th cross-modal fusion feature.

[0186] In an exemplary embodiment of this disclosure, the cross-modal fusion module 930 performs a first cross-modal interleaving fusion of the (t-1)th enhanced visual feature and the word-level text feature through a first encoding layer to obtain a first visual-text fusion feature and a first text-visual fusion feature, including:

[0187] The first query vector corresponding to the word-level text feature is calculated through the first encoding layer, and the second query vector corresponding to the (t-1)th enhanced visual feature is calculated.

[0188] Calculate the shared attention vector based on the first query vector, the second query vector, and the dimension value of the first query vector or the second query vector;

[0189] Calculate the first value vector corresponding to the (t-1)th enhanced visual feature, and determine the first visual text fusion feature based on the first value vector, the shared attention vector, and the preset first weight matrix;

[0190] Calculate the second value vector corresponding to the word-level text feature, and determine the first text visual fusion feature based on the second value vector, the shared attention vector, and the preset second weight matrix.

[0191] In an exemplary embodiment of this disclosure, when t equals 2, the (t-1)th associated feature is determined in the following manner:

[0192] After repeating the sentence-level text feature a specified number of times, it is superimposed with the pre-learned decoding auxiliary feature to obtain the (t-1)th associated feature.

[0193] In an exemplary embodiment of this disclosure, when t is greater than 2, the (t-1)th associated feature is determined in the following manner:

[0194] The (t-2)th decoded feature corresponding to the (t-2)th frame image is concatenated with the pre-learned decoding auxiliary feature to obtain the concatenated feature;

[0195] The sentence-level text feature is repeated a specified number of times and then superimposed with the concatenated feature to obtain the (t-1)th associated feature.

[0196] In an exemplary embodiment of this disclosure, the position marking information of each tracked target in the (t-2)th frame image includes the position parameters of the position box of each tracked target in the (t-2)th frame image.

[0197] In an exemplary embodiment of this disclosure, after predicting the tracking result of each of the tracked targets in the (t-1)th frame image using the (t-1)th decoded feature, the result output module 950 is configured to:

[0198] Extract the t-th visual feature corresponding to the t-th frame image in the video stream;

[0199] The t-th visual feature and the word-level text feature are interleaved and fused across modally to obtain the t-th cross-modal fused feature;

[0200] The t-th cross-modal fusion feature, the t-th association feature, and the position marker information of each tracked target in the (t-1)-th frame image are decoded to output the t-th decoded feature; the t-th association feature is determined based on the sentence-level text feature, the (t-1)-th decoded feature corresponding to the (t-1)-th frame image, and the pre-learned decoding auxiliary features;

[0201] The tracking result of each tracked target in the t-th frame image is predicted using the t-th decoding feature; the tracking result includes content recognition information and location marking information of each tracked target in the t-th frame image.

[0202] The specific details of each module in the target tracking device have been described in detail in the corresponding target tracking method, so they will not be repeated here.

[0203] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0204] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0205] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0206] This application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device.

[0207] Computer-readable storage media can be, for example—but not limited to—electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0208] A computer-readable storage medium can be sent, propagated, or transmitted for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0209] A computer-readable storage medium carries one or more programs that, when executed by an electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0210] Furthermore, this disclosure also provides an electronic device capable of implementing the above-described method.

[0211] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0212] The following reference Figure 10 To describe an electronic device 1000 according to such an embodiment of the present disclosure. Figure 10 The electronic device 1000 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0213] like Figure 10As shown, the electronic device 1000 is manifested in the form of a general-purpose computing device. The components of the electronic device 1000 may include, but are not limited to: at least one processing unit 1010, at least one storage unit 1020, a bus 1030 connecting different system components (including storage unit 1020 and processing unit 1010), and a display unit 1040.

[0214] The storage unit stores program code that can be executed by the processing unit 1010, causing the processing unit 1010 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 1010 can perform actions such as... Figure 1 As shown: Step S110, acquire the language instruction to be processed and the video stream; the language instruction is used to indicate the location of at least one tracking target, and the video stream contains multiple frames of images; Step S120, extract the (t-1)th visual feature corresponding to the (t-1)th frame of the video stream, and extract the word-level text features and sentence-level text features corresponding to the language instruction; t is an integer greater than or equal to 2; Step S130, perform cross-modal interleaving and fusion on the (t-1)th visual feature and the word-level text feature to obtain the (t-1)th cross-modal fusion feature; Step S140, decode the (t-1)th cross-modal fusion feature, the (t-1)th association feature, and the position marking information of each tracking target in the (t-2)th frame of the image, and output the (t-1)th decoded feature; Step S150, use the (t-1)th decoded feature to predict the tracking result of each tracking target in the (t-1)th frame of the image; the tracking result contains the content recognition information and position marking information of each tracking target in the (t-1)th frame of the image.

[0215] Storage unit 1020 may include readable media in the form of volatile storage units, such as random access memory (RAM) 10201 and / or cache memory 10202, and may further include read-only memory (ROM) 10203.

[0216] Storage unit 1020 may also include a program / utility 10204 having a set (at least one) program module 10205, such program module 10205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0217] Bus 1030 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.

[0218] Electronic device 1000 can also communicate with one or more external devices 1100 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 1000, and / or any device that enables electronic device 1000 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1050. Furthermore, electronic device 1000 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1060. As shown, network adapter 1060 communicates with other modules of electronic device 1000 via bus 1030. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0219] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

Claims

1. A target tracking method, characterized in that, include: Acquire language instructions and video streams to be processed; the language instructions are used to instruct the location of at least one tracked target, and the video stream contains multiple frames of images; Extract the (t-1)th visual feature corresponding to the (t-1)th frame image in the video stream, and extract the word-level text features and sentence-level text features corresponding to the language instruction; t is an integer greater than or equal to 2; The (t-1)th visual feature and the word-level text feature are fused across modalities to obtain the (t-1)th cross-modal fused feature; The (t-1)th cross-modal fusion feature, the (t-1)th associated feature, and the position marker information of each tracked target in the (t-2)th frame image are decoded to output the (t-1)th decoded feature; the (t-1)th associated feature is determined based on the sentence-level text feature, the (t-2)th decoded feature corresponding to the (t-2)th frame image, and the pre-learned decoding auxiliary features; The tracking result of each tracked target in the (t-1)th frame image is predicted using the (t-1)th decoding feature; the tracking result includes content recognition information and location marking information of each tracked target in the (t-1)th frame image.

2. The method according to claim 1, characterized in that, After extracting the word-level and sentence-level text features corresponding to the language instructions, the method further includes: The (t-1)th visual feature, the word-level text feature, and the sentence-level text feature are processed into the same dimension through a fully connected network layer.

3. The method according to claim 1, characterized in that, The step of performing cross-modal interleaving and fusion of the (t-1)th visual feature and the word-level text feature to obtain the (t-1)th cross-modal fused feature includes: The (t-1)th visual feature is enhanced based on a deformable attention mechanism to obtain the (t-1)th enhanced visual feature. The (t-1)th enhanced visual feature and the word-level text feature are fused together using a cross-modal interleaving encoder to obtain the (t-1)th cross-modal fused feature.

4. The method according to claim 3, characterized in that, The cross-modal interleaved encoder contains l coding layers, where l is an integer greater than 1; The step of performing cross-modal interleaving and fusion of the (t-1)th enhanced visual feature and the word-level text feature using a cross-modal interleaving encoder to obtain the (t-1)th cross-modal fused feature includes: The first cross-modal interleaving and fusion of the (t-1)th enhanced visual feature and the word-level text feature are performed through the first encoding layer to obtain the first visual text fusion feature and the first text visual fusion feature. The first visual text fusion feature and the first text visual fusion feature are subjected to a second cross-modal interleaving fusion through a second encoding layer to obtain the second visual text fusion feature and the second text visual fusion feature. After obtaining the l-th visual-text fusion feature and the l-th text-visual fusion feature output by the l-th coding layer, the l-th visual-text fusion feature is determined as the (t-1)-th cross-modal fusion feature.

5. The method according to claim 4, characterized in that, The first cross-modal interleaving and fusion of the (t-1)th enhanced visual feature and the word-level text feature through the first encoding layer to obtain the first visual-text fusion feature and the first text-visual fusion feature includes: The first query vector corresponding to the word-level text feature is calculated through the first encoding layer, and the second query vector corresponding to the (t-1)th enhanced visual feature is calculated. Calculate the shared attention vector based on the first query vector, the second query vector, or the dimension value of the first query vector or the second query vector; Calculate the first value vector corresponding to the (t-1)th enhanced visual feature, and determine the first visual text fusion feature based on the first value vector, the shared attention vector, and the preset first weight matrix; Calculate the second value vector corresponding to the word-level text feature, and determine the first text visual fusion feature based on the second value vector, the shared attention vector, and the preset second weight matrix.

6. The method according to any one of claims 1 to 5, characterized in that, When t equals 2, the (t-1)th associated feature is determined in the following way: After repeating the sentence-level text feature a specified number of times, it is superimposed with the pre-learned decoding auxiliary feature to obtain the (t-1)th associated feature.

7. The method according to any one of claims 1 to 5, characterized in that, When t is greater than 2, the (t-1)th associated feature is determined in the following way: The (t-2)th decoded feature corresponding to the (t-2)th frame image is concatenated with the pre-learned decoding auxiliary feature to obtain the concatenated feature; The sentence-level text feature is repeated a specified number of times and then superimposed with the concatenated feature to obtain the (t-1)th associated feature.

8. The method according to any one of claims 1 to 5, characterized in that, The location marker information for each tracked target in the (t-2)th frame image includes the location parameters of the location box of each tracked target in the (t-2)th frame image.

9. The method according to claim 1, characterized in that, After predicting the tracking result of each tracked target in the (t-1)th frame image using the (t-1)th decoded feature, the method further includes: Extract the t-th visual feature corresponding to the t-th frame image in the video stream; The t-th visual feature and the word-level text feature are interleaved and fused across modally to obtain the t-th cross-modal fused feature; The t-th cross-modal fusion feature, the t-th association feature, and the position marker information of each tracked target in the (t-1)-th frame image are decoded to output the t-th decoded feature; the t-th association feature is determined based on the sentence-level text feature, the (t-1)-th decoded feature corresponding to the (t-1)-th frame image, and the pre-learned decoding auxiliary features; The tracking result of each tracked target in the t-th frame image is predicted using the t-th decoding feature; the tracking result includes content recognition information and location marking information of each tracked target in the t-th frame image.

10. A target tracking device, characterized in that, include: An acquisition module is used to acquire language instructions and video streams to be processed; the language instructions are used to indicate the location of at least one tracking target, and the video stream contains multiple frames of images. The feature extraction module is used to extract the (t-1)th visual feature corresponding to the (t-1)th frame image in the video stream, and to extract the word-level text features and sentence-level text features corresponding to the language instructions; t is an integer greater than or equal to 2; The cross-modal fusion module is used to perform cross-modal interleaving and fusion of the (t-1)th visual feature and the word-level text feature to obtain the (t-1)th cross-modal fused feature; The decoding feature output module is used to decode the (t-1)th cross-modal fusion feature, the (t-1)th associated feature, and the position marker information of each tracked target in the (t-2)th frame image, and output the (t-1)th decoded feature; When t equals 2, the (t-1)th associated feature is determined based on the sentence-level text features and the pre-learned decoding auxiliary features; when t is greater than 2, the (t-1)th associated feature is determined based on the (t-2)th decoding feature and the (t-2)th associated feature of the (t-2)th frame image. The result output module is used to predict the tracking result of each tracking target in the (t-1)th frame image using the (t-1)th decoded feature; the tracking result includes content recognition information and location marking information of each tracking target in the (t-1)th frame image.

11. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the target tracking method according to any one of claims 1 to 9.

12. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the target tracking method according to any one of claims 1 to 9 by executing the executable instructions.

Images