Target tracking method and device, electronic equipment, program product and medium

Abstract

The invention provides a target tracking method and device, electronic equipment, a program product and a medium, and relates to the field of artificial intelligence. According to the method, firstly, a detection sequence vector and a tracking sequence vector used in multi-target tracking can be obtained; the detection sequence vector records detection information of the to-be-tracked object and is used for detecting the to-be-tracked object; the tracking sequence vector records the image features of the tracked object and is used for continuously tracking the tracked object. And then, enhancing the detection sequence vector by utilizing the category description text of the object to be tracked, and enhancing the tracking sequence vector by utilizing the attribute description text of the object to be tracked, so as to integrate text semantic features into the sequence vectors. In consideration of the fact that text semantic features have relatively high-order semantic description capability and are completely not affected by noise such as scale change, deformation and illumination change, the multi-target tracking performance can be improved after text semantic enhancement is performed on a detection sequence vector and a tracking sequence vector.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a target tracking method, device, electronic equipment, program product and medium. Background Art

[0002] In related technologies, end-to-end detection is a basic paradigm for solving multi-target tracking problems. This approach can integrate detection and tracking, and can learn to model the long-term temporal changes of target objects, implicitly perform temporal association, and is conducive to improving the overall performance of the tracking system. However, in related end-to-end detection schemes, visual image features are generally used for detection and tracking, and visual image features are easily affected by noise such as scale changes, deformation, and illumination changes, which can easily lead to reduced tracking performance. Summary of the invention

[0003] The purpose of the present invention is to provide a target tracking method, device, electronic device, program product and medium, which can improve tracking detection by fusing text features with image features to enhance target tracking performance.

[0004] In order to solve the above technical problems, the present invention provides a target tracking method, comprising: Obtaining global image features of the image to be detected, category description text corresponding to the object to be tracked, a detection sequence vector and a tracking sequence vector; the detection sequence vector records the detection information of the object to be tracked, and the tracking sequence vector records the image features of the tracked object; The category description text and the detection sequence vector are fused into an enhanced detection sequence vector; The global image features, the enhanced detection sequence vectors, and the tracking sequence vectors are decoded by using a pre-trained decoder to obtain tracking information of the object to be tracked in the image to be detected; the tracking information includes at least position information and image features; The tracking sequence vector is updated using the tracking information, and the updated tracking sequence vector is enhanced using the pre-trained feature interactor and the attribute description text corresponding to the object to be tracked, so as to continue to track the target using the enhanced tracking sequence vector.

[0005] Optionally, the category description text is fused with the detection sequence vector to form an enhanced detection sequence vector, including: Encode the category description text to obtain a category description text vector; The category description text vector is concatenated with the detection sequence vector, and the concatenated result is subjected to linear transformation and normalization in turn to obtain an enhanced detection sequence vector.

[0006] Optionally, the pre-trained feature interactor comprises a first mapping unit, a second mapping unit, a pre-trained text feature, and a pre-trained image feature; The updated tracking sequence vector is enhanced using the pre-trained feature interactor and the attribute description text corresponding to the object to be tracked, including: Obtaining text features and extracting image features of the tracked object from the tracking sequence vector; wherein the text features are generated using attribute description text; Using a first mapping unit to project the pre-trained text features into mapped image features, and using a second mapping unit to project the pre-trained image features into mapped text features; The text features, pre-trained text features, and mapped text features are fused to obtain fused text features, and the fused text features are encoded using a pre-trained text-image contrast learning model to obtain encoded text features; The image features, the pre-trained image features, and the mapped image features are fused to obtain fused image features, and the fused image features are encoded using a pre-trained text-image contrast learning model to obtain encoded image features; The encoded text features and the encoded image features are processed through a cross-attention mechanism to obtain an enhanced tracking sequence vector.

[0007] Optionally, the first mapping unit and the second mapping unit are both multi-layer perceptrons.

[0008] Optionally, obtain text features, including: According to the category description text, text features of the category corresponding to the category description text are obtained in a tracking text prompt library; the tracking text prompt library contains text features corresponding to various objects to be tracked.

[0009] Optionally, it also includes: Get the attribute description text corresponding to the object to be tracked; Encode the attribute description text to obtain text features; Establish a correspondence between the text features and the category of the object to be tracked, and save the correspondence to the tracking text prompt library.

[0010] Optionally, the pre-trained feature interactor comprises a first mapping unit, a pre-trained text feature; The updated tracking sequence vector is enhanced using the pre-trained feature interactor and the attribute description text corresponding to the object to be tracked, including: Obtaining text features and extracting image features of the tracked object from the tracking sequence vector; wherein the text features are generated using attribute description text; Projecting the pre-trained text features into mapped image features using a first mapping unit; The text features and the pre-trained text features are fused to obtain fused text features, and the fused text features are encoded using the pre-trained text-image contrast learning model to obtain encoded text features; The image features and the mapped image features are fused to obtain fused image features, and the fused image features are encoded using a pre-trained text-image contrast learning model to obtain encoded image features; The encoded text features and the encoded image features are processed through a cross-attention mechanism to obtain an enhanced tracking sequence vector.

[0011] Optionally, the pre-trained feature interactor comprises a second mapping unit, a pre-trained image feature; The updated tracking sequence vector is enhanced using the pre-trained feature interactor and the attribute description text corresponding to the object to be tracked, including: Obtaining text features and extracting image features of the tracked object from the tracking sequence vector; wherein the text features are generated using attribute description text; Projecting the pre-trained image features into mapped text features using a second mapping unit; The text features and the mapped text features are fused to obtain fused text features, and the fused text features are encoded using a pre-trained text-image contrast learning model to obtain encoded text features; The image features and the pre-trained image features are fused to obtain fused image features, and the fused image features are encoded using the pre-trained text-image contrast learning model to obtain encoded image features; The encoded text features and the encoded image features are processed through a cross-attention mechanism to obtain an enhanced tracking sequence vector.

[0012] Optionally, obtaining global image features of the image to be detected includes: Use the pre-trained convolutional neural network model to extract features from the image to be detected and obtain primary image features; According to the spatial position corresponding to the primary image feature in the image to be detected, the primary image feature is spatially encoded to obtain an encoded primary image feature; The encoded primary image features are used as the first key value and the first query value, and the primary image features are used as the first numerical value. A pre-trained image feature encoder is used to perform a self-attention mechanism on the first key value, the first query value and the first numerical value to obtain a first processing result. The pre-trained image feature encoder is used to perform forward propagation calculation on the first processing result to obtain a global image feature.

[0013] Optionally, a pre-trained decoder is used to decode the global image features, the enhanced detection sequence vector, and the tracking sequence vector to obtain tracking information of the object to be tracked in the image to be detected, including: The enhanced detection sequence vector and the tracking sequence vector are concatenated to obtain the object sequence vector; According to the spatial position corresponding to the global image feature in the image to be detected, the global image feature is spatially encoded to obtain the encoded global image feature; Using the pre-trained decoder to perform a self-attention mechanism on the object sequence vector to obtain a second processing result; The second processing result, the encoded global image features and the global image features are used as the second query value, the second key value and the second numerical value respectively, and the pre-trained decoder is used to perform attention mechanism processing on the second query value, the second key value and the second numerical value to obtain the third processing result, and the pre-trained decoder is used to perform forward propagation calculation on the third processing result to obtain tracking information.

[0014] Optionally, the tracking information further includes a tracking state category of the object to be tracked and a confidence level corresponding to the tracking state category; The tracking sequence vector is updated using the tracking information, including: If it is determined according to the tracking state category that the object to be tracked is a newly added tracking object, then when the confidence of the object to be tracked is greater than a first preset threshold, adding the image feature of the object to be tracked to the tracking sequence vector; If it is determined according to the tracking state category that the object to be tracked belongs to a tracked object, then when the confidence of the object to be tracked is less than a second preset threshold, the image features of the object to be tracked are moved out of the tracking sequence vector, or when the confidence of the object to be tracked is not less than the second preset threshold, the tracking sequence vector is updated using the image features of the object to be tracked.

[0015] Optionally, it also includes: Obtain multiple training images, category description texts corresponding to the training tracking objects, initial detection sequence vectors and tracking sequence vectors, and extract training image features of the training images; the training images are marked with real position information of the training tracking objects, and the initial tracking sequence vectors are empty; The initial detection sequence vector is randomly initialized, and the category description text and the initial detection sequence vector are fused into an initial enhanced detection sequence vector; Determine predicted position information of the training tracking object in each training image using the initial enhanced detection sequence vector, the tracking sequence vector, the training image features, the initial decoder and the initial feature interactor; The loss between the predicted position information and the actual position information is calculated using a preset loss function to obtain a loss value; The loss value is used to update the parameters of the initial detection sequence vector, the initial decoder, and the initial feature interactor to obtain the detection sequence vector, the pre-trained decoder, and the pre-trained feature interactor.

[0016] The present invention also provides a target tracking device, comprising: An acquisition module is used to acquire global image features of the image to be detected, a category description text corresponding to the object to be tracked, a detection sequence vector and a tracking sequence vector; the detection sequence vector records the detection information of the object to be tracked, and the tracking sequence vector records the image features of the tracked object; A detection sequence enhancement module is used to fuse the category description text with the detection sequence vector into an enhanced detection sequence vector; A tracking detection module is used to decode the global image features, the enhanced detection sequence vector, and the tracking sequence vector using a pre-trained decoder to obtain tracking information of the object to be tracked in the image to be detected; the tracking information includes at least position information and image features; The tracking sequence enhancement module is used to update the tracking sequence vector using the tracking information, and enhance the updated tracking sequence vector using the pre-trained feature interactor and the attribute description text corresponding to the object to be tracked, so as to continue to track the target using the enhanced tracking sequence vector.

[0017] The present invention also provides an electronic device, comprising: Memory for storing computer programs; A processor is used to implement the above-mentioned target tracking method when executing a computer program.

[0018] The present invention also provides a computer program product, including a computer program or an instruction, and the computer program or the instruction implements the above-mentioned target tracking method when executed by a processor.

[0019] The present invention also provides a non-volatile computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are loaded and executed by a processor, the above-mentioned target tracking method is implemented.

[0020] The present invention provides a target tracking method, comprising: obtaining global image features of an image to be detected, a category description text corresponding to an object to be tracked, a detection sequence vector and a tracking sequence vector; the detection sequence vector records the detection information of the object to be tracked, and the tracking sequence vector records the image features of the tracked object; fusing the category description text with the detection sequence vector into an enhanced detection sequence vector; using a pre-trained decoder to jointly decode the global image features, the enhanced detection sequence vector and the tracking sequence vector to obtain tracking information of the object to be tracked in the image to be detected; the tracking information at least includes position information and image features; using the tracking information to update the tracking sequence vector, and using a pre-trained feature interactor and the attribute description text corresponding to the object to be tracked to enhance the updated tracking sequence vector, so as to continue to track the target using the enhanced tracking sequence vector.

[0021] The beneficial effect of the present invention is that the present invention can first obtain the global image features of the image to be detected, the category description text corresponding to the object to be tracked, the detection sequence vector and the tracking sequence vector. Among them, the category description text is used to describe the category of the object to be tracked; the detection sequence vector records the detection information of the object to be tracked, which is used to detect the object to be tracked; the tracking sequence vector records the image features of the tracked object, which is used to continue to track the tracked object. Subsequently, the present invention can fuse the category description text with the detection sequence vector into an enhanced detection sequence vector, that is, the detection sequence vector can be enhanced by using the text semantic features. Subsequently, the present invention can use the pre-trained decoder to decode the global image features, the enhanced detection sequence vector, and the tracking sequence vector together to obtain the tracking information of the object to be tracked in the image to be detected, and the tracking information at least includes position information and image features. Subsequently, the present invention can use the tracking information to update the tracking sequence vector, and use the pre-trained feature interactor and the attribute description text corresponding to the object to be tracked to enhance the updated tracking sequence vector, that is, the tracking sequence vector can also be enhanced by using the text semantic features, so as to continue to track the target by using the enhanced tracking sequence vector. In simple terms, the present invention can enhance the detection sequence vector using the category description text of the object to be tracked, and enhance the tracking sequence vector using the attribute description text of the object to be tracked. Considering that the text semantic features have strong high-order semantic description capabilities and are completely unaffected by noise such as scale changes, deformation, and illumination changes, the overall performance of multi-target tracking can be significantly enhanced. The present invention also provides a target tracking device, an electronic device, a computer-readable storage medium, and a computer program product, which have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0023] Figure 1 A flow chart of a target tracking method provided by an embodiment of the present invention; Figure 2 A schematic diagram of a pre-trained image feature encoder provided by an embodiment of the present invention; Figure 3 A schematic diagram of a process for generating an enhanced detection sequence vector provided by an embodiment of the present invention; Figure 4 A schematic diagram of a pre-trained decoder provided by an embodiment of the present invention; Figure 5 A schematic diagram of a tracking sequence vector enhancement method based on text-image bidirectional prompt learning provided by an embodiment of the present invention; Figure 6 A schematic diagram of a tracking sequence vector enhancement method based on text-image unidirectional prompt learning provided by an embodiment of the present invention; Figure 7 A schematic diagram of a tracking sequence vector enhancement method based on image-text unidirectional prompt learning provided by an embodiment of the present invention; Figure 8 A schematic diagram of a system provided by an embodiment of the present invention; Fig. 9 A structural block diagram of a target tracking device provided by an embodiment of the present invention; Fig.10 The present invention provides a structural block diagram of an electronic device. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] In related technologies, end-to-end detection is a basic paradigm for solving the problem of multi-target tracking. This method can integrate detection and tracking, and can learn and model the long-term changes of target objects, implicitly perform temporal association, and help improve the overall performance of the tracking system. For example, the MOTR framework (End-to-End Multiple-object Tracking with Transformer) is an end-to-end multi-target tracking framework based on transformers. However, in related schemes, visual image features are generally used for tracking and detection, and visual image features are easily affected by noise such as scale changes, deformation, and illumination changes, which can easily lead to reduced tracking performance.

[0026] In view of this, in order to solve the technical problem of how to improve the performance of multi-target tracking, the present invention takes into account that text semantic features have strong high-order semantic description capabilities and are completely unaffected by noise such as scale changes, deformation, and illumination changes. Therefore, a target tracking method can be provided, which can improve tracking detection by fusing text features with image features to improve target tracking performance.

[0027] The target tracking method provided by the present invention is introduced below. For easy understanding, please refer to Figure 1 , Figure 1 A flow chart of a target tracking method provided by an embodiment of the present invention. The method may include: S101, obtaining global image features of an image to be detected, a category description text corresponding to an object to be tracked, a detection sequence vector and a tracking sequence vector; the detection sequence vector records detection information of the object to be tracked, and the tracking sequence vector records image features of a tracked object.

[0028] In this step, global image features, category description text, detection sequence vectors, and tracking sequence vectors are first obtained to use these data as model inputs. The global image features, category description text, detection sequence vectors, and tracking sequence vectors are introduced below.

[0029] The image to be detected is an image that needs to be tracked and detected, which can come from an image sequence, such as a video stream. The image to be detected can contain object objects, which can be objects to be tracked. For example, the present invention can track and detect objects such as people, animals, and vehicles in the image to be detected. In order to achieve the tracking effect, it is necessary to extract global image features from the image to be detected, and use the global image features for tracking and detection. Among them, the global image features are the image features of the image to be detected as a whole, which cover all the contents of the image to be detected.

[0030] The category description text is a text content used to describe the category of the object to be tracked. For example, when the object to be tracked is a person, the category description text may be "Person"; when the object to be tracked is a vehicle, the category description text may be "Car".

[0031] The detection query is a learnable vector used to detect the object to be tracked. It can contain multiple vector parameters, which can be used to record the detection information of multiple objects to be tracked. For example, a vector parameter corresponds to a potential category of objects to be tracked and records the detection information of the object to be tracked. Belonging to the learnable vector category means that the detection query needs to update parameters during the model training process to meet the tracking detection requirements.

[0032] The tracking query vector is a vector used to continue tracking the tracked object. It records the image features of the tracked object. The tracked object refers to the object that has been detected and tracked in the previous image. When performing the first tracking detection, the tracking query vector is empty.

[0033] It should be noted that this embodiment does not limit the method of extracting global image features, and can be set according to actual application requirements. For example, a traditional convolutional neural network model (CNN, Convolutional Neural Networks) can be used for feature extraction. In order to enhance the correlation between image features, on the basis of using the convolutional neural network model, an image feature encoder including a self-attention mechanism module (such as a multi-head attention module) can be introduced, and the primary image features output by the convolutional neural network model can be further processed based on the self-attention mechanism (Self-Attention) in the image feature encoder.

[0034] Based on this, obtaining the global image features of the image to be detected may include: Step 11: Use the pre-trained convolutional neural network model to extract features of the image to be detected and obtain primary image features.

[0035] In this step, the pre-trained convolutional neural network model refers to a trained convolutional neural network model. This embodiment does not limit the specific type of the pre-trained convolutional neural network model, for example, it can be a RestNet series model, a Darknet series model, etc.

[0036] Step 12: According to the spatial position corresponding to the primary image feature in the image to be detected, the primary image feature is spatially encoded to obtain the encoded primary image feature.

[0037] In this step, it is considered that each primary image feature is generated according to the image content of each spatial position in the image to be detected; and there is a correlation between the image contents at different spatial positions. For example, the image contents at multiple spatial positions in multiple images to be detected constitute a complete object to be tracked, that is, there is also a correlation between the primary image features at different spatial positions. Therefore, in order to ensure that the image feature encoder can learn the correlation between the primary image features, the primary image features can be spatially encoded according to the spatial position corresponding to the primary image features in the image to be detected to obtain the encoded primary image features.

[0038] It should be noted that this embodiment does not limit the specific process of spatial position encoding, and reference may be made to related technologies of artificial intelligence.

[0039] Step 13: Use the encoded primary image features as the first key value and the first query value, and the primary image features as the first numerical value, use the pre-trained image feature encoder to perform a self-attention mechanism on the first key value, the first query value and the first numerical value to obtain a first processing result, and use the pre-trained image feature encoder to perform forward propagation calculation on the first processing result to obtain the global image feature.

[0040] In this step, the pre-trained image feature encoder refers to an image feature encoder that has been trained, which is at least provided with a self-attention mechanism module and a feedforward neural network module, and can perform self-attention mechanism processing and forward propagation processing on the encoded primary image features. Specifically, the encoded primary image features can be used as the first key value (K, Key) and the first query value (Q, Query), and the primary image features can be used as the first value (V, Value), and the pre-trained image feature encoder can be used to perform self-attention mechanism processing on the first key value, the first query value and the first value to obtain a first processing result. Subsequently, the pre-trained image feature encoder can be used to perform forward propagation calculation on the first processing result to obtain the global image features.

[0041] For easier understanding, please refer to Figure 2 , Figure 2 A schematic diagram of a pre-trained image feature encoder provided in an embodiment of the present invention. In the schematic diagram, the structure framed by the dotted box is a transformer encoder in the pre-trained image feature encoder. The transformer encoder may include a multi-head attention calculation module (MHA, Multi-head Attention, i.e., a self-attention mechanism module), an addition normalization module, and a forward propagation module. The pre-trained image feature encoder may be composed of N transformer encoder units connected in series, N ≥ 1. High-level image features are global image features.

[0042] S102: Fusion the category description text and the detection sequence vector into an enhanced detection sequence vector.

[0043] In this step, the category description text can be fused with the detection sequence vector to integrate the text semantic information corresponding to the category of the object to be tracked into the detection sequence vector to obtain an enhanced detection sequence vector. In this way, when using the enhanced detection sequence vector for detection and tracking, the model can well perceive the category information of the object to be tracked, thereby improving the detection and tracking performance.

[0044] It should be noted that this embodiment does not limit how to fuse the category description text and the detection sequence vector into an enhanced detection sequence vector, which can be based on actual application requirements. For example, the category description text can be first encoded, for example, the category description text can be encoded using the WordPiece tokenization method to obtain a category description text vector. Subsequently, the category description text vector and the detection sequence vector can be linearly transformed and normalized in sequence to obtain a fused detection sequence vector to ensure that the category description text vector and the detection sequence vector can be better fused.

[0045] Based on this, the category description text and the detection sequence vector are fused into an enhanced detection sequence vector, which may include: Step 21: Encode the category description text to obtain a category description text vector; Step 22: Concatenate the category description text vector with the detection sequence vector, and perform linear transformation and normalization on the concatenation result in turn to obtain an enhanced detection sequence vector.

[0046] It should be noted that this embodiment does not limit the specific process of the above encoding processing, linear transformation processing and normalization processing, and reference may be made to relevant technologies in the field of artificial intelligence. Figure 3 , Figure 3 A schematic diagram of an enhanced detection sequence vector generation process provided by an embodiment of the present invention, wherein "text description" is the attribute description text, "detect query" is the detection sequence vector, and "detect query based on text prompt" is the enhanced detection sequence vector.

[0047] S103, using a pre-trained decoder to decode the global image features, the enhanced detection sequence vector, and the tracking sequence vector to obtain tracking information of the object to be tracked in the image to be detected; the tracking information includes at least position information and image features.

[0048] In this step, the pre-trained decoder can be used to decode the global image features, enhanced detection sequence vectors, and tracking sequence vectors to obtain tracking information of the object to be tracked in the image to be detected. The tracking information at least includes the position information and image features corresponding to the object to be tracked in the image to be detected. That is, the pre-trained decoder is used to perform target tracking detection according to the input.

[0049] It should be noted that this embodiment does not limit the specific process of using the pre-trained decoder to jointly decode the global image features, the enhanced detection sequence vector, and the tracking sequence vector. For example, the enhanced detection sequence vector and the tracking sequence vector can be first concatenated to obtain an object sequence vector (object query), and the pre-trained decoder can be used to perform a self-attention mechanism on the object sequence vector to obtain a second processing result. Subsequently, the pre-trained decoder can be used to continue to perform an attention mechanism on the above-mentioned second processing result and the global image features to obtain a third processing result, and the third processing result can be forward propagated to obtain tracking information.

[0050] Based on this, the pre-trained decoder is used to decode the global image features, enhanced detection sequence vectors, and tracking sequence vectors to obtain the tracking information of the object to be tracked in the image to be detected, which may include: Step 31: Concatenate the enhanced detection sequence vector and the tracking sequence vector to obtain the object sequence vector.

[0051] Step 32: According to the spatial position corresponding to the global image feature in the image to be detected, the global image feature is spatially encoded to obtain the encoded global image feature.

[0052] In this step, the description of spatial position encoding of global image features is similar to the description of spatial position encoding of processed image features, and will not be repeated here.

[0053] Step 33: Use the pre-trained decoder to perform self-attention mechanism processing on the object sequence vector to obtain a second processing result.

[0054] In this step, the object sequence vector is first processed by the self-attention mechanism to obtain a second processing result, so as to perform feature interaction on the enhanced detection sequence vector and the tracking sequence vector.

[0055] Step 34: Use the second processing result, the encoded global image feature and the global image feature as the second query value, the second key value and the second numerical value in turn, use the pre-trained decoder to perform attention mechanism processing on the second query value, the second key value and the second numerical value to obtain the third processing result, and use the pre-trained decoder to perform forward propagation calculation on the third processing result to obtain tracking information.

[0056] In this step, the attention mechanism will continue to be performed on the second processing result and the global image feature to obtain the third processing result, and the third processing result will be forward propagated to perform feature interaction on the second processing result and the global image feature.

[0057] For easier understanding, please refer to Figure 4 , Figure 4 A schematic diagram of a pre-trained decoder provided in an embodiment of the present invention. In the schematic diagram, the structure selected by the dotted box is a transformer decoder unit (Transformer decoder) in the pre-trained decoder. The transformer decoder unit may include a multi-head attention calculation module (MHA, Multi-headAttention), an addition normalization module and a forward propagation module. The pre-trained decoder may be composed of N transformer decoder units connected in series, N ≥ 1.

[0058] S104, using the tracking information to update the tracking sequence vector, and using the pre-trained feature interactor and the attribute description text corresponding to the object to be tracked to enhance the updated tracking sequence vector, so as to continue to track the target using the enhanced tracking sequence vector.

[0059] In this step, the tracking sequence vector can first be updated using the tracking information obtained in the previous step, such as adding the newly detected image features of the object to be tracked to the tracking sequence vector, updating the tracking sequence vector using the new image features of the tracked object, and removing the image features of the tracked object that has disappeared from the tracking sequence vector.

[0060] In order to facilitate the tracking sequence vector, the pre-trained decoder can output the tracking state category of the object to be tracked and the confidence corresponding to the tracking state category in addition to outputting the position information and image features of the object to be tracked in the image to be detected. In other words, in simple terms, the pre-trained decoder can also determine the tracking status of the object to be tracked. The tracking state category can specifically be a newly added tracking state (newborn) or a tracked state (terminated). The newly added tracking state indicates that the object to be tracked belongs to a newly added tracking object in the current image; the tracked state indicates that the object to be tracked belongs to a tracked object in the historical image. Confidence refers to the degree of credibility that the object to be tracked belongs to this state. The following introduces the specific method of updating the tracking sequence vector using image features, tracking state categories, and confidence.

[0061] Based on this, the tracking information also includes the tracking state category of the object to be tracked and the confidence corresponding to the tracking state category; using the tracking information to update the tracking sequence vector may include: Step 41: if it is determined according to the tracking state category that the object to be tracked is a newly added tracking object, then when the confidence of the object to be tracked is greater than a first preset threshold, the image feature of the object to be tracked is added to the tracking sequence vector.

[0062] In this step, if the object to be tracked is a newly added object, it is necessary to determine whether the confidence of the object is greater than a first preset threshold. If it is, it can be reliably determined that the object to be tracked is a newly added object in the current image, and then the image features of the object to be tracked can be added to the tracking sequence vector.

[0063] Step 42: If it is determined according to the tracking state category that the object to be tracked belongs to a tracked object, then when the confidence of the object to be tracked is less than a second preset threshold, the image features of the object to be tracked are moved out of the tracking sequence vector, or when the confidence of the object to be tracked is not less than the second preset threshold, the tracking sequence vector is updated using the image features of the object to be tracked.

[0064] In this step, if the object to be tracked belongs to an already tracked object, it is necessary to determine the confidence of the object for distinguishing processing. For example, when it is determined that the confidence of the object to be tracked is less than the second preset threshold, it is impossible to continue to reliably determine that the object to be tracked is in a tracked state, and the image features of the object to be tracked can be moved out of the tracking sequence vector. For another example, when it is determined that the confidence of the object to be tracked is not less than the second preset threshold, it can continue to reliably determine that the object to be tracked is in a tracked state, and the image features of the object to be tracked can be used to update the tracking sequence vector to add the latest image features of the tracked object to the tracking sequence vector.

[0065] It should be noted that this embodiment does not limit the specific values ​​of the first preset threshold and the second preset threshold, which can be set according to actual application requirements.

[0066] Furthermore, in step S104, the updated tracking sequence vector needs to be enhanced using the pre-trained feature interactor and the attribute description text corresponding to the object to be tracked. The attribute description text is used to describe the attributes of the object to be tracked. For example, when the tracking object is a pedestrian, the attributes of pedestrians commonly used for identification in the pedestrian tracking process are: gender (male, female), age (adult, child), clothing color, accessories (bag, bicycle, etc.), etc. By performing feature interaction between the attribute description text and the tracking sequence vector, it can be ensured that the model can pay attention to other attributes of the object to be tracked, thereby improving the tracking detection effect.

[0067] It should be noted that this embodiment does not limit how to enhance the updated tracking sequence vector using the pre-trained feature interactor and the attribute description text corresponding to the object to be tracked. For example, the attribute description text can be used to perform cross-attention mechanism processing on the tracking sequence vector to achieve feature interaction. Taking into account that the attribute description text and the tracking sequence vector belong to features of two different modalities, in order to enhance the degree of interaction between cross-modal features, this embodiment can also map the features of the above two modalities to each other, and can use the text-image contrast learning model to narrow the distance between the features of the above two modalities. Please refer to the description in the subsequent embodiments for details.

[0068] Based on the above embodiments, the present invention can first obtain the global image features of the image to be detected, the category description text corresponding to the object to be tracked, the detection sequence vector and the tracking sequence vector. Among them, the category description text is used to describe the category of the object to be tracked; the detection sequence vector records the detection information of the object to be tracked, which is used to detect the object to be tracked; the tracking sequence vector records the image features of the tracked object, which is used to continue to track the tracked object. Subsequently, the present invention can fuse the category description text with the detection sequence vector into an enhanced detection sequence vector, that is, the detection sequence vector can be enhanced using the text semantic features. Subsequently, the present invention can use the pre-trained decoder to decode the global image features, the enhanced detection sequence vector, and the tracking sequence vector together to obtain the tracking information of the object to be tracked in the image to be detected, and the tracking information at least includes position information and image features. Subsequently, the present invention can use the tracking information to update the tracking sequence vector, and use the pre-trained feature interactor and the attribute description text corresponding to the object to be tracked to enhance the updated tracking sequence vector, that is, the tracking sequence vector can also be enhanced using the text semantic features to continue to track the target using the enhanced tracking sequence vector. Simply put, the present invention can use the category description text of the object to be tracked to enhance the detection sequence vector, and use the attribute description text of the object to be tracked to enhance the tracking sequence vector. Considering that the text semantic features have strong high-order semantic description capabilities and are completely unaffected by noise such as scale changes, deformation, and illumination changes, the overall performance of multi-target tracking can be significantly enhanced.

[0069] Based on the above embodiment, the specific process of enhancing the updated tracking sequence vector using the pre-trained feature interactor and the attribute description text corresponding to the object to be tracked is introduced below. In one possible situation, the pre-trained feature interactor may include a first mapping unit, a second mapping unit, pre-trained text features, and pre-trained image features. The first mapping unit and the second mapping unit are both trainable module units; the pre-trained text features and the pre-trained image features are both learnable vectors, and parameter optimization is required in the model training stage. The above-mentioned enhancement processing of the updated tracking sequence vector using the pre-trained feature interactor and the attribute description text corresponding to the object to be tracked may include: S201, obtaining text features, and extracting image features of the tracked object from the tracking sequence vector; wherein the text features are generated using attribute description text.

[0070] In this step, the text features corresponding to the above-mentioned attribute description text can be obtained, and the image features of the tracked object can be obtained from the tracking sequence vector. It is worth noting that in order to improve the interaction effect, when obtaining the text features, all text features related to the attributes of the object to be tracked can be obtained.

[0071] Furthermore, to facilitate the acquisition of text features, the present embodiment may preset a tracking text prompt library, which contains text features corresponding to attribute description texts of various objects to be tracked. Furthermore, the present embodiment may acquire text features of categories corresponding to the category description texts in the tracking text prompt library according to the category description texts.

[0072] Based on this, text features can be obtained, including: Step 51: According to the category description text, obtain text features of the category corresponding to the category description text in the tracking text prompt library; the tracking text prompt library contains text features corresponding to various objects to be tracked.

[0073] It should be noted that this embodiment does not limit the specific process of converting the attribute description text into text features. For example, the attribute description text may be encoded, such as using the Wordpiece Tokenization method to encode the attribute description text to obtain text features.

[0074] Based on this, the method may further include: Step 61: Obtain the attribute description text corresponding to the object to be tracked.

[0075] Step 62: Encode the attribute description text to obtain text features.

[0076] Step 63: Establish a correspondence between the text feature and the category of the object to be tracked, and save the correspondence to the tracking text prompt library.

[0077] It should be noted that this embodiment does not limit the specific attribute description text, and can be set according to actual application requirements.

[0078] S202: Use a first mapping unit to project the pre-trained text features into mapped image features, and use a second mapping unit to project the pre-trained image features into mapped text features.

[0079] In this step, the pre-trained text features can be projected into mapped image features using the first mapping unit, and the pre-trained image features can be projected into mapped text features using the second mapping unit. The purpose of setting the pre-trained text features and pre-trained image features is that in terms of text semantics, although the prior attribute description text can cover most of the attributes of the object to be tracked, the actual tracking detection scene is dynamic and changeable. On the one hand, there are attributes that the attribute description text cannot cover. On the other hand, the attribute description text only statically describes the attributes of the object to be tracked, and cannot adapt well to the dynamic changes of the object to be tracked itself. Therefore, pre-trained text features can be introduced to use pre-trained text features to complete the information of static text features through dynamic training. Similarly, pre-trained image features can be used to complete the information of image features.

[0080] Furthermore, the first mapping unit is used to project the pre-trained text features into mapped image features, and the second mapping unit is used to project the pre-trained image features into mapped text features. The core purpose is to enhance the degree of interaction between text features and image features. It should be noted that this embodiment does not limit the specific types and structures of the first mapping unit and the second mapping unit, and can be set according to actual application requirements. For example, the first mapping unit and the second mapping unit can both be multi-layer perceptrons.

[0081] S203, fusing the text features, pre-trained text features, and mapped text features to obtain fused text features, and encoding the fused text features using a pre-trained text-image contrast learning model to obtain encoded text features.

[0082] In this step, the text features, pre-trained text features, and mapped text features can be fused to obtain fused text features to enrich the text semantic information in the fused text features. Subsequently, the fused text features can be encoded using the pre-trained text-image contrast learning model to obtain encoded text features, thereby narrowing the distance between text features and image features for better feature interaction.

[0083] It should be noted that this embodiment does not limit the specific type of the pre-trained text-image contrast learning model, for example, it can be a Clip model.

[0084] S204, fusing the image features, the pre-trained image features, and the mapped image features to obtain fused image features, and encoding the fused image features using a pre-trained text-image contrast learning model to obtain encoded image features.

[0085] In this step, the image features, pre-trained image features, and mapped image features can be fused to obtain fused image features to enrich the image information in the fused image features. Subsequently, the fused image features can be encoded using the pre-trained text-image contrast learning model to obtain encoded image features, thereby narrowing the distance between text features and image features for better feature interaction.

[0086] S205. Perform cross-attention mechanism processing on the encoded text features and the encoded image features to obtain an enhanced tracking sequence vector.

[0087] In this step, the encoded text features and the encoded image features can be processed by a cross-attention mechanism, and the processing results of the cross-attention mechanism can be used to form an enhanced tracking sequence vector, thereby improving the detection and tracking performance of the tracking sequence vector.

[0088] For easier understanding, please refer to Figure 5 , Figure 5 A schematic diagram of a tracking sequence vector enhancement method based on text-image bidirectional prompt learning provided by an embodiment of the present invention. In this schematic diagram, the gray part constitutes a pre-trained feature interactor, the "learnable text feature vector" is a pre-trained text feature, the "learnable image feature vector" is a pre-trained image feature, the "text-image projection calculation" uses the first mapping unit, the "image-text projection calculation" uses the second mapping unit, and the "text encoding" and "image encoding" use the text-image contrast learning model.

[0089] Furthermore, the above embodiment introduces a bidirectional mapping of text to image, but any unidirectional mapping of text to image or image to text is also a way of multimodal learning. The following introduces a tracking sequence vector enhancement method based on unidirectional mapping of text to image.

[0090] Based on this, the pre-trained feature interactor includes a first mapping unit and a pre-trained text feature. The updated tracking sequence vector is enhanced using the pre-trained feature interactor and the attribute description text corresponding to the object to be tracked, including: S301, obtaining text features, and extracting image features of the tracked object from the tracking sequence vector; wherein the text features are generated using attribute description text.

[0091] S302: Use a first mapping unit to project the pre-trained text features into mapped image features.

[0092] Different from the above-mentioned embodiment, in this embodiment, the pre-trained feature interactor only includes the first mapping unit and the pre-trained text features, so it is only necessary to map the pre-trained text features into mapped image features using the first mapping unit.

[0093] S303, fusing the text features and the pre-trained text features to obtain fused text features, and encoding the fused text features using the pre-trained text-image contrast learning model to obtain encoded text features.

[0094] S304, fusing the image features and the mapped image features to obtain fused image features, and encoding the fused image features using a pre-trained text-image contrast learning model to obtain encoded image features.

[0095] In steps S303 and S304, different from steps S203 and S204, since this embodiment only performs one-way text-image mapping, the fused text features are obtained by fusion of text features and pre-trained text features, and the fused image features are obtained by fusion of image features and mapped image features.

[0096] S305. Perform cross-attention mechanism processing on the encoded text features and the encoded image features to obtain an enhanced tracking sequence vector.

[0097] For easier understanding, please refer to Figure 6 , Figure 6 A schematic diagram of a tracking sequence vector enhancement method based on text-image unidirectional prompt learning provided by an embodiment of the present invention. In this schematic diagram, the gray part constitutes a pre-trained feature interactor, the "learnable text feature vector" is a pre-trained text feature, the "text-image projection calculation" uses the first mapping unit, and the "text encoding" and "image encoding" use the text-image contrast learning model.

[0098] The following is an introduction to the tracking sequence vector enhancement method based on image-text unidirectional mapping. Based on this, the pre-trained feature interactor includes a second mapping unit and a pre-trained image feature. The updated tracking sequence vector is enhanced using the pre-trained feature interactor and the attribute description text corresponding to the object to be tracked, which may include: S401, obtaining text features, and extracting image features of the tracked object from the tracking sequence vector; wherein the text features are generated using attribute description text.

[0099] S402: Use a second mapping unit to project the pre-trained image features into mapped text features.

[0100] Different from the above-mentioned embodiment, in this embodiment, the pre-trained feature interactor only includes the second mapping unit and the pre-trained image features, so it is only necessary to map the pre-trained image features into mapped text features using the second mapping unit.

[0101] S403, fusing the text feature and the mapped text feature to obtain a fused text feature, and encoding the fused text feature using a pre-trained text-image contrast learning model to obtain an encoded text feature.

[0102] S404, fusing the image features and the pre-trained image features to obtain fused image features, and encoding the fused image features using the pre-trained text-image contrast learning model to obtain encoded image features.

[0103] In steps S403 and S404, different from steps S203 and S204, since this embodiment only performs one-way image-text mapping, the fused text features are obtained by fusion of text features and mapped text features, and the fused image features are obtained by fusion of image features and pre-trained image features.

[0104] S405. Perform cross-attention mechanism processing on the encoded text features and the encoded image features to obtain an enhanced tracking sequence vector.

[0105] For easier understanding, please refer to Figure 7 , Figure 7 A schematic diagram of a tracking sequence vector enhancement method based on image-text unidirectional prompt learning provided by an embodiment of the present invention. In this schematic diagram, the gray part constitutes a pre-trained feature interactor, the "learnable image feature vector" is a pre-trained image feature, the "image-text projection calculation" uses the second mapping unit, and the "text encoding" and "image encoding" use the text-image contrast learning model.

[0106] Based on the above embodiment, the following introduces the training method of the detection sequence vector, the pre-trained decoder, and the pre-trained feature interactor. Based on this, the method may also include: S501, obtaining multiple training images, category description texts corresponding to the training tracking objects, initial detection sequence vectors and tracking sequence vectors, and extracting training image features of the training images; the training images are marked with real position information of the training tracking objects, and the initial tracking sequence vectors are empty.

[0107] In this step, first, a plurality of training images may be obtained, and the real position information of the training tracking object has been annotated on these training images. Subsequently, feature extraction may be performed on these training images to obtain training image features.

[0108] In addition, in this step, the category description text, initial detection sequence vector and tracking sequence vector corresponding to the training tracking object can also be obtained. Among them, the initial detection sequence vector refers to an untrained detection sequence vector, which can be initialized by random initialization.

[0109] S502: Randomly initialize the initial detection sequence vector, and merge the category description text with the initial detection sequence vector into an initial enhanced detection sequence vector.

[0110] S503: Determine the predicted position information of the training tracking object in each training image by using the initial enhanced detection sequence vector, the tracking sequence vector, the training image features, the initial decoder and the initial feature interactor.

[0111] In this step, the initial enhanced detection sequence vector, tracking sequence vector, training image features, initial decoder and initial feature interactor can be used to determine the predicted position information of the training tracking object in each training image. The specific training process can refer to the relevant description of steps S103 and S104.

[0112] S504: Calculate the loss between the predicted position information and the actual position information using a preset loss function to obtain a loss value.

[0113] In this step, in order to ensure that the model can reliably determine the position of the object to be tracked in the image after training, it is necessary to use a preset loss function to calculate the loss value between the predicted position information and the actual position information, and use the loss value to update the parameters of the initial detection sequence vector, initial decoder, and initial feature interactor.

[0114] It should be noted that this embodiment does not limit the specific preset loss function, and can be set according to actual application requirements. For example, a possible preset loss function can be: ; in, represents the predicted value, represents the true value, N is the number of training images, represents the total number of objects in N training images, Represents a tracking sequence vector ( , track query) produces the predicted value of the tracked object in the i-th training image, represents the true value of the tracked object in the i-th training image, Represents the initial detection sequence vector ( , detect query) in the i-th training image, represents the true value of the newly added tracked object in the i-th training image. The loss value between the predicted value and the true value in a single frame image is: ; in, Represents the classification loss between the predicted value and the true value, Represents the L1 loss between the predicted value and the true value, Represents the generalized IoU loss between the predicted value and the true value. , , are all hyperparameters.

[0115] S505. Use the loss value to update the parameters of the initial detection sequence vector, the initial decoder, and the initial feature interactor to obtain the detection sequence vector, the pre-trained decoder, and the pre-trained feature interactor.

[0116] It should be noted that, in this embodiment, steps S503 to S505 may be executed multiple times according to the training situation until the model converges.

[0117] Based on the above embodiment, the target tracking method is fully introduced below based on a specific schematic diagram. Figure 8 , Figure 8 A schematic diagram of a system provided by an embodiment of the present invention. For the input video image frame t, the primary image features are first extracted through the backbone network. The backbone network here is a CNN convolutional neural network, which can be any convolutional neural network, such as Resnet50, Darknet, etc. The primary image features are then input into the image feature encoder for high-level image feature encoding to obtain global image features. The global image features, the spliced ​​detection sequence vectors, and the tracking sequence vectors are then input into the decoder for feature decoding to obtain the tracking information of the target, such as location information, tracking ID information, etc. Then, according to the tracking information of the target, the tracking trajectory is parsed and updated. Finally, according to the target trajectory information, the tracking sequence vector is calculated, and the calculated tracking sequence vector is used to update the historical tracking sequence vector for use in the video image parsing of the next frame.

[0118] In this invention, the detection sequence vector "detect query" is a set of special learnable parameters, each vector parameter represents a potential target object, and is optimized by back propagation during the training process. Figure 3First, according to the tracking category, input the text description object, such as when the tracking object is "person", the text description is "person", and then convert "person" into a vector of fixed dimension. For example, the WordPiecetokenization method can be used for text vector conversion. Subsequently, the initialized detection sequence vector is usually initialized to a random value, and the vector dimension is consistent with the text description encoding dimension. Then the text vector encoding and the initialized detection sequence vector are spliced, and the detection sequence vector based on text prompts is learned through linear change fusion calculation. Compared with the detection sequence vector based on single visual feature learning, in the present invention, text prompts are introduced to enhance the target detection expression ability of the detect query.

[0119] In the present invention, image feature coding can use the Transformer Encoder structure to learn high-level image features, with the input being the primary image feature vector and the spatial position coding vector, and the output being the encoded high-level image feature vector. In the feature calculation process, the self-attention learning coding mode is used to facilitate the network to better extract the relationship between different objects at different positions. The calculation process is as follows Figure 2 As shown, N represents repeating the calculation within the dashed box N times, N=6.

[0120] In the present invention, feature decoding is used to analyze the object to be tracked in the current image frame, and obtain the location information, category information, confidence score of the corresponding target, and track ID information of the target. In the present invention, the feature decoding process can use Transformer Decoder to perform feature decoding, and the calculation process is as follows: Figure 4 As shown. First, the self-attention mechanism is used to perform relevant calculations on the concatenated detection sequence vector and tracking sequence vector. Then, based on the position encoding vector and the high-level image feature vector, the cross-attention mechanism is used to perform relevant calculations. This process is repeated N times, and finally the decoded tracking target related information is output. Figure 4 In the figure, N means repeating the calculation in the gray dashed box N times, N=6.

[0121] After feature decoding, relevant information of the target object is obtained, and the tracking trajectory is updated according to the relevant information of the target object. In the present invention, tracking trajectory update means obtaining the target object of the current frame according to feature decoding, and determining the trajectory state of the target object, which is divided into the following three situations: New track: When the target object has no assigned track ID number and the target confidence score is greater than the preset threshold, it indicates that the target object is a new detection target and a new track ID number is assigned to the target object.

[0122] Track disappearance: When the detection confidence score of the existing track object in the current frame is less than the preset threshold, and the time less than the preset threshold is greater than the preset time, it indicates that the target track has left the monitoring range and the target track will be eliminated.

[0123] Continuous tracking of tracks: When the detection confidence score of an existing track object in the current frame is greater than the preset threshold, the track object is retained and tracking continues.

[0124] After obtaining the latest tracking trajectory information, the system will update the tracking sequence vector based on the tracking result. In order to enhance the tracking feature expression ability of the tracking sequence vector, the present invention designs a tracking sequence vector calculation method based on text-image bidirectional prompt learning. The calculation process is as follows: Figure 5 shown.

[0125] In the present invention, a tracking text prompt library is introduced. The tracking text prompt library is used as a priori knowledge to describe the tracking target object through text information. The advantage of using text information to describe the target object is that the text description, as a more advanced semantic feature, can effectively deal with the influence of noise such as scale change, deformation, and illumination change, and can more robustly extract the semantic features of the tracking object for use in tracking trajectory recognition. When establishing a tracking text prompt library, a description text that is conducive to tracking and recognition can be designed based on the priori knowledge of the tracking category. For example, when the tracking object is a pedestrian, in the process of pedestrian tracking, the pedestrian attributes commonly used for identification are: gender (male, female), age (adult, child), clothing color, accessories (bags, bicycles, etc.), etc.

[0126] In this multi-target tracking system, the tracking sequence vector is calculated by integrating text features and image features. During the model training process, the text feature vector T consists of three parts: the text features obtained after encoding the tracking text prompt library , a learnable text feature vector , the mapped text feature vector The image feature vector V also consists of three parts: the image feature vector obtained by tracking the target , the learnable image feature vector , the mapped image feature vector In the following formulas, [] indicates concatenation.

[0127] ; ; Among them, the learnable text feature vector and the learnable image feature vector The dimension size can be set according to the actual situation, and the learnable text feature vector can be distributed through Gaussian distribution. and the learnable image feature vector Initialize and optimize learning through back propagation during training.

[0128] Mapped text feature vector The learnable image feature vector Through the coupling function It is calculated that in the present invention, the coupling function It can be composed of a multi-layer perceptron (MLP) structure. Similarly, the image feature vector after mapping By learnable text feature vector Through the coupling function It is calculated that in the present invention, the coupling function It consists of a multi-layer perceptron (MLP) structure.

[0129] ; ; ; Finally, the image feature vector V and the text feature vector T are input into the visual encoder respectively. (visionencoder) and text encoder (text encoder) generates a new feature vector and calculates the new tracking query vector through the cross attention mechanism crossAttn (cross attention) It is used for target tracking in the next frame. In the present invention, the visual encoder and the text encoder can be directly calculated using the pre-trained CLIP model parameters.

[0130] Therefore, in the training process of the tracking model, the model parameters that need to be learned and trained for the track query update module are: the learnable text feature vector , learnable image feature vector , text-image projection coupling function Parameters, image-text projection coupling function The remaining model parameters can be obtained by freezing the pre-trained model parameters.

[0131] In this invention, a text-image bidirectional prompt learning mechanism is used to update the trackquery. Coupling function and As a bridge, it effectively connects text-image features. With the help of CLIP's multimodal learning model, it brings the representation features between text and image closer, and enhances the feature expression ability of the tracking target from both semantic and visual aspects, so as to improve the generalization ability of the tracking system in different scenarios, which is conducive to the deployment of multi-target tracking systems in different monitoring scenarios.

[0132] The target tracking device, electronic device, computer-readable storage medium, and computer program product provided in the embodiments of the present invention are introduced below. The target tracking device, electronic device, computer-readable storage medium, and computer program product described below can be referenced to the target tracking method described above.

[0133] Please refer to Fig. 9 , Fig. 9 A structural block diagram of a target tracking device provided by an embodiment of the present invention, the device may include: The acquisition module 901 is used to acquire the global image features of the image to be detected, the category description text corresponding to the object to be tracked, the detection sequence vector and the tracking sequence vector; the detection sequence vector records the detection information of the object to be tracked, and the tracking sequence vector records the image features of the tracked object; A detection sequence enhancement module 902, configured to fuse the category description text with the detection sequence vector into an enhanced detection sequence vector; The tracking detection module 903 is used to decode the global image features, the enhanced detection sequence vectors, and the tracking sequence vectors using a pre-trained decoder to obtain tracking information of the object to be tracked in the image to be detected; the tracking information includes at least position information and image features; The tracking sequence enhancement module 904 is used to update the tracking sequence vector using the tracking information, and enhance the updated tracking sequence vector using the pre-trained feature interactor and the attribute description text corresponding to the object to be tracked, so as to continue to track the target using the enhanced tracking sequence vector.

[0134] Optionally, the detection sequence enhancement module 902 may include: The text encoding submodule is used to encode the category description text to obtain the category description text vector; The enhancement submodule is used to concatenate the category description text vector with the detection sequence vector, and perform linear transformation and normalization processing on the concatenation result in turn to obtain an enhanced detection sequence vector.

[0135] Optionally, the pre-trained feature interactor includes a first mapping unit, a second mapping unit, a pre-trained text feature, and a pre-trained image feature; the tracking sequence enhancement module 904 may include: The first acquisition submodule is used to acquire text features and extract image features of the tracked object from the tracking sequence vector; wherein the text features are generated using attribute description text; A first mapping submodule, configured to project the pre-trained text features into mapped image features using the first mapping unit, and project the pre-trained image features into mapped text features using the second mapping unit; The first fusion submodule is used to fuse the text features, the pre-trained text features, and the mapped text features to obtain fused text features, and encode the fused text features using the pre-trained text-image contrast learning model to obtain encoded text features; The second fusion submodule is used to fuse the image features, the pre-trained image features, and the mapped image features to obtain fused image features, and encode the fused image features using the pre-trained text-image contrast learning model to obtain encoded image features; The first enhancement submodule is used to perform cross-attention mechanism processing on the encoded text features and the encoded image features to obtain an enhanced tracking sequence vector.

[0136] Optionally, the first mapping unit and the second mapping unit are both multi-layer perceptrons.

[0137] Optionally, obtaining submodules may include: The text feature acquisition unit is used to acquire text features of a category corresponding to the category description text in a tracking text prompt library according to the category description text; the tracking text prompt library contains text features corresponding to various objects to be tracked.

[0138] Optionally, the device may further include: The attribute description text acquisition module is used to obtain the attribute description text corresponding to the object to be tracked; The attribute description text encoding module is used to encode the attribute description text to obtain text features; The prompt library saving module is used to establish a correspondence between the text features and the category of the object to be tracked, and save the correspondence to the tracking text prompt library.

[0139] Optionally, the pre-trained feature interactor includes a first mapping unit and pre-trained text features; the tracking sequence enhancement module 904 may include: The second acquisition submodule is used to acquire text features and extract image features of the tracked object from the tracking sequence vector; wherein the text features are generated using attribute description text; A second mapping submodule, configured to project the pre-trained text features into mapped image features using the first mapping unit; The third fusion submodule is used to fuse the text feature and the pre-trained text feature to obtain a fused text feature, and encode the fused text feature using the pre-trained text-image contrast learning model to obtain an encoded text feature; The fourth fusion submodule is used to fuse the image features and the mapped image features to obtain fused image features, and encode the fused image features using a pre-trained text-image contrast learning model to obtain encoded image features; The second enhancer module is used to perform cross-attention mechanism processing on the encoded text features and the encoded image features to obtain an enhanced tracking sequence vector.

[0140] Optionally, the pre-trained feature interactor includes a second mapping unit and pre-trained image features; the tracking sequence enhancement module 904 may include: The third acquisition submodule is used to acquire text features and extract image features of the tracked object from the tracking sequence vector; wherein the text features are generated using attribute description text; A third mapping submodule, configured to project the pre-trained image features into mapped text features using the second mapping unit; A fifth fusion submodule is used to fuse the text feature and the mapped text feature to obtain a fused text feature, and encode the fused text feature using a pre-trained text-image contrast learning model to obtain an encoded text feature; a sixth fusion submodule, for fusing the image features and the pre-trained image features to obtain fused image features, and encoding the fused image features using the pre-trained text-image contrast learning model to obtain encoded image features; The third enhancement submodule is used to perform cross-attention mechanism processing on the encoded text features and the encoded image features to obtain an enhanced tracking sequence vector.

[0141] Optionally, the acquisition module 901 may include: The primary feature extraction submodule is used to extract features from the image to be detected using a pre-trained convolutional neural network model to obtain primary image features; A first spatial encoding submodule is used to perform spatial position encoding on the primary image feature according to the spatial position corresponding to the primary image feature in the image to be detected, so as to obtain an encoded primary image feature; The first processing submodule is used to use the encoded primary image features as the first key value and the first query value, and the primary image features as the first numerical value, use the pre-trained image feature encoder to perform a self-attention mechanism on the first key value, the first query value and the first numerical value to obtain a first processing result, and use the pre-trained image feature encoder to perform forward propagation calculation on the first processing result to obtain a global image feature.

[0142] Optionally, the tracking detection module 903 may include: A concatenation submodule is used to concatenate the enhanced detection sequence vector and the tracking sequence vector to obtain an object sequence vector; A second spatial encoding submodule is used to perform spatial position encoding on the global image feature according to the spatial position corresponding to the global image feature in the image to be detected to obtain an encoded global image feature; A third processing submodule is used to use the pre-trained decoder to perform a self-attention mechanism on the object sequence vector to obtain a second processing result; The fourth processing submodule is used to use the second processing result, the encoded global image feature and the global image feature as the second query value, the second key value and the second numerical value in turn, use the pre-trained decoder to perform an attention mechanism on the second query value, the second key value and the second numerical value to obtain a third processing result, and use the pre-trained decoder to perform forward propagation calculation on the third processing result to obtain tracking information.

[0143] Optionally, the tracking information further includes a tracking state category of the object to be tracked and a confidence level corresponding to the tracking state category; the tracking sequence enhancement module 904 includes: A new processing submodule is added, for, if it is determined according to the tracking state category that the object to be tracked belongs to a newly-added tracking object, then when the confidence of the object to be tracked is greater than a first preset threshold, adding the image features of the object to be tracked to the tracking sequence vector; The update exit processing submodule is used to, if it is determined according to the tracking state category that the object to be tracked belongs to a tracked object, move the image features of the object to be tracked out of the tracking sequence vector when the confidence of the object to be tracked is less than a second preset threshold, or update the tracking sequence vector using the image features of the object to be tracked when the confidence of the object to be tracked is not less than the second preset threshold.

[0144] Optionally, the device may further include: An initialization module is used to obtain multiple training images, category description texts corresponding to the training tracking objects, initial detection sequence vectors and tracking sequence vectors, and extract training image features of the training images; the training images are annotated with the real position information of the training tracking objects, and the initial tracking sequence vector is empty; An initial detection sequence enhancement module is used to randomly initialize the initial detection sequence vector and fuse the category description text with the initial detection sequence vector into an initial enhanced detection sequence vector; A training module, used for determining predicted position information of a training tracking object in each training image using an initial enhanced detection sequence vector, a tracking sequence vector, training image features, an initial decoder and an initial feature interactor; A loss value calculation module is used to calculate the loss between the predicted position information and the actual position information using a preset loss function to obtain a loss value; The parameter updating module is used to update the parameters of the initial detection sequence vector, the initial decoder, and the initial feature interactor using the loss value to obtain the detection sequence vector, the pre-trained decoder, and the pre-trained feature interactor.

[0145] Please refer to Fig.10 , Fig.10 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. The embodiment of the present invention provides an electronic device 10, including a processor 11 and a memory 12; wherein the memory 12 is used to store a computer program; the processor 11 is used to execute the target tracking method provided in the aforementioned embodiment when executing the computer program.

[0146] For the specific process of the above target tracking method, reference may be made to the corresponding contents provided in the aforementioned embodiments, which will not be described in detail here.

[0147] Furthermore, the memory 12, as a carrier for storing resources, may be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the storage method may be temporary storage or permanent storage.

[0148] In addition, the electronic device 10 also includes a power supply 13, a communication interface 14, an input / output interface 15 and a communication bus 16; wherein the power supply 13 is used to provide working voltage for each hardware device on the electronic device 10; the communication interface 14 can create a data transmission channel between the electronic device 10 and an external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present invention, and is not specifically limited here; the input / output interface 15 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0149] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the target tracking method provided in the above embodiment when the computer program / instruction is executed by a processor.

[0150] Since the embodiments of the computer program product part correspond to the embodiments of the target tracking method part, the embodiments of this part refer to the description of the embodiments of the target tracking method part, which will not be repeated here.

[0151] An embodiment of the present invention further provides a non-volatile computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the target tracking method provided in the above embodiment is implemented.

[0152] Since the embodiments of the non-volatile computer-readable storage medium part correspond to the embodiments of the target tracking method part, the embodiments of this part refer to the description of the embodiments of the target tracking method part, which will not be repeated here.

[0153] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0154] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0155] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0156] The above is a detailed introduction to a target tracking method, device, electronic device, program product and medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A target tracking method, characterized in that: include: Obtaining global image features of the image to be detected, category description text corresponding to the object to be tracked, a detection sequence vector, and a tracking sequence vector; The detection sequence vector records the detection information of the object to be tracked, and the tracking sequence vector records the image features of the tracked object; Merging the category description text with the detection sequence vector into an enhanced detection sequence vector; Using a pre-trained decoder to decode the global image features, the enhanced detection sequence vector, and the tracking sequence vector, to obtain tracking information of the object to be tracked in the image to be detected; the tracking information includes at least position information and image features; The tracking sequence vector is updated using the tracking information, and the updated tracking sequence vector is enhanced using a pre-trained feature interactor and an attribute description text corresponding to the object to be tracked, so as to continue to track the target using the enhanced tracking sequence vector.

2. The target tracking method according to claim 1, characterized in that: The method further comprises: fusing the category description text with the detection sequence vector into an enhanced detection sequence vector, comprising: Encoding the category description text to obtain a category description text vector; The category description text vector is concatenated with the detection sequence vector, and the concatenated result is sequentially subjected to linear transformation processing and normalization processing to obtain the enhanced detection sequence vector.

3. The target tracking method according to claim 1, characterized in that: The pre-trained feature interactor comprises a first mapping unit, a second mapping unit, a pre-trained text feature, and a pre-trained image feature; Performing enhancement processing on the updated tracking sequence vector using the pre-trained feature interactor and the attribute description text corresponding to the object to be tracked, including: Acquire text features, and extract image features of the tracked object from the tracking sequence vector; wherein the text features are generated using the attribute description text; Using the first mapping unit to project the pre-trained text features into mapped image features, and using the second mapping unit to project the pre-trained image features into mapped text features; The text feature, the pre-trained text feature, and the mapped text feature are fused to obtain a fused text feature, and the fused text feature is encoded using a pre-trained text-image contrast learning model to obtain an encoded text feature; The image features, pre-trained image features, and mapped image features are fused to obtain fused image features, and the fused image features are encoded using a pre-trained text-image contrast learning model to obtain encoded image features; The encoded text features and the encoded image features are processed through a cross-attention mechanism to obtain an enhanced tracking sequence vector.

4. The target tracking method according to claim 3, characterized in that: The first mapping unit and the second mapping unit are both multi-layer perceptrons.

5. The target tracking method according to claim 3, characterized in that: Get text features, including: According to the category description text, a text feature of a category corresponding to the category description text is obtained in a tracking text prompt library; the tracking text prompt library contains text features corresponding to various objects to be tracked.

6. The target tracking method according to claim 5, characterized in that: Also includes: Obtaining the attribute description text corresponding to the object to be tracked; Encoding the attribute description text to obtain the text feature; A correspondence is established between the text feature and the category of the object to be tracked, and the correspondence is saved in the tracking text prompt library.

7. The target tracking method according to claim 1, characterized in that: The pre-trained feature interactor comprises a first mapping unit and pre-trained text features; Performing enhancement processing on the updated tracking sequence vector using the pre-trained feature interactor and the attribute description text corresponding to the object to be tracked, including: Acquire text features, and extract image features of the tracked object from the tracking sequence vector; wherein the text features are generated using the attribute description text; Projecting the pre-trained text features into mapped image features using the first mapping unit; The text feature and the pre-trained text feature are fused to obtain a fused text feature, and the fused text feature is encoded using a pre-trained text-image contrast learning model to obtain an encoded text feature; The image features and the mapped image features are fused to obtain fused image features, and the fused image features are encoded using a pre-trained text-image contrast learning model to obtain encoded image features; The encoded text features and the encoded image features are processed through a cross-attention mechanism to obtain an enhanced tracking sequence vector.

8. The target tracking method according to claim 1, characterized in that: The pre-trained feature interactor comprises a second mapping unit and pre-trained image features; Performing enhancement processing on the updated tracking sequence vector using the pre-trained feature interactor and the attribute description text corresponding to the object to be tracked, including: Acquire text features, and extract image features of the tracked object from the tracking sequence vector; wherein the text features are generated using the attribute description text; Projecting the pre-trained image features into mapped text features using the second mapping unit; The text feature and the mapped text feature are fused to obtain a fused text feature, and the fused text feature is encoded using a pre-trained text-image contrast learning model to obtain an encoded text feature; The image feature and the pre-trained image feature are fused to obtain a fused image feature, and the fused image feature is encoded using a pre-trained text-image contrast learning model to obtain an encoded image feature; The encoded text features and the encoded image features are processed through a cross-attention mechanism to obtain an enhanced tracking sequence vector.

9. The target tracking method according to claim 1, characterized in that: Obtain the global image features of the image to be detected, including: Using a pre-trained convolutional neural network model to extract features from the image to be detected, to obtain primary image features; According to the spatial position corresponding to the primary image feature in the image to be detected, the primary image feature is spatially encoded to obtain an encoded primary image feature; The encoded primary image feature is used as a first key value and a first query value, and the primary image feature is used as a first numerical value. A pre-trained image feature encoder is used to perform a self-attention mechanism on the first key value, the first query value and the first numerical value to obtain a first processing result, and the pre-trained image feature encoder is used to perform a forward propagation calculation on the first processing result to obtain the global image feature.

10. The target tracking method according to claim 1, characterized in that: The global image feature, the enhanced detection sequence vector, and the tracking sequence vector are decoded together by using a pre-trained decoder to obtain tracking information of the object to be tracked in the image to be detected, including: Concatenating the enhanced detection sequence vector and the tracking sequence vector to obtain an object sequence vector; According to the spatial position corresponding to the global image feature in the image to be detected, the global image feature is spatially encoded to obtain an encoded global image feature; Using the pre-trained decoder to perform self-attention mechanism processing on the object sequence vector to obtain a second processing result; The second processing result, the encoded global image feature and the global image feature are used as the second query value, the second key value and the second numerical value in turn, the pre-trained decoder is used to perform an attention mechanism on the second query value, the second key value and the second numerical value to obtain a third processing result, and the pre-trained decoder is used to perform forward propagation calculation on the third processing result to obtain the tracking information.

11. The target tracking method according to claim 1, characterized in that: The tracking information also includes the tracking state category of the object to be tracked and the confidence level corresponding to the tracking state category; Updating the tracking sequence vector using the tracking information includes: If it is determined according to the tracking state category that the object to be tracked is a newly added tracking object, then when the confidence of the object to be tracked is greater than a first preset threshold, adding the image feature of the object to be tracked to the tracking sequence vector; If it is determined according to the tracking state category that the object to be tracked belongs to a tracked object, then when the confidence of the object to be tracked is less than a second preset threshold, the image features of the object to be tracked are moved out of the tracking sequence vector, or when the confidence of the object to be tracked is not less than the second preset threshold, the tracking sequence vector is updated using the image features of the object to be tracked.

12. The target tracking method according to any one of claims 1 to 11, characterized in that: Also includes: Acquire multiple training images, category description texts corresponding to training tracking objects, initial detection sequence vectors and tracking sequence vectors, and extract training image features of the training images; The training image is annotated with the real position information of the training tracking object, and the initial tracking sequence vector is empty; Randomly initializing an initial detection sequence vector, and fusing the category description text with the initial detection sequence vector into an initial enhanced detection sequence vector; Determine the predicted position information of the training tracking object in each of the training images using the initial enhanced detection sequence vector, the tracking sequence vector, the training image features, the initial decoder and the initial feature interactor; Calculating the loss between the predicted position information and the actual position information using a preset loss function to obtain a loss value; The loss value is used to update the parameters of the initial detection sequence vector, the initial decoder, and the initial feature interactor to obtain the detection sequence vector, the pre-trained decoder, and the pre-trained feature interactor.

13. A target tracking device, characterized in that: include: An acquisition module, used to acquire global image features of an image to be detected, a category description text corresponding to an object to be tracked, a detection sequence vector and a tracking sequence vector; the detection sequence vector records detection information of the object to be tracked, and the tracking sequence vector records image features of a tracked object; A detection sequence enhancement module, used for fusing the category description text with the detection sequence vector into an enhanced detection sequence vector; A tracking detection module, configured to decode the global image features, the enhanced detection sequence vectors, and the tracking sequence vectors using a pre-trained decoder to obtain tracking information of the object to be tracked in the image to be detected; the tracking information includes at least position information and image features; The tracking sequence enhancement module is used to update the tracking sequence vector using the tracking information, and enhance the updated tracking sequence vector using a pre-trained feature interactor and an attribute description text corresponding to the object to be tracked, so as to continue to track the target using the enhanced tracking sequence vector.

14. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the target tracking method as claimed in any one of claims 1 to 12 when executing the computer program.

15. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the target tracking method according to any one of claims 1 to 12 is implemented.

16. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by the processor, the target tracking method according to any one of claims 1 to 12 is implemented.

Images