Target tracking method, device, equipment, medium and product

Abstract

The invention provides a target tracking method and device, equipment, a medium and a product, and the method comprises the steps: obtaining the feature data of each mode and the feature of an enhanced template, the feature data of each mode comprises a text feature corresponding to text data and an image feature corresponding to at least one image mode, the text data is used for describing a to-be-tracked target in at least one image mode; based on the feature data of each mode and an enhanced template feature target tracking result; the enhanced template features are obtained by performing feature fusion on the text features and template image features corresponding to the at least one image mode, and the template image features are features of a template image assisting target tracking in the at least one image mode. According to the invention, the target tracking accuracy can be improved.

Description

Technical Field

[0001] This application relates to the field of computer vision, and more particularly to a target tracking method, apparatus, device, medium, and product. Background Technology

[0002] Accurate target tracking is an important task in many fields such as security monitoring, industrial inspection, and ecological protection.

[0003] Currently, the main approach is to utilize the complementary advantages between some visual modalities to locate and track arbitrary targets in video sequences.

[0004] However, this visual modality-based target tracking method has limitations in terms of target representation and flexibility, resulting in low target tracking accuracy. Summary of the Invention

[0005] Based on the above-mentioned technological status, this application provides a target tracking method, apparatus, device, medium, and product that can improve the accuracy of target tracking.

[0006] To achieve the above-mentioned technical objectives, this application proposes the following technical solution:

[0007] According to a first aspect of the embodiments of this application, a target tracking method is provided, comprising: acquiring feature data and enhanced template features for each modality, wherein the feature data for each modality includes text features corresponding to text data and image features corresponding to at least one image modality, wherein the text data is used to describe a target to be tracked in the at least one image modality; performing target tracking based on the feature data and enhanced template features for each modality to obtain a target tracking result; wherein the enhanced template features are obtained by feature fusion of the text features and template image features corresponding to each of the at least one image modality, and the template image features are features of a template image used to assist in target tracking in the at least one image modality.

[0008] In some implementations, the step of performing target tracking based on the features of each modality and the enhanced template features to obtain the target tracking result includes: performing feature association on the feature data of each modality and the enhanced template features to obtain fused features; and performing target tracking based on the fused features to obtain the target tracking result.

[0009] In some implementations, the at least one image modality corresponds to text features; wherein, the step of associating the feature data of each modality with the enhanced template features to obtain fused features includes: for each image modality in the at least one image modality, establishing an association relationship between the image features of that image modality and the corresponding text features to obtain the associated features corresponding to that image modality; and obtaining the fused features based on the association between the associated features and the enhanced template features corresponding to each of the at least one image modality.

[0010] In some implementations, establishing the association between image features and corresponding text features under the image modality to obtain the associated features corresponding to the image modality includes: determining the similarity between the text features and each image region feature in the image features; using each similarity as a weight coefficient corresponding to each image region feature, and performing a weighted summation on each image region feature in the image features to obtain the associated features of the image modality.

[0011] In some implementations, obtaining the fused features based on the association between the associated features and the enhanced template features corresponding to each of the at least one image modality includes: establishing the association relationship between the associated features and the enhanced template features corresponding to each of the at least one image modality to obtain the enhanced associated features corresponding to each of the at least one image modality; and performing feature fusion on the enhanced associated features corresponding to each of the at least one image modality to obtain the fused features.

[0012] In some implementations, establishing the association relationship between the associated features and the enhanced template features corresponding to each of the at least one image modalities to obtain the enhanced associated features corresponding to each of the at least one image modalities includes: for each image modality in the at least one image modalities, determining the similarity between the enhanced template features and the associated features of each image modality; using each similarity as a weight coefficient corresponding to each image region feature, and performing a weighted summation of the image region features to obtain the enhanced associated features corresponding to the image modality.

[0013] In some implementations, the at least one image modality includes: a visible light modality and / or a thermal infrared modality; when the at least one image modality includes the visible light modality and the thermal infrared modality, the text features include text features corresponding to at least one of the visible light modality and the thermal infrared modality, and the enhanced template features are obtained by feature fusion of the text features corresponding to at least one of the visible light modality and the thermal infrared modality with the template image features corresponding to each of the at least one image modality.

[0014] In some implementations, the method is executed through a target tracking model, which is trained using the following steps: acquiring feature samples and enhanced template feature samples for each modality, wherein the feature samples for each modality include text feature samples corresponding to training text and image feature samples corresponding to at least one image modality, the training text being used to describe the target to be tracked in at least one image modality; performing target prediction using the target tracking model to be trained based on the feature samples and enhanced template feature samples of each modality to obtain a target prediction result; iteratively training the target tracking model to be trained based on the difference between the target prediction result and the real target annotation result to obtain the target tracking model; wherein the enhanced template feature samples are obtained by feature fusion of the text feature samples and the template image feature samples corresponding to each of the at least one image modality, the template image feature samples being features of template image samples that assist in target tracking in the at least one image modality.

[0015] According to a second aspect of the present application, a target tracking apparatus is provided, comprising: an acquisition unit, configured to acquire feature data and enhanced template features for each modality, wherein the feature data for each modality includes text features corresponding to text data and image features corresponding to at least one image modality, wherein the text data is used to describe a target to be tracked in the at least one image modality; and a target tracking unit, configured to perform target tracking based on the feature data and enhanced template features of each modality to obtain a target tracking result; wherein the enhanced template features are obtained by feature fusion of the text features and template image features corresponding to each of the at least one image modality, and the template image features are features of a template image used to assist in target tracking in the at least one image modality.

[0016] According to a third aspect of the present application, an electronic device is provided, including a memory and a processor; the memory is connected to the processor and is used to store a program; the processor is used to implement the target tracking method as described in the first aspect by running the program in the memory.

[0017] According to a fourth aspect of the embodiments of this application, a storage medium is provided, on which a computer program is stored, and when the computer program is run by a processor, it implements the target tracking method as described in the first aspect.

[0018] According to a fifth aspect of the present application, a computer program product is provided, including computer program instructions that, when executed by a processor, cause the processor to perform: the target tracking method as described in the first aspect.

[0019] This application provides a target tracking method, apparatus, device, medium, and product. It acquires multimodal feature data and enhanced template features. The multimodal feature data includes text features extracted from text data describing the target to be tracked, and image features corresponding to at least one image modality containing the target. Target tracking is performed based on the multimodal feature data and enhanced template features to obtain a target tracking result. The enhanced template features are obtained by feature fusion of the text features and template image features corresponding to each of the at least one image modality. The template image features refer to the feature representation of a template image used to assist in target tracking in the corresponding image modality. Because the template image features incorporate complementary information about the target from the text modality and at least one image modality, they have a stronger target representation capability. Therefore, when applied to target tracking tasks, they can improve the accuracy of target identification and localization, thereby improving the overall accuracy of target tracking. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0021] Figure 1 A flowchart illustrating a target tracking method provided in an embodiment of this application;

[0022] Figure 2 An architecture diagram of the target tracking model provided in the embodiments of this application;

[0023] Figure 3 A flowchart illustrating target tracking based on feature data and enhanced template features from various modalities, provided in this application embodiment;

[0024] Figure 4 A flowchart illustrating a training method for a target tracking model provided in an embodiment of this application;

[0025] Figure 5 A schematic diagram illustrating the model training process provided in the embodiments of this application;

[0026] Figure 6 This is a schematic diagram of the structure of a target tracking device provided in an embodiment of this application;

[0027] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0028] The technical solutions proposed in this application are applicable to various scenarios such as intelligent driving, monitoring and detection, and motion analysis.

[0029] The technical solutions provided in this application can be applied, by way of example, to hardware devices such as processors, electronic devices, and servers (including cloud servers), or packaged into software programs for execution. When the hardware device executes the processing procedure of the technical solutions in this application, or when the aforementioned software program is run, the target task can be automatically split and the application programming interfaces required by the task can be automatically invoked to achieve the purpose of the target task. This application only provides illustrative descriptions of the specific processing procedure of the technical solutions in this application and does not limit the specific implementation form of the technical solutions in this application. Any technical implementation form that can execute the processing procedure of the technical solutions in this application can be adopted by this application.

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] Before introducing the solution proposed in this application, the relevant technologies will first be introduced:

[0032] In the process of locating and tracking any target in a video sequence by leveraging the complementary advantages of different visual modalities, it is usually necessary to effectively fuse the information from these visual modalities. Taking visible light modalities and thermal infrared modalities as examples, common fusion schemes currently include the following:

[0033] (1) Through the learnable weight calculation module, the feature representations under visible light mode and thermal infrared mode are assigned corresponding weights respectively, and the fused features are obtained by weighted summation, thereby enhancing the target's expressive ability in complex scenarios.

[0034] (2) Divide the dataset according to the attribute information and train the corresponding attribute network to improve the model’s ability to learn and express different attribute features, thereby enhancing the overall recognition and description of the target.

[0035] (3) Adopt the self-attention mechanism or cross-attention mechanism in the Transformer architecture to realize adaptive modeling and information interaction between modalities, thereby further improving the multimodal fusion effect and the richness of target expression.

[0036] However, the above methods mainly rely on the fusion between visual modalities, which still has certain limitations in the representation of target features, resulting in low target tracking accuracy.

[0037] In recent years, with the continuous development of natural language processing technology, more natural and efficient human-computer interaction methods have gradually attracted increasing attention. As a highly expressive information carrier, natural language can not only accurately convey specific factual information, but also has the ability to express abstract concepts and complex semantics. By introducing natural language descriptive references to capture the appearance features and behavioral patterns of targets, it is helpful to improve the recognition accuracy and interactive experience of target tracking systems.

[0038] In view of this, the embodiments of this application aim to provide a target tracking method, apparatus, device, medium, and product, which, by combining natural language information with image data from various image modalities, more effectively mines the complementary characteristics between multimodalities, promotes the deep fusion of cross-modal information, and thus improves the accuracy of target tracking. These will be described in detail in the following embodiments.

[0039] Exemplary methods

[0040] Figure 1 This is a flowchart illustrating a target tracking method provided in an embodiment of this application. Figure 1 As shown, the target tracking method provided in this embodiment includes steps S101-S102:

[0041] S101. Obtain feature data and enhanced template features for each modality.

[0042] The feature data for each modality includes text features corresponding to text data and image features corresponding to at least one image modality.

[0043] Text data is used to describe a target to be tracked in at least one image modality. Text features can be obtained by feature extraction from the text data. Text data can include descriptions of the target's appearance, behavior, or other salient characteristics using natural language. In surveillance video analysis, text data can be textual descriptions of people or objects appearing in the video.

[0044] Each modality includes at least one text modality and at least one image modality. The relationship between the text modality and the image modality can be flexibly set. For example, in some application scenarios, each image modality's image data is accompanied by text data, forming a one-to-one correspondence; while in other cases, some image modalities in at least one image modality may correspond to a text modality.

[0045] In some examples, at least one image mode includes: a visible light mode and / or a thermal infrared mode. Accordingly, the image features corresponding to at least one image mode include: image features of a visible light image and / or image features of a thermal infrared image.

[0046] In cases where the image features corresponding to at least one image modality include the image features of a visible light image, the visible light image is accompanied by text data to describe the target to be tracked from feature dimensions such as color, shape, spatial location, and direction of motion.

[0047] In cases where the image features corresponding to at least one image modality include the image features of a thermal infrared image, the thermal infrared image is accompanied by text data to describe the target to be tracked from feature dimensions such as thermal radiation intensity and temperature distribution differences.

[0048] When the image features corresponding to at least one image modality include image features of a visible light image and image features of a thermal infrared image, the text features may include text features corresponding to the visible light image modality, text features corresponding to the thermal infrared image modality, or text features corresponding to both the visible light image modality and the thermal infrared image modality.

[0049] The enhanced template features are obtained by fusing text features and template image features corresponding to at least one image modality. The template image features are the features of the template image that assist in target tracking in at least one image modality.

[0050] Template images are typically reference images containing the target to be tracked. They reflect the target's appearance, structural information, or texture information in that modality, and are highly representative and recognizable. Template image features can be obtained by extracting features from the template image.

[0051] In a video sequence, after the target is located in the current frame, the target region corresponding to the location result is cropped from the current frame and used as a template image for target tracking in the next frame.

[0052] It is worth noting that in the first frame of the video, since no initial information about the target has been obtained, the template image is a blank image or in a default initial state. Subsequently, the initial template image is generated based on the target detection results in the first frame, and then the continuous tracking process is started.

[0053] In some embodiments, when obtaining feature data and enhanced template features for each modality in step S101, a target tracking model can be used. The target tracking model includes a text feature extraction module and an image feature extraction module; the text feature extraction module is used to extract text features from the text data; and the image feature extraction module is used to extract image features from the image data.

[0054] The following section uses at least one image mode, including visible light mode and thermal infrared mode, as an example to describe the specific implementation process of step S101 in conjunction with the architecture diagram of the target tracking model:

[0055] Figure 2 This is an architecture diagram of the target tracking model provided in an embodiment of this application. Figure 2 As shown, the target tracking model includes a first text feature extraction module 201, a visible light image feature extraction module 202, a second text feature extraction module 203, a thermal infrared image feature extraction module 204, and a first feature fusion module 205.

[0056] The first text feature extraction module 201 is used to extract first text features from the input first text data. The first text data is text data describing the target to be tracked in the visible light modal image using natural language.

[0057] The visible light image feature extraction module 202 is used to extract visible light image features from the input visible light image and to extract visible light template image features from the input visible light template image. The visible light template image can be obtained by target detection and localization based on the previous frame of the input visible light image.

[0058] The second text feature extraction module 203 is used to extract second text features from the input second text data. The second text data is text data describing the target to be tracked in the thermal infrared modal image using natural language.

[0059] The thermal infrared image feature extraction module 204 is used to extract thermal infrared image features from the input thermal infrared image and to extract thermal infrared template image features from the input thermal infrared template image. The thermal infrared template image can be obtained by target detection and localization based on the previous frame of thermal infrared image corresponding to the input thermal infrared image.

[0060] The first feature fusion module 205 is used to fuse the first text features, visible light template image features, second text features, and thermal infrared template image features to obtain enhanced template features. These enhanced template features are a joint feature representation of at least one image modality and a text modality, containing not only feature information from both visible light and thermal infrared image modalities but also semantic information from two text feature extraction modules. This multi-level, multi-perspective information fusion method enhances the expressive power of the template features, enabling them to more comprehensively and accurately describe various attributes of the target object. Specifically, text features can provide a high-level semantic description of the target to be tracked, helping to understand the target's function, purpose, or behavior; while image features capture the appearance details and structural features of the target from a visual perspective, ensuring recognition accuracy in complex environments, thus facilitating more stable and accurate target tracking performance in different application scenarios.

[0061] Both the first text feature extraction module 201 and the second text feature extraction module 203 can be implemented using a text encoder, such as the BERT (Bidirectional Encoder Representations from Transformers) model. BERT is a pre-trained language model based on the Transformer architecture, which has powerful contextual understanding and semantic expression capabilities. It can map text information into dense, continuous vector representations, thereby effectively capturing deep semantic information in the text.

[0062] Both the visible light image feature extraction module 202 and the thermal infrared image feature extraction module 204 can be implemented using an image encoder, such as the open-source Swin-Transformer model. Swin-Transformer is a vision model based on the Transformer architecture. Through window partitioning and self-attention mechanisms, it can capture local details and global structural information of images at different scales, thereby enhancing the model's feature extraction capabilities.

[0063] Taking a visible light image as an example, the image is first segmented into image blocks of a preset size (e.g., 4x4 pixels), and these blocks are flattened into vectors. These vectors are then mapped to a high-dimensional feature space through a linear embedding layer to obtain an initial token sequence. Next, the initial token sequence is reorganized into a two-dimensional grid to restore its original spatial position, and further divided into multiple fixed-size non-overlapping windows (e.g., each window contains 7x7 tokens). Within each non-overlapping window, a multi-head self-attention mechanism is used to calculate attention values ​​to capture the contextual relationships within the local region. To capture cross-window interactions, a sliding window approach can be used, where the current window is moved half a window size in a specific direction each time, and self-attention calculation is performed within the moved window. It is worth noting that when performing self-attention calculation on a new window, a mask matrix can be used to mask some content from the previous window, ensuring that it only focuses on other tokens in the new window. This process is repeated continuously to obtain the initial image feature representation.

[0064] The first feature fusion module 205 can be implemented using a linear transformation layer. The linear transformation layer learns a set of weight parameters to map the aforementioned multi-source features into a new feature space, thereby effectively integrating information from different modalities. For example, when processing the fusion task of text and image features, the linear transformation layer first performs a weighted combination of features from each modality, and then further enhances the model's expressive power through a non-linear activation function (such as ReLU). Furthermore, to improve the fusion effect, normalization operations or dropout techniques can be introduced on top of the linear transformation to increase the model's generalization ability and stability.

[0065] Continue reading Figure 1 After step S101, step S102 may also be included:

[0066] S102. Target tracking is performed based on the feature data of each modality and the enhanced template features to obtain the target tracking result.

[0067] Figure 3 This is a flowchart illustrating target tracking based on feature data and enhanced template features from various modalities, as provided in embodiments of this application. Figure 3 As shown, step S102 includes the following steps S301-S302:

[0068] S301. Perform feature association between the feature data of each modality and the enhanced template features to obtain fused features.

[0069] Step S301 aims to enhance feature interactions between different image modalities. In some embodiments, step S301 includes the following steps a1 and a2:

[0070] Step a1: For each image modality in at least one image modality, establish the association relationship between the image features and the corresponding text features under that image modality, and obtain the associated features corresponding to that image modality.

[0071] Taking image mode and thermal infrared mode as examples again, step a1 includes: establishing the association between visible light image features and the first text feature to obtain the associated feature corresponding to the visible light mode; establishing the association between thermal infrared image features and the second text feature to obtain the associated feature corresponding to the thermal infrared mode. The specific implementation process of step S301 will be described in detail below with reference to the accompanying drawings:

[0072] Continue reading Figure 2 The target tracking model also includes a first feature association module 206 and a second feature association module 207.

[0073] The first feature association module 206 is used to associate the image features of the visible light mode with the first text features to obtain the first associated feature, which is the association feature between the visible light image and the first text data.

[0074] The second feature association module 207 is used to associate the image features of the thermal infrared mode with the second text features to obtain the second associated feature, that is, the association feature between the thermal infrared image and the second text data.

[0075] Both the first feature association module 206 and the second feature association module 207 can be implemented using a cross attention mechanism, which aims to enhance the feature interaction between the two heterogeneous modalities of image modality and text modality.

[0076] Specifically, when step a1 is implemented using a cross-attention mechanism, it includes: determining the similarity between text features and image features of each image region; using each similarity as the weight coefficient corresponding to each image region feature, and performing a weighted summation of each image region feature in the image features to obtain the associated features of the image modality.

[0077] In step a1, when establishing the association between image features and corresponding text features in the image modality, the similarity between text features and image features of each image region can be determined to quantify the correlation between text description and different regions within the image. The similarity score can measure the importance of each image region to the text description.

[0078] Next, the calculated similarity scores are used as weight coefficients and applied to each image region feature within the image features. That is, each image region is assigned a corresponding weight based on the strength of its association with the text description. Image regions highly correlated with the text description receive higher weights, while regions with lower correlation are assigned lower weights.

[0079] Finally, the weighted summation of the image region features in the image features is performed using the weighted coefficients mentioned above to obtain the associated features under this image modality. This integrates the relationship between image information and text description, emphasizes the image parts that best represent or explain the text content, and generates an associated feature representation that is more focused on the target object and its text description.

[0080] Step a2: Based on the association between the associated features and the enhanced template features corresponding to at least one image modality, the fused features are obtained.

[0081] In some embodiments, step a2 includes the following steps b1 and b2:

[0082] Step b1: Establish the association relationship between the associated features and the enhanced template features corresponding to at least one image modality, and obtain the enhanced associated features corresponding to at least one image modality.

[0083] Continue reading Figure 2 The target tracking model also includes a third feature association module 208 and a fourth feature association module 209.

[0084] The third feature association module 208 is used to associate the first association feature with the enhanced template feature to obtain the first enhanced association feature.

[0085] The fourth feature association module 209 is used to associate the second association feature with the enhanced template feature to obtain the second enhanced association feature.

[0086] Both the third feature association module 206 and the fourth feature association module 207 can be implemented using a cross attention mechanism, which aims to enhance feature interaction between different image modalities.

[0087] Specifically, when step b1 is implemented using a cross-attention mechanism, it includes the following steps: for each image modality in at least one image modality, determine the similarity between the enhanced template features and the image region features in the associated features corresponding to that image modality; use each similarity as the weight coefficient corresponding to each image region feature, and perform a weighted summation of each image region feature to obtain the enhanced associated features corresponding to that image modality.

[0088] Taking the visible light modality as an example, firstly, by determining the similarity between the enhanced template features and the image region features in the first associated features, the correlation between different regions in the first associated features and the enhanced template features is quantified. The importance of the image region features of this modality to the enhanced template features can be measured by the similarity score.

[0089] Next, the calculated similarity scores are used as weighting coefficients and applied to each image region feature of this image modality. That is, each image region feature is assigned a corresponding weight based on the strength of its correlation with the enhancement template feature. Image region features that are highly correlated with the enhancement template feature will receive higher weights, while those with lower correlation will be assigned lower weights.

[0090] Finally, the weighted summation of the features of each image region in the image modality is performed using the aforementioned weighting coefficients to obtain the third association feature in the image modality. This emphasizes the image parts that best represent or explain the text content, thereby generating an enhanced association feature representation that is more focused on the target object.

[0091] In this embodiment, enhanced template features are used as queries to retrieve relevant information about the target to be tracked from the first associated features and the second associated features, respectively. Since the enhanced template features combine at least one image modality and text modality as a joint template, they can more comprehensively represent the information of the target to be tracked, realize the transmission and enhancement of the contextual information of the target to be tracked, and subsequently, when using the enhanced template features to query the target to be tracked from the first associated features corresponding to the visible light modality and the second associated features corresponding to the thermal infrared modality, more attention can be paid to the information related to the target to be tracked, and the contextual information of the target will be better represented.

[0092] Step b2: Perform feature fusion on the enhanced correlation features corresponding to at least one image modality to obtain fused features.

[0093] Wherein, at least one image modality corresponds to an enhanced correlation feature, including: a first enhanced correlation feature corresponding to the visible light modality and a second enhanced correlation feature corresponding to the thermal infrared modality. Then step b2 includes: fusing the first enhanced correlation feature corresponding to the visible light modality and the second enhanced correlation feature corresponding to the thermal infrared modality to obtain a fused feature.

[0094] Continue reading Figure 2The target tracking model also includes a second feature fusion module 210, which is used to fuse the first enhanced correlation feature and the second enhanced correlation feature to obtain fused features. In the specific implementation process, the first enhanced correlation feature and the second enhanced correlation feature corresponding to the thermal infrared mode are first concatenated, and then input into the second feature fusion module 210 to obtain fused features.

[0095] In this embodiment, the second feature fusion module 210 can be implemented using a multilayer perceptron (MLP) to perform nonlinear transformations and information integration on the stitched high-dimensional features, thereby improving the quality and expressive power of the fused features. (Continue reading...) Figure 3 After step S301, step S302 may also be included:

[0096] S302. Target tracking is performed based on fused features to obtain target tracking results.

[0097] Continue reading Figure 2 The target tracking model also includes a target decoder 211. This target decoder receives the fused features output by the second feature fusion module 210 and combines them with a pre-input target query to perform target decoding.

[0098] Specifically, the target decoder 211 uses the synergistic effect of self-attention and cross-attention mechanisms to interactively match target queries with fused features, thereby achieving state estimation and position prediction for each target to be tracked in the video sequence. Based on this, the target decoder 211 further performs classification prediction and bounding box regression prediction tasks, respectively, to identify the identity category (such as pedestrians, vehicles, etc.) of each target in the current frame and its spatial location coordinates in the image (usually represented in the form of normalized [x_center, y_center, width, height] or absolute pixel coordinates [x_min, y_min, x_max, y_max]).

[0099] Finally, the structured prediction result output by the target decoder 211 is the target tracking result, which includes the category information, two-dimensional image coordinate position and corresponding target identifier (ID) of each tracked target, thereby realizing continuous tracking of multiple targets in consecutive video frames.

[0100] In this embodiment, the target query serves as the general input to the target decoder 211, and its output is driven by the fused features of the current frame. Specifically, the target query can be viewed as an "empty slot" or placeholder, which does not carry any prior information but dynamically fills semantic information by interacting with the fused features of the current frame.

[0101] For example, a target query can be initialized as a "blank image" or a set of randomly initialized vectors that do not contain specific target information. In practice, these query vectors enter the target decoder and are matched and interacted with the fused features of the current frame through self-attention and cross-attention mechanisms. This allows the target query to be updated based on the visual content in the current frame, thereby obtaining specific semantic information, such as the target's location, category, and bounding box coordinates.

[0102] exist Figure 2 In this context, HEAD represents the last layer of the network, which is usually a classifier or regressor.

[0103] It should be noted that before inputting the feature data of each modality into the first feature association module and the second feature association module, it is necessary to project them into a latent space with the same dimension through a linear projection layer.

[0104] In some embodiments, before applying the target tracking model for target tracking, it is necessary to train the target tracking model based on training samples. The training process of the target tracking model is described below with reference to the accompanying drawings:

[0105] Figure 4 This is a flowchart illustrating a training method for a target tracking model provided in an embodiment of this application. Figure 4 As shown, the training process of the target tracking model includes the following steps S401-S403:

[0106] S401. Obtain feature samples and enhanced template feature samples for each modality.

[0107] The feature samples for each modality include text feature samples corresponding to the training text and image feature samples corresponding to at least one image modality.

[0108] Training text is used to describe the target to be tracked in at least one image modality. Features of the training text can be obtained by feature extraction. Training text can include descriptions of the target's appearance, behavior, or other salient characteristics using natural language. In surveillance video analysis, training text can be a textual description of people or objects appearing in the video.

[0109] In some embodiments, the training text can be obtained by: providing auxiliary text descriptions for video images corresponding to at least one image modality based on a large visual language model; and then manually fine-tuning any inappropriate text descriptions.

[0110] The large visual language model can be implemented using InternVL2.5 (38B).

[0111] In the visible light mode, the text description primarily describes the color and texture characteristics of the target to be tracked in the video image. In some examples, the text description in the visible light mode might be: Tracking the head of a man wearing a silver-white cotton jacket.

[0112] In the thermal infrared mode, the text description primarily focuses on depicting the thermal radiation intensity and distribution differences of the target being tracked in the video image, thereby enabling deep binding between natural language description and multimodal images. In some examples, the text description in the thermal infrared mode might be: "The overall thermal radiation intensity of the human head is very high, with the face being particularly prominent, showing a significant difference in temperature distribution compared to the surrounding environment."

[0113] Each modality includes at least one text modality and at least one image modality. The relationship between the text modality and the image modality can be flexibly set. For example, in some application scenarios, the image data of each image modality is accompanied by training text, forming a one-to-one correspondence; while in other cases, some image modalities in at least one image modality may correspond to the text modality.

[0114] In some examples, at least one image mode includes: a visible light mode and / or a thermal infrared mode. Accordingly, the image feature samples corresponding to at least one image mode include: image feature samples of visible light image samples and / or image feature samples of thermal infrared image samples.

[0115] In the case where the image feature samples corresponding to at least one image modality include image feature samples of visible light images, the visible light image samples are accompanied by training text to describe the target from feature dimensions such as the target's color, shape, spatial location, and direction of motion.

[0116] In the case where the image feature samples corresponding to at least one image modality include image feature samples of thermal infrared images, the thermal infrared images are accompanied by training text to describe the target to be tracked from feature dimensions such as the target's thermal radiation intensity and temperature distribution differences.

[0117] When the image feature samples corresponding to at least one image modality include image feature samples of visible light image samples and image feature samples of thermal infrared image samples, the text feature samples may include text feature samples corresponding to the visible light image modality, text feature samples corresponding to the thermal infrared image modality, or text feature samples corresponding to both the visible light image modality and the thermal infrared image modality.

[0118] The enhanced template feature sample is obtained by fusing the text feature sample and the template image feature sample corresponding to at least one image modality. The template image feature sample is a feature of the template image sample that assists in target tracking in at least one image modality.

[0119] Template image samples are typically reference images containing the target to be tracked. They reflect the target's appearance, structural information, or texture information in that modality, and are highly representative and recognizable. By extracting features from template image samples, template image feature samples can be obtained.

[0120] The network architecture of the target tracking model to be trained can be found in [reference needed]. Figure 5 As shown, by inputting training text and video image samples corresponding to at least one image modality into the target tracking model to be trained, feature samples and enhanced template feature samples of each modality can be obtained.

[0121] Taking at least one image modality including a visible light modality and a thermal infrared modality, and training text including a first training text corresponding to the visible light modality and a second training text corresponding to the thermal infrared modality, as an example, the first text feature extraction module 201 can extract a first text feature sample from the input first training text. The first training text is text data describing the target to be tracked in the visible light image sample using natural language.

[0122] The visible light image feature extraction module 202 can extract visible light image feature samples from the input visible light image samples and visible light template image feature samples from the input visible light template image samples. The visible light template image samples can be obtained by target detection and localization based on the previous frame of the visible light video image sample corresponding to the input visible light video image sample.

[0123] The second text feature extraction module 203 can extract second text feature samples from the input second training text. The second training text is text data describing the target to be tracked in the thermal infrared image samples using natural language.

[0124] The thermal infrared image feature extraction module 204 can extract thermal infrared image feature samples from input thermal infrared image samples and thermal infrared template image feature samples from input thermal infrared template image samples. The thermal infrared template image samples can be obtained by target detection and localization based on the previous frame of thermal infrared image samples corresponding to the input thermal infrared image samples.

[0125] The first feature fusion module 205 can perform feature fusion on the first text feature sample, the visible light template image feature sample, the second text feature sample, and the thermal infrared template image feature sample to obtain an enhanced template feature sample.

[0126] Continue reading Figure 4 After step S401, step S402 may also be included:

[0127] S402. The target prediction result is obtained by using the target tracking model to be trained to predict the target based on the feature samples of each modality and the enhanced template feature samples.

[0128] Step S402 includes the following steps c1-c2:

[0129] Step c1: Use the target tracking model to be trained to perform feature association between the feature samples of each modality and the enhanced template feature samples to obtain fused feature samples.

[0130] Step c2: Perform target tracking based on fused feature samples to obtain target tracking results.

[0131] In some embodiments, step c1 includes the following steps c11-c12:

[0132] Step c11: For each image modality in at least one image modality, establish the association between the image feature samples and the corresponding text feature samples under that image modality, and obtain the associated feature samples corresponding to that image modality.

[0133] Step c12: Based on the association of the associated feature samples and the enhanced template feature samples corresponding to each of at least one image modality, a fused feature sample is obtained.

[0134] Step c11 includes: determining the similarity between the text feature samples and the image feature samples of each image region; using each similarity as the weight coefficient corresponding to each image region feature sample, and performing a weighted summation of the image region feature samples in the image feature samples to obtain the associated feature samples of the image modality.

[0135] In some embodiments, step c12 includes the following steps c121-c122:

[0136] Step c121: Establish the association relationship between the associated feature samples and the enhanced template feature samples corresponding to at least one image modality, and obtain the enhanced associated feature samples corresponding to at least one image modality.

[0137] Step c122: Perform feature fusion on the enhanced correlation feature samples corresponding to at least one image modality to obtain fused feature samples.

[0138] Step c121 includes: for each image modality in at least one image modality, determining the similarity between the enhanced template feature sample and each image region feature sample in the associated feature sample corresponding to the image modality; using each similarity as the weight coefficient corresponding to each image region feature sample, and performing a weighted summation on each image region feature sample to obtain the enhanced associated feature sample corresponding to the image modality.

[0139] The specific implementation process of step S402 will still be described using image mode and thermal infrared mode as examples. Through the first feature association module 206, the image feature samples of the visible light mode and the first text feature samples can be associated to obtain the first associated feature sample, which is the associated feature sample corresponding to the visible light mode; through the second feature association module 207, the image feature samples of the thermal infrared mode and the second text feature samples can be associated to obtain the second associated feature sample, which is the associated feature sample corresponding to the thermal infrared mode.

[0140] After obtaining the first associated feature sample and the second associated feature sample, the first associated feature sample can be associated with the enhanced template feature sample through the third feature association module 208 to obtain the first enhanced associated feature sample.

[0141] And the second associated feature sample is obtained by associating the second associated feature sample with the enhanced template feature sample through the fourth feature association module 209.

[0142] Then, the second feature fusion module 210 fuses the first enhanced correlation feature sample and the second enhanced correlation feature sample to obtain the fused feature sample.

[0143] S403. Based on the difference between the target prediction result and the real target annotation result, the target tracking model to be trained is iteratively trained to obtain the target tracking model.

[0144] In this step, by comparing the difference between the target prediction results output by the model and the corresponding real target annotation results, backpropagation and parameter optimization are performed on the target tracking model to be trained, thereby achieving iterative training and performance improvement of the model.

[0145] Specifically, during training, two loss functions can be used to jointly supervise the network's learning process:

[0146] GIoU Loss (Generalized Intersection over Union Loss): This measure measures the degree of overlap between the predicted bounding box and the ground truth bounding box. Compared to the traditional IoU Loss, GIoU Loss is more robust when dealing with non-intersecting bounding boxes and can more effectively guide the accurate localization of the bounding boxes.

[0147] L1 Loss (Mean Absolute Error): Used to constrain the regression error of the bounding box coordinates, ensuring that the model has higher accuracy in predicting the target location.

[0148] By jointly optimizing these two loss terms, the model can achieve better performance in classification and localization tasks, ultimately obtaining a target tracking model with high-precision target prediction capabilities.

[0149] It should be noted that the first text feature extraction module 201, the visible light image feature extraction module 202, the second text feature extraction module 203, and the thermal infrared image feature extraction module 204 are all set to a frozen state during training. That is, after loading the pre-trained parameters of these modules throughout the entire model training process, their parameters remain unchanged and do not participate in gradient updates. This accelerates convergence and thus achieves better performance gains.

[0150] In addition, since the feature data of each modality is obtained by feature extraction through a frozen encoder, there may be a problem of feature space misalignment. Therefore, before inputting the feature data of each modality into the first feature association module and the second feature association module, it is necessary to project the features extracted by the frozen encoder into a latent space with the same dimension through a linear projection layer.

[0151] Furthermore, it should be noted that the visible light mode and thermal infrared mode used in this embodiment are merely illustrative examples and do not limit the application to at least one image mode. This embodiment can be applied to any two aligned multimodal images, such as RGB images and depth maps, RGB images and near-infrared images, or other combinations of image modes with complementary characteristics. As long as there is a corresponding spatial structural relationship between the images, they can be applied to the technical framework described in this embodiment.

[0152] In some embodiments, the model also needs to be tested to evaluate the training effect. During model testing, various initialization methods can be used for target localization and tracking to verify the model's robustness and generalization ability under different modal input conditions. These initialization methods include:

[0153] (1) The first training text corresponding to the visible light image sample, the thermal infrared image sample and the visible light mode.

[0154] (2) The second training text corresponding to the visible light image samples, thermal infrared image samples and thermal infrared modes.

[0155] (3) Visible light image samples, thermal infrared image samples, the first training text corresponding to the visible light mode and the second training text corresponding to the thermal infrared mode.

[0156] (4) Visible light image samples, thermal infrared image samples, the first training text corresponding to the visible light mode, the second training text corresponding to the thermal infrared mode, and the ground truth box.

[0157] If text information for a certain modality is missing (usually indicating that the modality is of poor quality or unreliable), an empty string is used to replace the text input for that modality to ensure consistency in input format.

[0158] In the testing process, the model first performs initial localization based on the information in the first frame, and then crops the target region from that frame as a template feature. In subsequent video frames, the model uses this template to match the multimodal features of the current frame, thereby predicting the target tracking box in each frame and achieving continuous tracking of the target.

[0159] By testing the model's performance using multiple initialization methods, we can ensure that the model can adapt to various input modes in subsequent target tracking applications. This means that even when text information in a certain modality is missing, the model can still effectively perform target tracking tasks. This flexibility and robustness guarantee that the model provides stable and reliable tracking performance regardless of the completeness of the input data. Therefore, through diverse testing schemes, we not only enhance the model's applicability in different scenarios but also improve its ability to handle incomplete or partially missing information, making it more reliable and efficient in practical applications.

[0160] In summary, this embodiment uses multimodal natural language description pairs and image pairs as inputs to a four-branch structure to achieve precise target localization and robust tracking. This method effectively establishes semantic reference relationships between natural language and multimodal images regarding target information, enhancing the model's understanding and reasoning capabilities in complex visual environments. Simultaneously, by combining four different initialization methods, the system possesses richer interaction modes, enhancing user freedom in complex application scenarios and enabling flexible and diverse initialization settings. Furthermore, this embodiment introduces a joint enhanced template mechanism. This mechanism uses visual templates from two modalities and their corresponding natural language description classification labels as input. Through deep interaction with the visual search region, it mines rich semantic information within the target's context, thereby further improving the accuracy and robustness of target tracking.

[0161] Exemplary device

[0162] Corresponding to the target tracking method described above, this application also provides a target tracking device. Figure 6 This is a schematic diagram of the structure of a target tracking device provided in an embodiment of this application. Figure 6 As shown, the target tracking device provided in this application embodiment includes: an acquisition unit 601 and a target tracking unit 602; wherein, the acquisition unit 601 is used to acquire feature data and enhanced template features of each modality, wherein the feature data of each modality includes text features corresponding to text data and image features corresponding to at least one image modality, wherein the text data is used to describe the target to be tracked in the at least one image modality; the target tracking unit 602 is used to perform target tracking based on the feature data and enhanced template features of each modality to obtain a target tracking result; wherein, the enhanced template features are obtained by feature fusion of the text features and the template image features corresponding to each of the at least one image modality, wherein the template image features are features of a template image that assist in target tracking in the at least one image modality.

[0163] In some embodiments, the target tracking unit 602 performs target tracking based on the features of each modality and the enhanced template features to obtain a target tracking result, including: performing feature association on the feature data of each modality and the enhanced template features to obtain fused features; and performing target tracking based on the fused features to obtain the target tracking result.

[0164] In some embodiments, the at least one image modality corresponds to text features; wherein, the target tracking unit 602 performs feature association on the feature data of each modality and the enhanced template features to obtain fused features, including: for each image modality in the at least one image modality, establishing the association relationship between the image features and the corresponding text features under that image modality to obtain the associated features corresponding to that image modality; and obtaining the fused features based on the association between the associated features and the enhanced template features corresponding to each of the at least one image modality.

[0165] In some embodiments, the target tracking unit 602 establishes the association between image features and corresponding text features under the image modality to obtain the associated features corresponding to the image modality, including: determining the similarity between the text features and each image region feature in the image features; using each similarity as the weight coefficient corresponding to each image region feature, and performing a weighted summation on each image region feature in the image features to obtain the associated features of the image modality.

[0166] In some embodiments, the target tracking unit 602 obtains the fused features based on the association between the associated features and the enhanced template features corresponding to each of the at least one image modality, including: establishing the association relationship between the associated features and the enhanced template features corresponding to each of the at least one image modality to obtain the enhanced associated features corresponding to each of the at least one image modality; and performing feature fusion on the enhanced associated features corresponding to each of the at least one image modality to obtain the fused features.

[0167] In some embodiments, the target tracking unit 602 establishes the association relationship between the associated features and the enhanced template features corresponding to each of the at least one image modalities, and obtains the enhanced associated features corresponding to each of the at least one image modalities, including: for each image modal in the at least one image modalities, determining the similarity between the enhanced template features and each image region feature in the associated features corresponding to that image modalities; using each similarity as a weight coefficient corresponding to each image region feature, and performing a weighted summation of the image region features to obtain the enhanced associated features corresponding to that image modalities.

[0168] In some embodiments, the at least one image modality includes: a visible light modality and / or a thermal infrared modality; when the at least one image modality includes the visible light modality and the thermal infrared modality, the text features include text features corresponding to at least one of the visible light modality and the thermal infrared modality, and the enhanced template features are obtained by feature fusion of the text features corresponding to at least one of the visible light modality and the thermal infrared modality with the template image features corresponding to each of the at least one image modality.

[0169] In some embodiments, the method is executed by a target tracking model, which is trained using the following steps: acquiring feature samples and enhanced template feature samples for each modality, wherein the feature samples for each modality include text feature samples corresponding to training text and image feature samples corresponding to at least one image modality, wherein the training text is used to describe the target to be tracked in at least one image modality; performing target prediction based on the feature samples and enhanced template feature samples of each modality using the target tracking model to be trained, thereby obtaining a target prediction result; and iteratively training the target tracking model to be trained based on the difference between the target prediction result and the real target annotation result, thereby obtaining the target tracking model; wherein the enhanced template feature samples are obtained by feature fusion of the text feature samples and the template image feature samples corresponding to each of the at least one image modality, and the template image feature samples are features of template image samples that assist in target tracking in the at least one image modality.

[0170] The target tracking device provided in this embodiment belongs to the same concept as the target tracking method provided in the above embodiments of this application. It can execute the target tracking method provided in any of the above embodiments of this application and has the corresponding functional modules and beneficial effects for executing the target tracking method. Technical details not described in detail in this embodiment can be found in the specific processing content of the target tracking method provided in the above embodiments of this application, and will not be repeated here.

[0171] The functions implemented by the acquisition unit 601 and the target tracking unit 602 described above can be implemented by the same or different processors, and this application embodiment does not limit them.

[0172] It should be understood that the units in the above device can be implemented by a processor calling software. For example, the device includes a processor connected to a memory containing instructions. The processor calls the instructions stored in the memory to implement any of the above methods or to implement the functions of each unit in the device. The processor can be a general-purpose processor, such as a CPU or microprocessor, and the memory can be internal or external to the device. Alternatively, the units in the device can be implemented as hardware circuits. By designing the hardware circuits, some or all of the unit functions can be implemented. The hardware circuits can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are implemented by designing the logical relationships between the components within the circuit. In another implementation, the hardware circuit can be implemented using a PLD, such as an FPGA, which can include a large number of logic gates. The connection relationships between the logic gates are configured through configuration files to implement the functions of some or all of the above units. All units in the above device can be implemented entirely by a processor calling software, entirely by hardware circuits, or partially by a processor calling software with the remaining parts implemented by hardware circuits.

[0173] In this application embodiment, a processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction reading and execution capabilities, such as a CPU, microprocessor, GPU, or DSP. In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented as an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the processor loading instructions to implement the functions of some or all of the above units. Furthermore, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, or DPU.

[0174] As can be seen, each unit in the above device can be one or more processors (or processing circuits) configured to implement the above methods, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0175] Furthermore, the units in the above devices can be integrated in whole or in part, or they can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a System-on-Chip (SoC). The SoC may include at least one processor for implementing any of the above methods or implementing the functions of the units in the device. The at least one processor may be of different types, such as CPU and FPGA, CPU and artificial intelligence processor, CPU and GPU, etc.

[0176] Exemplary electronic devices

[0177] This application provides an electronic device, see [link to relevant documentation] Figure 7 As shown, the electronic device includes:

[0178] Memory 200 and processor 210;

[0179] The memory 200 is connected to the processor 210 and is used to store programs;

[0180] The processor 210 is configured to implement the target tracking method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0181] Specifically, the aforementioned electronic device may also include: a bus, a communication interface 220, an input device 230, and an output device 240.

[0182] The processor 210, memory 200, communication interface 220, input device 230, and output device 240 are interconnected via a bus. Among them:

[0183] A bus can include a pathway for transmitting information between various components of a computer system.

[0184] The processor 210 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present invention. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0185] Processor 210 may include a main processor, as well as a baseband chip, modem, etc.

[0186] The memory 200 stores a program that executes the technical solution of this invention, and may also store an operating system and other key business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 200 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0187] Input device 230 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0188] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0189] The communication interface 220 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0190] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement the various steps of any of the target tracking methods provided in the above embodiments of this application.

[0191] This application also proposes a chip including a processor and a data interface. The processor reads and runs a program stored in a memory through the data interface to execute the target tracking method described in any of the above embodiments. For details of the processing and its beneficial effects, please refer to the above embodiments of the target tracking method.

[0192] Exemplary computer program products and storage media

[0193] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the target tracking methods according to various embodiments of this application as described in any of the above embodiments of this specification.

[0194] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0195] Furthermore, embodiments of this application may also be storage media storing a computer program, which is executed by a processor to perform the steps of the target tracking method according to various embodiments of this application described in any of the above embodiments of this specification, specifically implementing the steps of the above target tracking method.

[0196] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0197] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0198] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0199] The modules and sub-modules in the various embodiments of the present application's devices and terminals can be merged, divided, and deleted according to actual needs.

[0200] It should be understood that the disclosed terminals, devices, and methods can be implemented in other ways, given the several embodiments provided in this application. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0201] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0202] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or in the form of software functional modules or sub-modules.

[0203] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0204] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0205] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0206] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A target tracking method, characterized in that, include: The feature data and enhanced template features of each modality are obtained. The feature data of each modality includes text features corresponding to text data and image features corresponding to at least one image modality. The text data is used to describe the target to be tracked in the at least one image modality. Target tracking is performed based on the feature data of each modality and the enhanced template features to obtain the target tracking result; The enhanced template features are obtained by feature fusion of the text features and the template image features corresponding to each of the at least one image modality. The template image features are features of the template image that assist in target tracking in the at least one image modality.

2. The method according to claim 1, characterized in that, The target tracking based on the features of each modality and the enhanced template features, to obtain the target tracking result, includes: Feature association is performed on the feature data of each modality and the enhanced template features to obtain fused features; Target tracking is performed based on the fused features to obtain the target tracking result.

3. The method according to claim 2, characterized in that, The at least one image modality corresponds to text features; The step of associating the feature data of each modality with the enhanced template features to obtain fused features includes: For each image modality in the at least one image modality, establish the association relationship between the image features and the corresponding text features under that image modality to obtain the associated features corresponding to that image modality; The fused features are obtained based on the association between the associated features and the enhanced template features corresponding to each of the at least one image modality.

4. The method according to claim 3, characterized in that, The process of establishing the association between image features and corresponding text features under this image modality to obtain the associated features corresponding to this image modality includes: Determine the similarity between the text features and the image features of each image region; Each similarity is used as a weight coefficient corresponding to each image region feature. The image region features are then weighted and summed to obtain the associated features of the image modality.

5. The method according to claim 3, characterized in that, The fusion feature is obtained based on the association of the associated features and enhanced template features corresponding to each of the at least one image modality, including: Establish the association relationship between the association features corresponding to each of the at least one image modality and the enhanced template features to obtain the enhanced association features corresponding to each of the at least one image modality; The enhanced correlation features corresponding to each of the at least one image modality are fused to obtain the fused features.

6. The method according to claim 5, characterized in that, The step of establishing the association relationship between the association features corresponding to each of the at least one image modality and the enhanced template features, to obtain the enhanced association features corresponding to each of the at least one image modality, includes: For each image modality in the at least one image modality, determine the similarity between the enhanced template features and the associated features of each image region in the image modality corresponding to that image modality; Each similarity is used as a weight coefficient corresponding to each image region feature, and the image region features are weighted and summed to obtain the enhanced association features corresponding to the image modality.

7. The method according to any one of claims 1-6, characterized in that, The at least one image mode includes: visible light mode and / or thermal infrared mode; When the at least one image modality includes the visible light modality and the thermal infrared modality, the text features include text features corresponding to at least one of the visible light modality and the thermal infrared modality. The enhanced template features are obtained by feature fusion of the text features corresponding to at least one of the visible light modality and the thermal infrared modality with the template image features corresponding to each of the at least one image modality.

8. The method according to any one of claims 1-6, characterized in that, The method is executed through a target tracking model, which is trained using the following steps: Obtain feature samples and enhanced template feature samples for each modality. The feature samples for each modality include text feature samples corresponding to the training text and image feature samples corresponding to at least one image modality. The training text is used to describe the target to be tracked in at least one image modality. The target prediction result is obtained by using the target tracking model to be trained to predict the target based on the feature samples of each modality and the enhanced template feature samples. Based on the difference between the target prediction result and the actual target annotation result, the target tracking model to be trained is iteratively trained to obtain the target tracking model; The enhanced template feature sample is obtained by feature fusion of the text feature sample and the template image feature sample corresponding to each of the at least one image modality. The template image feature sample is a feature of the template image sample that assists in target tracking in the at least one image modality.

9. A target tracking device, characterized in that, include: The acquisition unit is used to acquire feature data and enhanced template features for each modality. The feature data for each modality includes text features corresponding to text data and image features corresponding to at least one image modality. The text data is used to describe the target to be tracked in the at least one image modality. The target tracking unit is used to perform target tracking based on the feature data of each modality and the enhanced template features, and obtain the target tracking result; The enhanced template features are obtained by feature fusion of the text features and the template image features corresponding to each of the at least one image modality. The template image features are features of the template image that assist in target tracking in the at least one image modality.

10. An electronic device, characterized in that, Including memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the method as described in any one of claims 1 to 8 by running a program in the memory.

11. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, It includes computer program instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1 to 8.

Images