Target tracking method, product and device based on multi-modal matching, and storage medium

Abstract

The application provides a target tracking method, product and equipment based on multi-modal matching and a storage medium. The method comprises: acquiring multi-modal template information and a search image containing a target object; the multi-modal template information comprises at least one image template and at least one text template; performing feature extraction on the multi-modal template information to obtain template embedding features; the template embedding features are feature representations fused with the multi-modal template information; performing feature extraction on the search image to obtain search image features; based on the template embedding features and the search image features, a similarity heat map is generated through contrast learning calculation; the similarity heat map represents the matching confidence of each position in the search image and the template; according to the similarity heat map, a bounding box of the target object in the search image is determined, and target tracking is performed according to the bounding box. In the identification process, the visual appearance and semantic description of the target can be understood at the same time, and the tracking adaptability to the target of an un-predefined category is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a target tracking method, product, device, and storage medium based on multimodal matching. Background Technology

[0002] The current approach for drones in target tracking tasks typically involves first locating the target in an image and outputting an initial bounding box using a pre-trained target detection model (such as YOLO), and then continuously tracking it using a tracking algorithm (such as SiameseRPN). However, this approach has significant limitations: the system can only recognize predefined categories and cannot initialize based on user-provided images of unknown categories or specific targets, resulting in insufficient tracking accuracy and poor adaptability in complex scenarios. Summary of the Invention

[0003] The purpose of this application is to provide a target tracking method, program product, electronic device and storage medium based on multimodal matching to improve the above-mentioned problems.

[0004] In a first aspect, embodiments of this application provide a target tracking method based on multimodal matching, comprising: acquiring multimodal template information and a search image containing a target object; the multimodal template information includes at least one image template and at least one text template, the image template containing the bounding box of the target object; performing feature extraction on the multimodal template information to obtain template embedding features; the template embedding features are feature representations that fuse the multimodal template information; performing feature extraction on the search image to obtain search image features; generating a similarity heatmap by comparative learning based on the template embedding features and the search image features, the similarity heatmap representing the matching confidence of each position in the search image with the template; determining the bounding box of the target object in the search image according to the similarity heatmap, and performing target tracking according to the bounding box.

[0005] In the aforementioned implementation process, multimodal template information containing images and text is uniformly extracted and fused, enabling the simultaneous understanding of the target's visual appearance and semantic description during recognition, thus improving the tracking adaptability for targets without predefined categories. By generating similarity heatmaps through contrastive learning, the region that best matches the template can be efficiently located directly in complex search images. Using similarity heatmaps to locate bounding boxes ensures the accuracy of initialization, providing a reliable foundation for subsequent stable tracking. This approach overcomes the shortcomings of traditional methods, such as reliance on fixed categories, inability to utilize textual information, and inaccurate initialization, enabling the tracking process to handle diverse targets, complex instructions, and scenarios, improving flexibility, accuracy, and robustness.

[0006] Optionally, in this embodiment of the application, feature extraction of multimodal template information to obtain template embedding features includes: encoding text templates into text features using a pre-trained cross-modal model; extracting visual feature maps of image templates using a convolutional neural network, generating visual cue masks based on the bounding boxes contained in the image templates, extracting target visual features from the visual feature maps based on the visual cue masks, and generating template embedding features based on text features and target visual features.

[0007] In the above implementation process, using a pre-trained cross-modal model to process text enables accurate understanding of the user's linguistic intent and its transformation into computable semantic features. By combining convolutional networks with visual cue masks to process image templates, background interference can be effectively eliminated, extracting the core visual features of the target object. Finally, by fusing the two types of features to generate a unified template embedding feature, the system can comprehensively utilize the semantic information of the text description and the appearance information of the image example to form a more comprehensive and stable feature representation of the target. This process improves the system's adaptability to diverse target descriptions and enhances its ability to extract key features from complex template information, laying the foundation for subsequent matching in search images.

[0008] Optionally, in this embodiment of the application, generating template embedding features based on text features and target visual features includes: dynamically fusing text features corresponding to multiple text templates and target visual features corresponding to multiple image templates through an attention mechanism to generate template embedding features.

[0009] In the aforementioned implementation process, by intelligently and comprehensively utilizing template information from multiple possible modalities, the attention mechanism prevents the system from treating all templates equally or simply overlaying them. Instead, it determines which template features are more relevant or reliable based on the specific search scenario and assigns them higher weights. This dynamic, context-aware fusion approach improves the quality and relevance of the generated template embedding features. This makes the final target prototype features used for matching more adaptable to complex real-world environments and enhances robustness against multi-source information that may contain redundancy or contradictions.

[0010] Optionally, in this embodiment of the application, after encoding the text template into text features using a pre-trained cross-modal model, the method further includes: inputting the text features into a feature alignment network module for nonlinear transformation, and outputting fine-tuned text features; wherein, the feature alignment network module is used to perform nonlinear transformation on the text features to enhance the semantic alignment between the text features and visual features in the task feature space; generating template embedding features based on the text features and target visual features, including: generating template embedding features based on the fine-tuned text features and target visual features.

[0011] In the above implementation process, to ensure that the text features not only contain rich semantics but also better align with the visual feature distribution in the UAV tracking scenario, this application uses a feature alignment network module to perform nonlinear transformations on the text features, making targeted fine-tuning of the original text features to better adapt them to the needs of the tracking task. This process enhances the semantic alignment between text features and visual features in the task feature space, bringing text features and image features with the same meaning closer together in the feature space. This improves the ability to understand and respond to text commands, thereby increasing the accuracy of subsequent cross-modal matching.

[0012] Optionally, in this embodiment of the application, a template embedding feature is generated by dynamically fusing text features corresponding to multiple text templates and target visual features corresponding to multiple image templates through an attention mechanism. This includes: dynamically weighting and fusing text features corresponding to multiple text templates and target visual features corresponding to multiple image templates according to search image features through an attention mechanism to generate template embedding features; wherein, the attention mechanism is a cross-attention mechanism, which uses search image features as the query and uses the combined features corresponding to multiple text templates and image templates as the key and value for calculation.

[0013] In the above implementation process, by using the search image features as queries to inquire about various template features, the system can automatically determine which template information is more critical and reliable based on the current specific environment, and allocate fusion weights accordingly. This dynamic weighting method, compared to simple averaging or concatenation, can more effectively integrate multi-source information, reduce interference from irrelevant or contradictory templates, and thus generate a higher-quality, more targeted template embedding feature. This improves robustness and adaptability when facing multiple potentially ambiguous, redundant, or cross-modal template inputs, provides a more reliable basis for the search image, and enhances the success rate and accuracy of target tracking.

[0014] Optionally, in this embodiment of the application, a similarity heatmap is generated by contrastive learning based on template embedding features and search image features, including: converting the search image features into a set of spatial location feature vectors, wherein each spatial location feature vector corresponds to a candidate region in the search image; calculating the cosine similarity between the template embedding features and each spatial location feature vector in the set; and mapping all the calculated cosine similarity values ​​back to a two-dimensional grid that matches the spatial dimension of the search image features to form a similarity heatmap.

[0015] In the above implementation process, the search image features are converted into a vector set, enabling similarity calculation for all candidate regions in the image. This effectively measures the similarity of semantic features, and the calculation process is efficient and stable. The resulting similarity heatmap more intuitively identifies the regions in the search image most likely to contain the target and their confidence distribution.

[0016] Optionally, in this embodiment of the application, determining the bounding box of the target object in the search image based on the similarity heatmap includes: using the similarity heatmap as a confidence guide and inputting it together with the search image features into the target detection head; the target detection head focuses on the potential target region based on the search image features and using the confidence guide, and regresses and outputs the bounding box coordinates and category confidence of the target object; wherein, the target detection head uses distributed focus loss to optimize the regression prediction of the bounding box coordinates.

[0017] In the above implementation, by using a similarity heatmap as a confidence guide and integrating it into the object detection process, a clear spatial prior is provided. This allows computational resources to be focused on the regions most likely to contain the target, reducing computational waste on irrelevant backgrounds and improving localization efficiency. The use of distributed focus loss to optimize bounding box coordinate regression enhances the ability to model subtle changes in target position, resulting in more accurate and stable final bounding box coordinates. By fully utilizing information from the preceding matching stage, object detection is no longer a blind search across the entire image, but rather precise localization guided by strong confidence. This improves the accuracy and reliability of the final output bounding box, providing high-quality initial input for subsequent tracking tasks.

[0018] Secondly, embodiments of this application also provide a target tracking device based on multimodal matching, comprising: an acquisition module for acquiring multimodal template information and a search image containing a target object; the multimodal template information includes at least one image template and at least one text template, the image template containing the bounding box of the target object; a template extraction module for extracting features from the multimodal template information to obtain template embedding features; the template embedding features are feature representations that fuse the multimodal template information; an image extraction module for extracting features from the search image to obtain search image features; a matching module for generating a similarity heatmap by comparative learning based on the template embedding features and the search image features, the similarity heatmap representing the matching confidence of each position in the search image with the template; and a boundary determination module for determining the bounding box of the target object in the search image according to the similarity heatmap, and performing target tracking based on the bounding box.

[0019] Thirdly, embodiments of this application also provide a computer program product, including computer program instructions, which are executed by a processor to perform the method provided in the first aspect or any implementation thereof.

[0020] Fourthly, embodiments of this application also provide an electronic device, including: a processor and a memory, the memory storing computer program instructions, which are executed by the processor to perform the method provided in the first aspect or any implementation thereof.

[0021] Fifthly, embodiments of this application also provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, perform the method provided in the first aspect or any implementation thereof.

[0022] This application presents a multimodal matching-based target tracking method, program product, electronic device, and storage medium. It performs unified feature extraction and fusion of multimodal template information containing images and text, enabling simultaneous understanding of the target's visual appearance and semantic description during recognition, thus improving the tracking adaptability for targets with undefined categories. By generating similarity heatmaps through contrastive learning, the most matching region can be efficiently located directly in complex search images. Using similarity heatmaps to locate bounding boxes ensures the accuracy of initialization, providing a reliable foundation for subsequent stable tracking. This approach overcomes the shortcomings of traditional methods, such as reliance on fixed categories, inability to utilize textual information, and inaccurate initialization, enabling the tracking process to handle diverse targets, complex instructions, and scenarios, improving flexibility, accuracy, and robustness. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating a target tracking method based on multimodal matching provided in this application embodiment; Figure 2 This is a schematic diagram of the structure of a target tracking device based on multimodal matching provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this application.

[0027] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0028] Currently, drones generally employ a "detect first, track later" approach for target tracking. Specifically, the system first relies on a pre-trained general target detection model (such as the YOLO series) to analyze real-time footage captured by the drone, identifying targets belonging to predefined categories (such as vehicles and pedestrians) and generating an initial bounding box. Subsequently, the system passes this initial bounding box to a tracking algorithm (such as SiameseRPN), which continuously follows the target in subsequent video frames.

[0029] However, this mainstream approach reveals several significant shortcomings in practical applications: First, the system can only identify and initialize targets belonging to "known categories" defined during the training of the detection model. If the operator wants to track a specific object not in the list, such as a vehicle with a special paint job, the system cannot provide the correct initial bounding box. Second, the entire process only accepts image input. Without an image example, the system cannot initiate the tracking task if only text instructions are given, such as "track the person in red ahead." Third, during the initialization phase, the system only analyzes the currently searched image itself, without fully utilizing the various reference information that the operator may provide simultaneously, such as an example image plus a descriptive text. This results in potentially inaccurate initial localization, making it prone to failure when the background is cluttered or the target is partially occluded.

[0030] This application provides a target tracking method based on multimodal matching. It can receive image templates and text descriptions as input in real time. Through feature extraction and fusion, it unifies different forms of template information into a single feature representation. By comparative learning, it finds the region in the search image that best matches the template features and outputs an accurate bounding box. Matching can be performed based on both text and image templates, significantly improving the system's responsiveness to unknown targets and cross-modal commands, as well as its robustness in initializing using prior information.

[0031] Please see Figure 1The illustrated diagram shows a flowchart of a target tracking method based on multimodal matching provided in this application embodiment. The target tracking method based on multimodal matching provided in this application embodiment can be applied to electronic devices, which may include physical devices such as servers, PCs, tablets, or smartphones, or virtual devices such as virtual machines or containers. The electronic device can be a single device, a combination of multiple devices, or a cluster of a large number of devices. The target tracking method based on multimodal matching may include: Step S110: Obtain multimodal template information and a search image containing the target object; the multimodal template information includes at least one image template and at least one text template, and the image template contains the bounding box of the target object.

[0032] Step S120: Extract features from the multimodal template information to obtain template embedding features; the template embedding features are feature representations that integrate multimodal template information.

[0033] Step S130: Extract features from the search image to obtain the search image features.

[0034] Step S140: Based on template embedding features and search image features, a similarity heatmap is generated through contrastive learning. The similarity heatmap represents the matching confidence of each position in the search image with the template.

[0035] Step S150: Determine the bounding box of the target object in the search image based on the similarity heatmap, and perform target tracking based on the bounding box.

[0036] In step S110, multimodal template information refers to information describing the same target in different forms (modals), mainly including two categories: text templates and image templates. An image template is a static image containing the target object, with a precise bounding box marked on it. The bounding box can be represented by rectangular coordinates, used to define the specific location and range of the target in the image, providing the user with a visual example of the target's appearance. A text template is a description of the target in natural language, such as "red car" or "kite in the sky," providing semantic information about the target. The search image can be a single-frame image captured in real-time by an onboard camera of a drone or the current frame in a video stream; the system needs to find the target that matches the template within this image.

[0037] As one implementation method, user input can be received through a human-computer interaction interface. Users can upload an image via ground station software and manually or automatically annotate the target box, using this as an image template. Simultaneously, users can also type a description of the target in a text input box, using this as a text template. Only image templates, only text templates, or both can be acquired. Search images are acquired in real-time via the UAV's image transmission link. After acquiring this data, it can be cached and passed to subsequent processing modules.

[0038] In step S120, the template information of different modalities, which may be multiple, is transformed into a unified, machine-computable digital feature representation, namely, the template embedding feature. This feature should integrate information from all available templates and be able to effectively represent the target in the feature space.

[0039] For text templates, the system utilizes a pre-trained cross-modal model text encoder to transform the natural language description into a high-dimensional text feature or text feature vector. In one implementation, the alignment accuracy between these text features and the visual task can be further improved by aligning the features with network modules, performing non-linear transformations and fine-tuning, and outputting text features with more focused semantics.

[0040] For image templates, convolutional neural networks or other feature extraction models can be used to extract visual features from the template image. Based on the bounding box attached to the image template, a corresponding visual cue mask is generated on the feature map. The visual cue mask has high weights in the target region and low weights in the background region. Then, through pooling or attention operations, the target visual feature vector focused on the target region is extracted from the feature map. Finally, if multiple templates exist, such as multiple template images or multiple text templates, the features corresponding to these templates can be fused to generate a unified template embedding feature. In this way, the final template embedding feature incorporates feature representations that integrate multimodal template information.

[0041] In step S130, the searched image features need to retain rich spatial location information to facilitate subsequent position-by-position matching. The searched image features can be one or more feature maps.

[0042] One implementation method is to use a convolutional neural network (CNN) or other feature extraction model, which is used to extract visual features from template images, to extract features from the search image. This CNN shares a backbone with the network that processes the image template and performs forward propagation on the input search image. Taking a CNN as an example, the CNN performs convolution, downsampling, and other operations on the search image layer by layer, ultimately outputting deep feature maps at one or more scales. Each spatial location on each feature map corresponds to a feature vector; the spatial location on the feature map can be understood as an abstract representation of a corresponding region in the original image. These feature vectors collectively constitute the features of the search image.

[0043] For example, a feature map might have the shape [C, H, W], where C is the dimension (number of channels) of the feature vector, and H and W are the height and width of the feature map, respectively, corresponding to the original image being divided into H x W candidate regions. This step transforms the original pixel image into a structured feature representation rich in high-level semantics, enabling subsequent refined matching.

[0044] In step S140, a contrastive learning mechanism is used to measure the similarity between the template and various regions of the search image. The similarity heatmap is a two-dimensional image in which the brightness or value of each pixel represents the degree of matching between the corresponding position in the search image and the embedded features of the template. The higher the value, the greater the probability that the region is the target.

[0045] For example, the cosine similarity between the search image features and the template embedding feature vector can be calculated. This similarity can be either cosine similarity or negative Euclidean distance. After calculation, a two-dimensional numerical matrix with the same dimensions (H, W) as the search feature map space is obtained. Post-processing this matrix, such as normalization, yields a similarity heatmap. The brightest region in the heatmap indicates the location of the candidate target that best matches the template. Essentially, this transforms the target localization problem into finding the nearest neighboring search region to the template point within the feature space.

[0046] In step S150: using the matching confidence information in the similarity heatmap, the bounding box of the target object in the search image is determined, and target tracking is performed based on the bounding box. The bounding box is a rectangular box used to accurately define the position and size of the target, usually represented by the coordinates of the center point and the width and height.

[0047] A similarity heatmap, serving as a strong confidence guide for the probability of target presence, is input along with the original search image features into a target detection head. This head can employ a distributed focal loss method. Guided by the high-confidence regions of the similarity heatmap, the target detection head focuses on and performs refined regression on these potential regions, ultimately outputting one or more bounding boxes with precise coordinates and confidence scores. The bounding box with the highest confidence score is selected as the final localization result.

[0048] This bounding box is then used as the initial tracking box and passed to a tracker module. The tracker module uses this box as a starting point to continuously predict the target's position in subsequent image frames, thus achieving stable and long-term tracking of the target. This step effectively connects the one-time precise matching initialization with the continuous tracking process.

[0049] In the implementation of the above embodiments: multimodal template information containing images and text is uniformly extracted and fused, enabling the simultaneous understanding of the target's visual appearance and semantic description during recognition, thus improving the tracking adaptability for targets without predefined categories. By generating similarity heatmaps through contrastive learning, the region that best matches the template can be efficiently located directly in complex search images. Using similarity heatmaps to locate bounding boxes ensures the accuracy of initialization, providing a reliable foundation for subsequent stable tracking. This approach overcomes the shortcomings of traditional methods, such as reliance on fixed categories, inability to utilize textual information, and inaccurate initialization, enabling the tracking process to handle diverse targets, complex instructions, and scenarios, improving flexibility, accuracy, and robustness.

[0050] Optionally, in this embodiment of the application, feature extraction is performed on the multimodal template information to obtain template embedding features, including: A pre-trained cross-modal model is used to encode text templates into text features. The pre-trained cross-modal model is a pre-trained deep learning model (such as the CLIP model) that maps images and text to the same feature space, ensuring that the feature vectors of semantically similar images and texts are also close to each other in space. Text features are fixed-dimensional numerical vectors obtained after encoding the text template using this model; they encapsulate the semantic information of the text.

[0051] The system receives a text template input by the user. The text template can be a text string. First, the sentence is decomposed into lexical tokens that the model can recognize using the built-in tokenizer of a pre-trained cross-modal model. Then, these lexical tokens are input into the model's text encoder. The text encoder processes these tokens through self-attention layers and feedforward network layers, ultimately outputting one or more text feature vectors. As one implementation, the output vector corresponding to the [CLS] token, which represents the semantics of the entire sentence, can be taken, or the output vectors of all tokens can be averaged and pooled as the final text features. Text features capture the global semantics of the text; for example, the feature vector corresponding to "red car" is close to all image features of red cars in the feature space. This step achieves the structuring and digitization of text information, laying the foundation for subsequent alignment and matching with visual information.

[0052] A convolutional neural network is used to extract the visual feature map of an image template, and a visual cue mask is generated based on the bounding box contained in the image template. The target visual features are then extracted from the visual feature map based on the visual cue mask.

[0053] Convolutional Neural Networks (CNNs) are deep learning models specifically designed for processing data such as images. They extract high-level features, from edges and textures to object parts, layer by layer through operations such as convolutional layers and pooling layers. The visual feature map is the output of the intermediate layers of the CNN. It can be a three-dimensional tensor whose height and width correspond to the downsampled spatial grid of the original image. Each grid point (or "position") corresponds to a multi-channel feature vector, collectively representing the visual content of the corresponding region in the original image. The bounding box is a rectangular area marked by the user on the image template to select the target object. The visual cue mask is a two-dimensional matrix with the same spatial dimensions as the visual feature map. Its elements have values ​​between 0 and 1, used to indicate which regions of the feature map the network focuses on; regions within the target bounding box are assigned higher weights (e.g., close to 1), while background regions outside the box are assigned lower weights (e.g., close to 0). The target visual feature refers to the final extracted visual feature vector that primarily represents the target object rather than the background.

[0054] One implementation approach is to input an image template into a convolutional neural network (such as the backbone of ResNet or YOLO), propagate it forward to an intermediate layer, and obtain its visual feature map. Simultaneously, based on the bounding box coordinates attached to the image template, these coordinates are mapped onto the coordinate space of the visual feature map using the same downsampling ratio, generating a binary or soft visual cue mask. The visual feature map is then weighted using this visual cue mask.

[0055] For example, masked average pooling can be used: the feature vectors covered by the mask are summed according to their weights, and then divided by the sum of the weights to obtain a global visual feature vector of the target region.

[0056] Template embedding features are generated based on textual features and target visual features.

[0057] For a single template, if there is only a text template, the text features are directly used as the template embedding features. If there is only an image template, the target visual features are directly used as the template embedding features.

[0058] If text templates and image templates, or multiple similar templates, exist, fusion is required. A basic approach is to concatenate or weighted average these feature vectors. Another approach is to use an attention fusion module. For example, text features and target visual features can be treated as a set of "key-value" pairs, and based on the query vector, the weights of each feature can be calculated using an attention mechanism, followed by weighted fusion to generate template embedding features that dynamically reflect the importance of information from different modalities.

[0059] In the implementation of the above embodiments: A pre-trained cross-modal model is used to process text, accurately understanding the user's linguistic intent and transforming it into computable semantic features. Image templates are processed using convolutional networks combined with visual cue masks, effectively eliminating background interference and extracting the core visual features of the target object. Finally, by fusing the two types of features to generate a unified template embedding feature, the system can comprehensively utilize the semantic information of the text description and the appearance information of the image example to form a more comprehensive and stable feature representation of the target. This process improves the system's adaptability to diverse target descriptions and enhances its ability to extract key features from complex template information, laying the foundation for subsequent matching in search images.

[0060] Optionally, in this embodiment of the application, template embedding features are generated based on text features and target visual features, including: By using an attention mechanism, text features corresponding to multiple text templates and target visual features corresponding to multiple image templates are dynamically fused to generate template embedding features.

[0061] The attention mechanism is used to assign appropriate importance weights to different parts of the input information according to the needs of the current task. In this embodiment, the attention mechanism is used to evaluate the importance of each template feature to the final unified representation of the target. Dynamic fusion means that this allocation of importance is not fixed in advance, but is calculated in real time and adaptively based on the specific search image content. The template embedding feature is a feature vector that integrates all input template information after weighted fusion.

[0062] This is typically achieved through a cross-attention module. First, multiple template features, which may include text features from text templates and target visual features from image templates, are concatenated in terms of quantity to form a feature set. Each feature vector in this set is considered a potential, valuable clue. The search image features of the current frame are used as the query. Then, the relevance of this query to each template feature in the feature set (as the "key") is calculated. Relevance calculation can be performed using methods such as dot product or scaled dot product. The results are normalized using a Softmax function to obtain a set of "attention weights," which reflect the importance of each template feature for locating the target in the current search scenario.

[0063] Finally, this set of weights is used to sum the corresponding template features (as "values") to generate a unified template embedding feature that is highly relevant to the current search context. For example, if the search image has a complex background, template features describing the details of the target's appearance may receive higher weights; if the target is small, text features indicating the target category may be more critical.

[0064] In the implementation of the above embodiments: by intelligently and comprehensively utilizing multiple template information that may come from different modalities, the attention mechanism enables the system to no longer treat all templates equally or simply superimpose them, but rather to determine which template features are more relevant or reliable based on the specific search scenario, and assign them higher weights. This dynamic, context-aware fusion method improves the quality and relevance of the generated template embedding features. This makes the target prototype features used for matching more adaptable to complex real-world environments and enhances robustness to multi-source information that may contain redundancy or contradictions.

[0065] Optionally, in this embodiment of the application, after encoding the text template into text features using a pre-trained cross-modal model, the method further includes: Text features are input into the feature alignment network module for nonlinear transformation, and the output is a fine-tuned text feature. The feature alignment network module is used to perform nonlinear transformation on the text features to enhance the semantic alignment between the text features and visual features in the task feature space.

[0066] Nonlinear transformation refers to mathematical operations performed within this module where the input and output change disproportionately; for example, it can be achieved through a combination of activation functions and linear layers. The fine-tuned text features are the new feature vectors obtained after the original text features have been transformed by this module. Semantic alignment measures the degree of proximity between text features and visual features in the same task feature space, representing the same concept or object; higher alignment means that the text description and its corresponding target visual appearance are closer in the feature space.

[0067] The feature alignment network module can employ a simple multi-layer feedforward network structure. For example, the feature alignment network module receives text feature vectors from a pre-trained model as input. These text feature vectors first pass through one or more fully connected layers, i.e., linear transformation layers. Each fully connected layer can be followed by a non-linear activation function to provide non-linear transformation capabilities. For instance, a feedforward network based on gated linear units can be used, with gating mechanisms controlling the information flow, allowing for more flexible learning of feature transformations. The output dimension of the feature alignment network module is typically consistent with the input dimension, i.e., the fine-tuned text features.

[0068] During model training, the feature alignment network module is trained end-to-end along with the entire tracking network. The loss function drives the feature alignment network module to learn to bring general textual semantic features closer to regions that are more consistent with the visual feature distribution of the target in the current tracking task, thereby enhancing semantic alignment. For example, after processing by this module, the textual features of the red car will be closer to the visual features of various red cars in the feature space, while being farther away from the features of the blue truck.

[0069] Based on text features and target visual features, template embedding features are generated, including: generating template embedding features based on fine-tuned text features and target visual features.

[0070] Understandably, if the text features are input to the feature alignment network module and undergo a non-linear transformation to generate fine-tuned text features, then the feature fusion step will be based on the fine-tuned text features and the target visual features. In the implementation of the above embodiments: to ensure that the text features not only contain rich semantics but also better match the visual feature distribution in the UAV tracking scenario, this application uses a feature alignment network module to perform nonlinear transformation on the text features, making targeted fine-tuning of the original text features to better adapt them to the needs of the tracking task. This process enhances the semantic alignment between text features and visual features in the task feature space, making text features and image features with the same meaning closer together in the feature space. This improves the ability to understand and respond to text commands, thereby increasing the accuracy of subsequent cross-modal matching.

[0071] Optionally, in this embodiment, an attention mechanism is used to dynamically fuse text features corresponding to multiple text templates and target visual features corresponding to multiple image templates to generate template embedding features, including: Through an attention mechanism, based on the search image features, the text features corresponding to multiple text templates and the target visual features corresponding to multiple image templates are dynamically weighted and fused to generate template embedding features. The attention mechanism is a cross-attention mechanism, which uses the search image features as the query and the combined features corresponding to multiple text templates and image templates as the key and value for calculation.

[0072] Cross-attention mechanisms allow query information to interact and focus with "key" and "value" information. Search image features refer to feature maps containing spatial location information extracted from the current real-time image to be searched. In this embodiment, search image features serve as the query, and their semantics can be "what I am looking for." The text features corresponding to multiple text templates and the target visual features corresponding to multiple image templates are collectively referred to as multiple template features, which can serve as "keys" and "values." The key is used to calculate relevance with the query, and the value is the actual weighted and aggregated information ontology.

[0073] Dynamic weighted fusion refers to the process of summing the corresponding values ​​(also template features) based on the weights of the query (search image features) and each key (each template feature), which are calculated in real time and reflect their relevance. Template embedding features are the result of the weighted summation, a unified feature vector that integrates all template information and is highly relevant to the current search scenario.

[0074] The calculation method is as follows: The features corresponding to all templates to be fused (e.g., multiple text templates and multiple image templates) are concatenated together, mapping them to a key matrix and a value matrix. The dot product similarity between the query vector and each vector in the key matrix is ​​calculated, and then normalized using the Softmax function to obtain attention weights. This set of attention weights quantifies the importance of each template feature for locating the target in the current search scenario; as the search image changes, this weight information also changes, thus being dynamic. Then, this set of weights is used to perform a weighted summation of all vectors in the value matrix, resulting in a single vector, which is the template embedding feature.

[0075] In the implementation of the above embodiments: by using the search image features as query vectors to query various template features, it is possible to automatically determine which template information is more critical and reliable based on the current specific environment, and allocate fusion weights accordingly. This dynamic weighting method, compared with simple averaging or concatenation, can more effectively integrate multi-source information, reduce the interference of irrelevant or contradictory templates, and thus generate a higher-quality and more targeted template embedding feature. This improves robustness and adaptability when facing multiple potentially ambiguous, redundant, or cross-modal template inputs, provides a more reliable basis for the search image, and improves the success rate and accuracy of target tracking.

[0076] Optionally, in this embodiment of the application, a similarity heatmap is generated by contrastive learning based on template embedding features and search image features, including: The search image features are transformed into a set of spatial location feature vectors, where each spatial location feature vector corresponds to a candidate region in the search image. The search image features can be a 3D tensor feature map with spatial dimensions (height H and width W) and channel dimension C. The set of spatial location feature vectors refers to unfolding the feature map in space, treating each C-dimensional feature vector at position (H, W) as an independent vector, resulting in a set of H x W feature vectors. Each vector is a spatial location feature vector, representing the visual content of a corresponding region in the original search image; this region is a candidate region.

[0077] The search image features can be reshaped from a tensor of shape [C, H, W] to a two-dimensional matrix of shape [H*W, C]. Each row of this matrix is ​​a spatial location feature vector, forming a set for comparison.

[0078] Calculate the cosine similarity between the template embedding features and the feature vectors at each spatial location in the set. Map all the calculated cosine similarity values ​​back to a two-dimensional grid that matches the spatial dimension of the search image features to form a similarity heatmap.

[0079] The cosine similarity between the template embedding features and each vector in the set is calculated. Cosine similarity is obtained by calculating the dot product of the two vectors and dividing by the product of their respective magnitudes. Its value ranges from -1 to 1; a larger value indicates greater similarity in direction (semantics) between the two vectors. This calculation can be efficiently performed with a single matrix multiplication. Finally, all H*W calculated cosine similarity values ​​are rearranged into a two-dimensional matrix of height H and width W according to their original spatial positions in the feature map (i.e., which candidate region they correspond to). This matrix is ​​the similarity heatmap, where the grayscale value of each pixel intuitively represents the matching confidence between the corresponding image location and the template; brighter areas indicate a higher probability of containing the target.

[0080] In the implementation of the above embodiments: the search image features are converted into a vector set, enabling similarity calculation for all candidate regions in the image. This effectively measures the similarity of semantic features, and the calculation process is efficient and stable. The resulting similarity heatmap more intuitively identifies the regions in the search image most likely to contain the target and their confidence distribution.

[0081] In an optional embodiment, during model training, positive sample pairs are defined as "templates with matching targets in the search graph," and negative sample pairs are defined as "templates without matching targets in the search graph." For image templates, a tracking dataset can be selected, and the tracking sequences can be automatically converted into template matching pairs, or template and search graph pairs can be manually constructed. For text templates, the CLIP model can be used in advance to generate feature codes corresponding to text categories offline, and positive and negative sample pairs can be constructed using images from the detection dataset.

[0082] During model training, positive and negative sample pairs are used as input for end-to-end joint optimization. The parameters of the entire network, including feature extraction, fusion, contrastive learning, and the DFL detection head, are updated together through backpropagation. The total loss function of training consists of two parts: first, the distribution focus loss (DFL) used to optimize bounding box regression, which improves localization accuracy by making the network's predicted coordinate probability distribution approximate the true values; second, the cross-entropy loss used to optimize classification, which drives the model's output class confidence to be consistent with the true matching situation (positive / negative samples). By minimizing the overall loss, the model learns to extract effective features from multimodal templates and accurately match and locate targets in the search image.

[0083] It is understood that the feature extraction network, feature fusion mechanism, and similarity calculation method used in the above embodiments are merely examples. In other embodiments of this application, the feature extraction network may also adopt lightweight architectures such as ShuffleNet and MobileNet; feature fusion may also adopt weighted summation or direct concatenation; and similarity calculation may also be based on an attention mechanism.

[0084] Optionally, in this embodiment of the application, determining the bounding box of the target object in the search image based on the similarity heatmap includes: A similarity heatmap, serving as a confidence guide, is input into the object detection head along with search image features. The confidence guide can be viewed as each pixel value representing an initial probability or importance weight of the presence of an object at its corresponding image location. The object detection head is a dedicated network module attached to the backend of the feature extraction network, responsible for outputting the final detection results.

[0085] The object detection head is based on search image features and uses confidence to guide the focusing of potential target regions, and regresses the bounding box coordinates and category confidence of the target object; among them, the object detection head uses distributed focus loss to optimize the regression prediction of bounding box coordinates.

[0086] A potential target region refers to an image region with a higher confidence score, indicating a greater likelihood of containing a target. Class confidence is another scalar output by the target detection head, representing the degree of certainty that a target exists within the bounding box and belongs to the category specified by the template. Distributed focus loss is a specialized loss function used to optimize bounding box regression. It models the coordinates as a discrete probability distribution, indirectly obtaining more stable and accurate coordinate predictions by having the network learn this distribution.

[0087] One implementation approach is to use the similarity heatmap as a confidence guide, i.e., spatial attention weights, and multiply it element-wise with the search image features. This amplifies the feature response of high-confidence regions and suppresses low-confidence regions, resulting in features adjusted for similarity heatmaps. Alternatively, the original features can be directly concatenated with the heatmap along the channel dimension to form combined features. The adjusted features or combinations are then input into the object detection head. The object detection head can contain two parallel sub-networks: one for classification (outputting class confidence) and one for regression (outputting bounding box coordinates).

[0088] In the regression branch, bounding box coordinates are predicted for each preset anchor point or directly for each spatial location. The role of the distribution focus loss is to predict a discrete probability distribution within a range of possible values ​​for each coordinate (e.g., center point x). During training, supervisory signals encourage the network to focus the probability quality around the true coordinate values. During inference, the expected value of this distribution is calculated to obtain the final coordinate prediction. This makes the bounding box regression process more robust to small changes in coordinate values, thus achieving more accurate localization results in complex scenes.

[0089] In the implementation of the above embodiments: by using a similarity heatmap as a confidence guide and integrating it into the target detection process, a clear spatial prior is provided, enabling the concentration of computational resources on the regions most likely to contain the target, reducing computational waste on irrelevant backgrounds, and improving localization efficiency. The use of distributed focus loss to optimize the regression of bounding box coordinates enhances the ability to model subtle changes in target position, resulting in more accurate and stable final output bounding box coordinates. By fully utilizing information from the preceding matching stage, target detection is no longer a blind search across the entire image, but rather precise localization guided by strong confidence, thereby improving the accuracy and reliability of the final output bounding box and providing high-quality initialization input for subsequent tracking tasks.

[0090] Please see Figure 2 The diagram shown is a schematic representation of the structure of a target tracking device based on multimodal matching provided in an embodiment of this application. This application provides a target tracking device 200 based on multimodal matching, comprising: The acquisition module 210 is used to acquire multimodal template information and a search image containing the target object; the multimodal template information includes at least one image template and at least one text template, and the image template contains the bounding box of the target object; The template extraction module 220 is used to extract features from multimodal template information to obtain template embedding features; the template embedding features are feature representations that integrate multimodal template information. Image extraction module 230 is used to extract features from the search image to obtain the search image features; The matching module 240 is used to generate a similarity heatmap by contrastive learning based on template embedding features and search image features. The similarity heatmap represents the matching confidence of each position in the search image with the template. The boundary determination module 250 is used to determine the bounding box of the target object in the search image based on the similarity heatmap, and to perform target tracking based on the bounding box.

[0091] Optionally, in this embodiment of the application, the target tracking device 200 based on multimodal matching and the template extraction module 220 are used to encode the text template into text features using a pre-trained cross-modal model; extract the visual feature map of the image template using a convolutional neural network, and generate a visual cue mask based on the bounding box contained in the image template; extract the target visual features from the visual feature map based on the visual cue mask; and generate template embedding features based on the text features and the target visual features.

[0092] Optionally, in this embodiment of the application, the target tracking device 200 based on multimodal matching and the template extraction module 220 are used to dynamically fuse the text features corresponding to multiple text templates and the target visual features corresponding to multiple image templates through an attention mechanism to generate template embedding features.

[0093] Optionally, in this embodiment, the target tracking device 200 based on multimodal matching further includes a linear transformation module, used to input text features into a feature alignment network module for nonlinear transformation, and output fine-tuned text features; wherein, the feature alignment network module is used to perform nonlinear transformation on the text features to enhance the semantic alignment between the text features and visual features in the task feature space; generating template embedding features based on text features and target visual features includes: generating template embedding features based on fine-tuned text features and target visual features.

[0094] Optionally, in this embodiment of the application, the target tracking device 200 based on multimodal matching and the template extraction module 220 are used to dynamically weight and fuse the text features corresponding to multiple text templates and the target visual features corresponding to multiple image templates according to the search image features through an attention mechanism to generate template embedding features; wherein, the attention mechanism is a cross-attention mechanism, which uses the search image features as the query and uses the combined features corresponding to multiple text templates and image templates as the key and value for calculation.

[0095] Optionally, in this embodiment of the application, the target tracking device 200 based on multimodal matching includes a matching module 240, which is used to convert the search image features into a set of spatial location feature vectors, wherein each spatial location feature vector corresponds to a candidate region in the search image; calculate the cosine similarity between the template embedding features and each spatial location feature vector in the set; and map all the calculated cosine similarity values ​​back to a two-dimensional grid that matches the spatial dimension of the search image features to form a similarity heatmap.

[0096] Optionally, in this embodiment of the application, the target tracking device 200 based on multimodal matching has a boundary determination module 250, which is used to input the target detection head together with the search image features, using a similarity heatmap as a confidence guide; the target detection head focuses on the potential target region based on the search image features and using the confidence guide, and regresses and outputs the bounding box coordinates and category confidence of the target object; wherein, the target detection head uses distributed focus loss to optimize the regression prediction of the bounding box coordinates.

[0097] It should be understood that this device corresponds to the above-described target tracking method embodiment based on multimodal matching, and is capable of performing the various steps involved in the above method embodiment. The specific functions of this device can be found in the description above, and detailed descriptions are omitted here to avoid repetition. The device includes at least one software functional module that can be stored in memory or embedded in the device's operating system (OS) in the form of software or firmware.

[0098] Please see Figure 3 The diagram shows a structural schematic of an electronic device provided in an embodiment of this application. An electronic device 300 provided in this application includes a processor 310 and a memory 320. The memory 320 stores machine-readable instructions executable by the processor 310. When the machine-readable instructions are executed by the processor 310, the method described above is performed.

[0099] Figure 3The components shown can be implemented using hardware, software, or a combination thereof. Electronic device 300 may be a physical device, such as a server or PC, or a virtual device, such as a virtual machine or virtualization container. Furthermore, electronic device 300 is not limited to a single device; it can be a combination of multiple devices or a cluster of numerous devices.

[0100] This application also provides a storage medium storing a computer program, which is executed by a processor to perform the above-described method.

[0101] The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0102] This application also provides a computer program product, including computer program instructions, which are executed by a processor to perform the method described above.

[0103] It should be understood that the disclosed apparatus and methods can also be implemented in other ways, as provided in the embodiments of this application. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and flowcharts, as well as combinations of blocks in the block diagrams and flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0104] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0105] The above description is only an optional implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application.

Claims

1. A target tracking method based on multimodal matching, characterized in that, include: Obtain multimodal template information and a search image containing the target object; the multimodal template information includes at least one image template and at least one text template, the image template containing the bounding box of the target object; Feature extraction is performed on the multimodal template information to obtain template embedding features; the template embedding features are feature representations that incorporate the multimodal template information. Feature extraction is performed on the search image to obtain search image features; Based on the template embedding features and the search image features, a similarity heatmap is generated through contrastive learning. The similarity heatmap represents the matching confidence of each position in the search image with the template. Based on the similarity heatmap, the bounding box of the target object in the search image is determined, and target tracking is performed based on the bounding box; Feature extraction is performed on the multimodal template information to obtain template embedding features, including: The text template is encoded into text features using a pre-trained cross-modal model; The visual feature map of the image template is extracted using a convolutional neural network, and a visual cue mask is generated based on the bounding box contained in the image template. The target visual features are then extracted from the visual feature map based on the visual cue mask. The template embedding features are generated based on the text features and the target visual features; Based on the text features and the target visual features, the template embedding features are generated, including: Through an attention mechanism, based on the search image features, the text features corresponding to multiple text templates and the target visual features corresponding to multiple image templates are dynamically weighted and fused to generate the template embedding features; wherein, the attention mechanism is a cross-attention mechanism, which uses the search image features as the query and uses the combined features corresponding to multiple text templates and image templates as the key and value for calculation; The method further includes: The similarity heatmap is used as a spatial attention weight and multiplied element-wise with the search image features to amplify the feature response of high-confidence regions and suppress low-confidence regions, thereby obtaining features adjusted by similarity heatmap.

2. The method according to claim 1, characterized in that, After encoding the text template into text features using a pre-trained cross-modal model, the method further includes: The text features are input into the feature alignment network module for nonlinear transformation, and the fine-tuned text features are output. The feature alignment network module is used to perform nonlinear transformation on the text features to enhance the semantic alignment between the text features and visual features in the task feature space. Based on the text features and the target visual features, the template embedding features are generated, including: Based on the fine-tuned text features and the target visual features, the template embedding features are generated.

3. The method according to claim 1, characterized in that, Based on the template embedding features and the search image features, a similarity heatmap is generated through contrastive learning, including: The search image features are converted into a set of spatial location feature vectors, where each spatial location feature vector corresponds to a candidate region in the search image; Calculate the cosine similarity between the template embedding features and each spatial location feature vector in the set; All calculated cosine similarity values ​​are mapped back to a two-dimensional grid that matches the spatial dimension of the search image features to form the similarity heatmap.

4. The method according to claim 1, characterized in that, Determining the bounding box of the target object in the search image based on the similarity heatmap includes: Using the similarity heatmap as a confidence guide, it is input into the target detection head along with the search image features; The target detection head focuses on potential target regions based on the search image features and uses the confidence level as a guide, and regresses the bounding box coordinates and category confidence of the target object; wherein, the target detection head uses distributed focus loss to optimize the regression prediction of the bounding box coordinates.

5. A computer program product, characterized in that, It includes computer program instructions that are executed by a processor to perform the method as described in any one of claims 1 to 4.

6. An electronic device, characterized in that, include: A processor and a memory, the memory storing computer program instructions that, when executed by the processor, perform the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, perform the method as described in any one of claims 1 to 4.

Images