A target tracking method, device and equipment

Abstract

The present disclosure provides a target tracking method, apparatus and device, belonging to the technical field of target tracking, and aims to solve the problem that the target tracking method in the related art cannot effectively track a target. The method includes: obtaining a search area sequence corresponding to search area images of each modality, and obtaining a first target sequence corresponding to each dynamic template and a second target sequence corresponding to each static template, where the static template represents the initial features of the tracking target, and the dynamic template represents the motion features of the tracking target; inputting the search area sequences, the first target sequence, and the second target sequence of each modality into an encoding layer with shared weights to perform temporal information fusion and inter-modal information fusion, so as to obtain a fusion feature sequence for each modality; and bounding the tracking target in the search area image according to the fusion feature sequences of each modality.

Description

Technical Field

[0001] The present disclosure relates to the technical field of object tracking, and particularly to an object tracking method, apparatus and device. Background Art

[0002] Visual object tracking is a basic task in computer vision. Its main research task is to accurately locate and track an object in each subsequent video frame given the initial state of the current frame. Given its great potential and excellent performance, visual object tracking has been widely applied in fields such as video surveillance, autonomous driving, and robot navigation. However, due to the limitations of the visible light image imaging mechanism, VOT (Visual Object Tracking) still faces challenging problems, especially in complex scenarios and environments such as extreme lighting and bad weather. This greatly limits the application scope of object tracking technology.

[0003] Thermal infrared (TIR) imaging is insensitive to light changes, has a certain penetration ability, and complements the rich color, detail, and texture information of visible light. Therefore, TIR images can well serve as supplementary information for visible light images.

[0004] Therefore, in related technologies, object tracking technology based on the fusion of visible light and thermal infrared images (RGB-T, RGB-T Image Fusion) has become a popular research topic in recent years and has potential application value in fields such as intelligent monitoring, autonomous driving, and robots.

[0005] However, existing RGB-T multimodal object tracking methods lack sufficient utilization of time information and the fusion between modalities is not sufficient and reasonable enough, resulting in the model being unable to track the object more robustly. Summary of the Invention

[0006] To overcome the problems existing in related technologies, the present disclosure provides an object tracking method, apparatus and device. The technical solution of the present disclosure is as follows:

[0007] According to the first aspect of the embodiments of the present disclosure, an object tracking method is provided, including:

[0008] Obtaining a search region sequence corresponding to search region images of each modality, and obtaining a first target sequence corresponding to each dynamic template and a second target sequence corresponding to each static template, where the static template represents the initial features of the tracking object, and the dynamic template represents the motion features of the tracking object;

[0009] Input the search region sequences, first target sequence, and second target sequence of each modality into an encoding layer with shared weights to perform temporal information fusion and cross-modal information fusion, obtaining a fused feature sequence for each modality;

[0010] Based on the fused feature sequences of each modality, select the tracking target in the search region image.

[0011] Optionally, inputting the search region sequences, first target sequence, and second target sequence of each modality into an encoding layer with shared weights for temporal information fusion and cross-modal information fusion to obtain a fused feature sequence for each modality includes:

[0012] Perform temporal information fusion on the first target sequence and the second target sequence to obtain a target feature sequence, which is used to capture the behavior of the tracking target;

[0013] Based on the target feature sequence, perform feature enhancement on the search region sequence to obtain an enhanced feature sequence, where the feature enhancement is used to distinguish background information and tracking target information in the search region sequence;

[0014] Perform cross-modal information interaction according to the search region sequences of different modalities to obtain cross-modal prompt information;

[0015] Fuse the enhanced feature sequence and the cross-modal prompt information into a fused feature sequence.

[0016] Optionally, performing temporal information fusion on the first target sequence and the second target sequence to obtain a target feature sequence includes:

[0017] Perform dimensionality reduction information extraction on the second target sequence to obtain cross-time prompt information;

[0018] Upscale the cross-time prompt information to the same dimension as the feature information in the first target sequence and superimpose it with the feature information in the first target sequence to obtain a first target sequence with superimposed cross-time prompt information;

[0019] Determine the first target sequence with superimposed cross-time prompt information as the target feature sequence.

[0020] Optionally, performing cross-modal information interaction according to the search region sequences of different modalities to obtain cross-modal prompt information includes:

[0021] Perform dimensionality reduction information extraction on the search region sequences of each modality to obtain first cross-modal prompt information;

[0022] Dimensionality-expand the first cross-modal prompt information to the same dimension as the feature information in the search region sequence to obtain a first dimensionality-expanded result; the first dimensionality-expanded result is used for superimposing with the enhanced feature sequence of another modality.

[0023] Optionally, after enhancing the features of the search region sequence based on the target feature sequence to obtain an enhanced feature sequence, it further includes:

[0024] Receive a second dimensionality-expanded result obtained from the search region sequence of another modality;

[0025] Superimpose the second dimensionality-expanded result with the enhanced feature sequence to obtain the fused feature sequence after cross-modal fusion.

[0026] Optionally, after bounding the tracking target in the search region image, it further includes:

[0027] Determine the image to be updated through the bounding region in the search region image;

[0028] Evaluate the image to be updated through position information, discriminant information, and appearance information to determine whether the image to be updated passes the evaluation;

[0029] In the case where the image to be updated passes the evaluation, update the dynamic template and determine the image to be updated as the updated dynamic template;

[0030] In the case where the image to be updated fails to pass the evaluation, do not update the dynamic template.

[0031] Optionally, bounding the tracking target in the search region image according to the fused feature sequences of each modality includes:

[0032] Determine the classification maps of the fused feature sequences corresponding to each modality respectively;

[0033] According to each of the classification maps, perform weighted fusion on the fused feature sequences of each modality to obtain a weighted fused feature sequence;

[0034] Respectively determine the corresponding tracking results through the fused feature sequences corresponding to each modality and the weighted fused feature sequence;

[0035] Evaluate each of the tracking results through evaluation metrics to obtain the corresponding evaluation results;

[0036] Determine the tracking result with the best evaluation result as the target tracking result;

[0037] Bounding the tracking target in the search region image through the target tracking result.

[0038] Optionally, the search area sequence, the first target sequence, and the second target sequence are determined in the following manner, including:

[0039] Segment and expand the image into a sequence composed of multiple patches according to a set patch size; the image is a search area image, a static template image, or a dynamic template image;

[0040] Project the sequence composed of the multiple patches into a high-dimensional space through a convolutional network to obtain a feature sequence;

[0041] Wherein, when the image is a search area image, after projecting the sequence composed of the multiple patches into a high-dimensional space, add the projection result to the position information embedding to obtain a feature sequence; the position information is the position of the patch in the search area image.

[0042] According to a second aspect of the embodiments of the present disclosure, there is provided an object tracking device, including:

[0043] An acquisition module, configured to acquire the search area sequence corresponding to the search area image of each modality, and acquire the first target sequence corresponding to each dynamic template and the second target sequence corresponding to each static template, where the static template represents the initial feature of the tracking target, and the dynamic template represents the motion feature of the tracking target;

[0044] A fusion module, configured to input the search area sequences, the first target sequences, and the second target sequences of each modality into an encoding layer with shared weights to perform temporal information fusion and inter-modal information fusion, and obtain a fusion feature sequence for each modality;

[0045] A tracking module, configured to frame the tracking target in the search area image according to the fusion feature sequences of each modality.

[0046] According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where when the computer program is executed by the processor, the steps of the object tracking method described in the first aspect are implemented.

[0047] According to a fourth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the object tracking method described in the first aspect are implemented.

[0048] According to a fifth aspect of the embodiments of the present disclosure, there is provided a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of the object tracking method described in the first aspect are implemented.

[0049] By performing temporal fusion and cross-modal information fusion, the present disclosure can effectively enhance the information interaction between different modalities, capture the features of the target more comprehensively, and improve the accuracy and robustness of tracking. The dynamic template can reflect the motion characteristics of the tracked target, while the static template represents the initial characteristics of the target. The present disclosure can better adapt to the appearance changes and motion states of the target at different time points, thereby improving the tracking stability. By processing the fusion feature sequences of each modality, the tracked target can be more accurately framed in the search area image, effectively reducing the situations of false detection and missed detection, and improving the overall tracking performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] To more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings required for the description of the embodiments of the present disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0051] Figure 1 is a schematic diagram of the steps of a target tracking method shown in an embodiment of the present disclosure;

[0052] Figure 2 is a schematic diagram of the working process of a spatio-temporal bidirectional fusion multi-modal fusion target tracking method based on template update shown in an embodiment of the present disclosure;

[0053] Figure 3 is a block diagram of a target tracking device shown in an embodiment of the present disclosure;

[0054] Figure 4 is a schematic diagram of an electronic device shown in an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present disclosure with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are some, but not all, of the embodiments of the present disclosure. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present disclosure without creative efforts fall within the scope of protection of the present disclosure.

[0056] The terms "first", "second", etc. in the description and claims of the present disclosure are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same category, and the number of objects is not limited. For example, the first object can be one or multiple. In addition, "and / or" in the description and claims means at least one of the connected objects, and the character " / " generally indicates an "or" relationship between the associated objects before and after.

[0057] The method of using a Vision Transformer as the backbone network for feature extraction and correlation modeling of images can be divided into two categories: full fine-tuning method and prompt fine-tuning method.

[0058] The full fine-tuning method fully optimizes the entire network parameters. This method can largely perform in-depth interaction of feature information between the two modalities, improving the robustness of model tracking. However, this method is parameter-inefficient and requires sufficient data to optimize all parameters.

[0059] As a new paradigm of parameter-efficient tuning, prompt fine-tuning well alleviates the defect of insufficient data volume in the RGB-T field. This method was first applied in the field of natural language processing (NLP, Natural Language Processing). Some early methods generated effective visual prompts by introducing modality-complementary prompt generators, simplifying the auxiliary modality into a small amount of prompt information. These methods will preset a dominant modality (usually the RGB modality). However, when the dominant modality switches, the modality complementarity cannot be fully and effectively utilized. To solve this problem, subsequent methods often set up bidirectional feature interactors to perform feature interaction in a bidirectional prompt manner.

[0060] However, although prompt fine-tuning, as a parameter-efficient tuning method, has improved the tracking ability of RGB-T trackers to a certain extent, their effects are still limited by various factors, mainly including two problems.

[0061] The first problem: Existing trackers lack sufficient utilization of time information. Current research mainly focuses on further enhancing the complementary information interaction and fusion between the visible light modality and the thermal modality to achieve more efficient tracking. Most methods ignore the utilization of time series information. For long-term tracking, relying solely on the template information of the initial frame is insufficient to handle the complex scenarios (target appearance changes, deformations, rapid movements, etc.) that the tracker may encounter. Some existing methods use the tracking results of each frame to generate new training samples and online train the tracker during the tracking process. Or the training sample update is divided into long-term dynamic updates that are continuously positive samples and short-term updates where the predicted target score is less than a manually set threshold. This update method requires offline training of the tracker and updating of domain-specific layers during tracking. However, there is a common problem with the above methods. Due to the lack of judgment on the reliability of the tracking results, this continuous update is prone to introducing cumulative errors and losing the original feature information of the target.

[0062] The second problem: Existing trackers do not fuse modalities sufficiently and reasonably. Existing tracking methods believe that during the feature interaction process of the backbone network, the feature information of the two modalities has been fully interacted and fused. In the final classification and regression stage of generating the target bounding box based on the feature information, the feature information of the visible light modality and the thermal infrared modality is often directly added or concatenated. This method generally does not have a large impact. When a certain modality fails completely, this direct superposition or concatenation will inevitably introduce the noise of the failed modality into the feature information of the other modality, making the cross-modal interaction rough and insufficient.

[0063] To solve the above problems, the present disclosure proposes an object tracking method that can make good use of time information and strengthen the fusion between modalities.

[0064] Figure 1 is a schematic diagram of the steps of an object tracking method shown in an embodiment of the present disclosure. According to Figure 1 as shown, the method may specifically include the following steps:

[0065] Step S11: Obtain the search region sequence corresponding to the search region images of each modality, and obtain the first target sequence corresponding to each dynamic template and the second target sequence corresponding to each static template, where the static template represents the initial features of the tracking target, and the dynamic template represents the motion features of the tracking target.

[0066] Obtain image data of the tracking area from different sensors or data sources, such as visible light cameras, infrared cameras, radars, etc. Ensure the synchronization of the timestamps of the search region images of each modality.

[0067] and obtain the dynamic template and static template corresponding to each modality. Both the dynamic template and the static template are a kind of image. Combining the dynamic template and the static template can significantly improve the accuracy and robustness of target tracking.

[0068] The dynamic template refers to the template image that is continuously updated over time during the target tracking process. It can be the image of the tracking target in the search area image determined during the latest target tracking. The dynamic template is usually adjusted based on the appearance and motion state of the target in the latest frame to reflect the real-time changes of the target. The latest motion features of the target can be captured through the dynamic template, which helps the tracker better adapt to the dynamic changes of the target, thereby improving the accuracy and robustness of tracking. And when the appearance of the target changes significantly, the dynamic template can help the tracker avoid tracking drift caused by template failure.

[0069] The static template refers to the fixed template image representing the initial features of the target during the target tracking process. The static template is usually the target image captured at the beginning of tracking, which reflects the basic appearance features of the target. The static template provides a reference point for target tracking, enabling the tracker to quickly perform matching and recognition when the appearance of the target does not change much. And when the dynamic template fails or is updated in a timely manner, the static template can be used as an alternative to help maintain the stability of tracking.

[0070] Apply the feature extraction algorithm to the search area image, dynamic template, and static template of each modality to generate their respective feature sequences. Obtain the search area sequence from the search area image, the first target sequence from the dynamic template, and the second target sequence from the static template.

[0071] Step S12: Input the search area sequences, the first target sequences, and the second target sequences of the respective modalities into the weight-sharing encoding layer for temporal information fusion and cross-modal information fusion to obtain the fusion feature sequence of each modality.

[0072] The corresponding branch networks in the weight-sharing encoding layer process the search area sequences, the first target sequences, and the second target sequences corresponding to each modality respectively.

[0073] In the weight-sharing encoding layer, perform temporal information fusion and cross-modal information fusion on the feature sequences of each modality. Temporal information fusion acts within each modality and realizes temporal information fusion based on the first target sequence and the second target sequence corresponding to the modality itself; cross-modal information fusion acts between modalities and realizes cross-modal information fusion based on the search area sequences of each modality.

[0074] Temporal information fusion helps the tracker better understand the dynamic changes of the tracked object by integrating the information in the time series. It can provide the context of the movement trajectory and appearance changes of the tracked object in the time series, thereby improving the prediction accuracy of the object's position.

[0075] Inter-modal information fusion can effectively integrate the feature information from different modalities and provide complementary information of another modality for different modalities.

[0076] Step S13: Select the tracked object in the search area image according to the fusion feature sequences of each modality.

[0077] Perform object tracking based on the fusion feature sequences of each modality respectively to determine the corresponding tracking results. Determine the object tracking result according to the tracking results corresponding to each modality. Select the tracked object in the search area images corresponding to each modality according to the object tracking result.

[0078] Adopting the embodiments of the present disclosure, through inter-modal information fusion, the complementarity between different modality data can be fully utilized to enhance the comprehensive understanding of the tracked object. Data of different modalities can often provide different effective information under different environmental conditions, such as light changes, occlusion, complex backgrounds, etc., thereby improving the robustness of tracking. Using a static template to represent the initial features of the tracked object helps to maintain the consistent recognition of the object's identity during the tracking process, while the dynamic template can capture the appearance changes of the object in a timely manner. Combining the static template and the dynamic template enables the tracking system to adapt to both the static attributes and dynamic behaviors of the object.

[0079] Figure 2 It is a schematic diagram of the working process of a spatio-temporal bidirectional fusion multi-modal fusion object tracking method based on template update shown in the embodiments of the present disclosure. According to Figure 2 As shown, the encoding layer with weight sharing has two branch networks, which process the data of the RGB modality and the thermal infrared modality respectively. For example, the upper branch network processes the search area sequence, the first object sequence, and the second object sequence corresponding to the RGB modality. Inside the branch network, bidirectional spatio-temporal fusion modules are configured between the Transformer blocks at each level of the branch network for temporal information fusion within a single modality, and bidirectional spatio-temporal fusion modules are configured between the Transformer blocks at the same level between the branch networks for inter-modal information fusion between different modalities. After each branch network is processed by the last Transformer block, the fusion feature sequences corresponding to each modality are obtained; subsequently, the tracked object is selected in the search area image through the fusion feature sequences of each modality.

[0080] Among them, in an optional embodiment, the search area sequence, the first target sequence, and the second target sequence are determined in the following manner, including: segmenting and unfolding an image into a sequence composed of multiple patches according to a set patch size; the image being a search area image, a static template image, or a dynamic template image; projecting the sequence composed of the multiple patches into a high-dimensional space through a convolutional network to obtain a feature sequence; wherein, when the image is a search area image, after projecting the sequence composed of the multiple patches into the high-dimensional space, adding the projection result to the position information embedding to obtain a feature sequence; the position information being the position of the patch in the search area image.

[0081] According to the requirements of image processing and the limitations of computing resources, set an appropriate patch size. The size of the patch can capture local features in the image while maintaining computational efficiency.

[0082] Segment the image into multiple small blocks according to the set patch size, and each small block is called a patch. Unfold the segmented patches into a sequence in a certain order, such as from left to right and from top to bottom.

[0083] After obtaining the sequence composed of patches, use a convolutional neural network to extract the features of the image patches. Through the layer-by-layer processing of the convolutional neural network, key information is extracted from the patches and mapped into a high-dimensional feature space to obtain a feature sequence.

[0084] When the processed image is a search area image, in order to retain the position information of the patch in the search area image, it is necessary to combine the position information with the projected feature sequence. Specifically, it can be achieved by encoding the position information as a vector and adding it to the feature vector of each patch. After adding the position information to the feature sequence, the search area sequence corresponding to the search area image is obtained. The search area sequence not only contains the feature information of the image patches but also the position information of the patches in the search area image.

[0085] Adopting the embodiment of the present disclosure, segmenting the image into multiple patches according to the set patch size and unfolding them into a sequence not only simplifies the image processing process but also makes subsequent feature extraction more efficient. Through the convolutional network, the patch sequence is projected into a high-dimensional space, thereby extracting rich feature information and being able to capture subtle changes and important information in the image. When the image is a tracking area image, after projecting the patch sequence into the high-dimensional space, adding the projection result to the position information embedding of the patch in the tracking area image makes the feature sequence not only contain the feature information of the image itself but also the position information of the patch. Introducing the position information enhances the accuracy of target localization.

[0086] Among them, in an optional embodiment, the search region sequences, the first target sequence, and the second target sequence of each modality are input into an encoding layer with shared weights for temporal information fusion and cross-modal information fusion to obtain a fusion feature sequence for each modality, including: performing temporal information fusion on the first target sequence and the second target sequence to obtain a target feature sequence, where the target feature sequence is used to capture the behavior of the tracking target; based on the target feature sequence, enhancing the features of the search region sequence to obtain an enhanced feature sequence, where the feature enhancement is used to distinguish background information and tracking target information in the search region sequence; performing cross-modal information interaction according to the search region sequences of different modalities to obtain cross-modal prompt information; and fusing the enhanced feature sequence and the cross-modal prompt information to obtain a fusion feature sequence.

[0087] After obtaining the first target sequence and the second target sequence corresponding to each modality, temporal information fusion is performed on the first target sequence and the second target sequence of a single modality to obtain a target feature sequence. Refer to Figure 2 , before using the Transformer block to process the feature data of a single modality, a bidirectional spatio-temporal fusion module is used to perform temporal information fusion on the first target sequence and the second target sequence.

[0088] The target feature sequence can effectively fuse the feature information at different times. The target feature sequence can capture the change trend of the tracking target in the time series, helping the tracking system better understand the dynamic characteristics of the tracking target. This is because the target feature sequence reflects the state of the tracking target at different time points, thus enhancing the tracking system's understanding of the behavior of the tracking target.

[0089] After obtaining the target feature sequence, based on the target feature sequence, the features of the search region sequence are enhanced. Refer to Figure 2 , the target feature sequence and the search region sequence are input into the Transformer block, so as to realize enhancing the features of the search region sequence based on the target feature sequence to obtain an enhanced feature sequence. Enhancing the features of the search region sequence based on the target feature sequence can use the method of template search and matching. The meaning of template search and matching is to study where the pattern of a specific object is located in the image, and then identify the object. It is achieved by sliding on the collected original image to find the target similar to the template image. Through template search and matching, the feature information of the tracking target at different times in the target feature sequence can be effectively referred to, thereby improving the feature enhancement effect of the Transformer block on the target feature sequence, effectively realizing the enhancement of the background features and tracking target features in the target feature sequence, and providing support for subsequent identification of the tracking target.

[0090] In the case of enhancing the features of a single-modal search region sequence, cross-modal prompt information transmitted by other modalities can be received, and the enhanced feature sequence obtained by enhancement is fused with the cross-modal prompt information to obtain the fused feature sequence corresponding to this modality.

[0091] By adopting the embodiments of the present disclosure, through fusing the temporal information of the first target sequence and the second target sequence, the generated target feature sequence can effectively capture the dynamic behavior of the target. Therefore, it can better understand the changes of the target in the time dimension, thereby improving the accuracy and robustness of tracking. Based on the target feature sequence, the features of the search region sequence are enhanced, and the system can more effectively distinguish background information and tracking target information. Thus, false detections and missed detections can be reduced, and the precision of target tracking can be improved. By performing information interaction between modalities according to the search region sequences of different modalities to generate cross-modal prompt information, the complementary nature of different modalities can be fully utilized to enhance the system's ability to understand and identify the target.

[0092] Among them, in an optional embodiment, fusing the temporal information of the first target sequence and the second target sequence to obtain a target feature sequence includes: extracting dimensionality reduction information from the second target sequence to obtain cross-time prompt information; ascending the dimensionality of the cross-time prompt information to the same dimension as the feature information in the first target sequence and superimposing it with the feature information in the first target sequence to obtain the first target sequence with superimposed cross-time prompt information; and determining the first target sequence with superimposed cross-time prompt information as the target feature sequence.

[0093] Reference Figure 2 , the temporal information of the first target sequence and the second target sequence is fused through a bidirectional spatio-temporal fusion module between different hierarchical Transformer blocks in a single-branch network. Each Transformer block consists of multi-head self-attention, a normalization layer, a feed-forward neural network, and a residual connection.

[0094] The bidirectional spatio-temporal fusion module first receives the feature information from the dynamic template and the static template, that is, the first target sequence and the second target sequence. The feature information included in the first target sequence is the feature information of the dynamic template; the feature information included in the second target sequence is the feature information of the static template.

[0095] The received feature information is processed for dimensionality reduction to extract the most important features and remove redundant information, thereby obtaining concise cross-time prompt information. Specifically, methods such as principal component analysis, autoencoder, or convolutional layer can be used to map high-dimensional features to a low-dimensional space. The cross-time prompt information formed by the dimensionality-reduced feature information can reflect the change trend and important features of the tracking target in the time series.

[0096] The downsampled cross-temporal cue information is upsampled to the same dimension as the template feature information, i.e., the feature information in the second target sequence, so as to ensure that the cross-temporal cue information can be effectively combined with the feature information at other times. Specifically, upsampling can be achieved through transposed convolution, fully connected layers, or interpolation methods. During the upsampling process, the integrity of the cross-temporal cue information needs to be maintained to ensure that important features are not lost during upsampling.

[0097] The upsampled cross-temporal cue information is added to the feature information from another time, specifically, the upsampled cross-temporal cue information is added to the feature information in the second target sequence corresponding to the static template. The addition means feature fusion, thus combining the feature information at different time points to form a richer feature representation. Through the addition operation, a new feature representation, i.e., the target feature sequence, is generated. The target feature sequence contains feature information from different times, enhancing the understanding of the tracking target.

[0098] In addition, the cue information of the generated same-modal different-time feature information can be used as a visual cue and cross-transmitted to another modality, so that the time feature information of one modality can be used to guide the feature extraction and processing of another modality. Through this cross-transmission, the time series information can be fully utilized, the complementarity between modalities can be enhanced, and the overall tracking performance can be improved.

[0099] By adopting the embodiments of the present disclosure, the understanding of the tracking target can be enhanced through the temporal information fusion of the first target sequence and the second target sequence. By performing downsampling on the second target sequence, more concise and important cross-temporal cue information can be extracted, which helps to remove redundant information and retain the features most useful for the tracking task, thereby reducing the computational burden. After obtaining the concise cross-temporal cue information, it is upsampled to the same dimension as the feature information of the template image, ensuring that the feature information at different times can be effectively compared and fused in the same space.

[0100] Reference Figure 2 , inter-modal information fusion is achieved by setting a bidirectional spatio-temporal fusion module between Transformer blocks at the same level of different branch networks.

[0101] The bidirectional spatio-temporal fusion module can receive the search region sequences corresponding to each modality, process them to obtain the corresponding cross-modal cue information, and transmit the processed cross-modal cue information to other modalities different from the source of the cross-modal cue information.

[0102] Among them, in an optional embodiment, inter-modal information interaction is performed according to the search region sequences of different modalities to obtain cross-modal prompt information, including: performing dimensionality reduction information extraction on the search region sequences of each modality to obtain first cross-modal prompt information; dimensionality-increasing the first cross-modal prompt information to the same dimension size as the feature information in the search region sequence to obtain a first dimensionality-increasing result; the first dimensionality-increasing result is used for superposition with the enhanced feature sequence of another modality.

[0103] For the search region sequences corresponding to each modality, the bidirectional spatio-temporal fusion module processes them using dimensionality reduction techniques. Through dimensionality reduction processing, the most representative features are extracted to obtain refined cross-modal prompt information.

[0104] Use interpolation or other dimensionality-increasing techniques to dimensionality-increase the dimensionality-reduced cross-modal prompt information to the same dimension as the feature information in the search region sequence, so as to ensure that the dimensionality-increased cross-modal prompt information is spatially compatible with the feature information in the search region sequence of another modality.

[0105] After obtaining the cross-modal prompt information, the bidirectional spatio-temporal fusion module transmits it to other modalities different from the modality corresponding to this cross-modal prompt information, specifically transmitted to the Transformer blocks of other modalities.

[0106] Adopting the embodiments of the present disclosure, by performing dimensionality reduction processing on the search region sequences of different modalities, extracting the most representative features, and generating cross-modal prompt information, it is possible to effectively exchange information between modalities and enhance the understanding of the target. Using dimensionality reduction techniques to process the search region sequences of each modality and extract important features can remove redundant information and make subsequent processing more efficient. Dimensionality-increasing the dimensionality-reduced cross-modal prompt information to the same dimension as the original search region sequence ensures the spatial compatibility of feature information of different modalities. The cross-modal prompt information of different modalities can reflect the state and feature changes of the target in different modalities.

[0107] Among them, in an optional embodiment, after strengthening the feature of the search region sequence based on the target feature sequence to obtain an enhanced feature sequence, it further includes: receiving a second dimensionality-increasing result obtained from the search region sequence of another modality; superimposing the second dimensionality-increasing result and the enhanced feature sequence to obtain the fused feature sequence after cross-modal fusion.

[0108] Take the dimensionality-increasing result obtained from cross-modal prompt information corresponding to one modality as a visual prompt and cross-transmit it to another modality.

[0109] After a modality receives the second up-dimension result obtained from the search area sequence of another modality, it superimposes it with the reinforcement feature sequence. Specifically, it can be a simple element-wise addition or a weighted addition according to specific weights to emphasize the features of a certain modality.

[0110] After the superimposition operation, a fused feature sequence containing complementary features from both modalities is generated.

[0111] Reference Figure 2 , after a Transformer block at a certain level of a certain modality obtains the corresponding reinforcement feature sequence at that level, the individual output of this modality at the current layer is obtained. In the next step, the reinforcement feature sequence is superimposed with the up-dimension result corresponding to the cross-modal prompt information of another modality to obtain the output of the entire current layer.

[0112] Adopting the embodiments of the present disclosure, using the generated prompt information as a visual prompt and cross-transmitting it to another modality can make full use of the complementarity of different modalities. The cross-transmission process enhances the information interaction between modalities, enabling the system to maintain a high tracking performance under different environments and conditions.

[0113] Among them, in an optional embodiment, according to the fused feature sequences of each modality, the tracking target is framed in the search area image, including: determining the classification map of the fused feature sequence corresponding to each modality; performing weighted fusion on the fused feature sequences of each modality according to each classification map to obtain a weighted fused feature sequence; respectively determining the corresponding tracking results through the fused feature sequences corresponding to each modality and the weighted fused feature sequence; evaluating each tracking result through an evaluation index to obtain the corresponding evaluation result; determining the tracking result with the best evaluation result as the target tracking result; and framing the tracking target in the search area image through the target tracking result.

[0114] After obtaining the fused feature sequences corresponding to each modality, the weighted fused feature sequence obtained after performing weighted fusion on the fused feature sequences corresponding to each modality is the feature sequence of RGB-T. Specifically, it can be the classification result map obtained by classifying the fused feature sequences of the RGB and TIR modalities respectively.

[0115] Determine the classification map of the fused feature sequence corresponding to each modality. The classification map refers to the classification result map obtained by classifying the fused feature sequence during the classification task. Based on the classification maps of each modality, calculate the contribution degree of each modality to the final classification result; determine the weight of each modality according to the contribution degree. After obtaining the weights, perform weighted fusion on the fused feature sequences of each modality to obtain a weighted fused feature sequence.

[0116] During the weighted fusion process of the multi-modal fusion feature sequences, data of different modalities will be interfered by noise to varying degrees. Through the learned weights, the contribution degree of each modality in the fusion process can be automatically adjusted, thereby filtering out those modality data with larger noise and smaller contribution to the final result. Thus, the accuracy and robustness of the final classification and regression tasks are improved.

[0117] Design one or more stacked convolutional neural networks for the fusion feature sequences of each modality and the weighted fusion feature sequences. Through their respective convolutional neural networks, classify each of the fusion feature sequences and the weighted fusion feature sequences respectively to determine their respective corresponding tracking results.

[0118] Set appropriate evaluation metrics, which are used to evaluate the accuracy of the tracking results. Use the selected evaluation metrics to evaluate each of the tracking results to obtain the corresponding evaluation results. The evaluation results can be a numerical value, such as the value of a certain evaluation metric, or a ranking, such as the ranking of the evaluation metrics from the best to the worst.

[0119] According to the evaluation results corresponding to the fusion feature sequences and the weighted fusion feature sequences of each modality, select the tracking result with the best evaluation result as the target tracking result.

[0120] Use the target tracking result, such as the position and size of the tracking target, to draw a bounding box in the search area image to frame the tracking target.

[0121] Reference Figure 2 , the embodiments of the present disclosure can be executed by a bounding box determination module. In the case where the modalities include: visible light modality and thermal infrared modality, determine the first fusion feature sequence corresponding to the visible light modality and the second fusion feature sequence corresponding to the thermal infrared modality, input the first fusion feature sequence and the second fusion feature sequence into the bounding box determination module, and the bounding box determination module performs weighted fusion on the first fusion feature sequence and the second fusion feature sequence based on the weighted selection of the classification map to obtain a weighted fusion feature sequence, and respectively determine the corresponding tracking results through the first fusion feature sequence, the second fusion feature sequence, and the weighted fusion feature sequence; evaluate each of the tracking results through the evaluation metrics to obtain the corresponding evaluation results; determine the tracking result with the best evaluation result as the target tracking result; frame the tracking target in the tracking area image through the target tracking result.

[0122] With the embodiments of the present disclosure, through the weighted selection mechanism, the module can effectively fuse feature information from different modalities, and assign weights according to the confidence of the classification map, which can reduce the influence of noise when a modality fails, thereby enhancing the feature expression ability of the effective modality. In the case where a certain modality completely fails, the module can also adaptively select the best tracking result by evaluating the performance of each modality, which can ensure that the system can still maintain good tracking performance under adverse conditions.

[0123] Among them, in an alternative embodiment, after the tracking target is framed in the search area image, it further includes: determining the image to be updated through the framed area in the search area image; evaluating the image to be updated through position information, discriminant information, and appearance information to determine whether the image to be updated passes the evaluation; in the case where the image to be updated passes the evaluation, updating the dynamic template and determining the image to be updated as the updated dynamic template; in the case where the image to be updated fails to pass the evaluation, not updating the dynamic template.

[0124] Through the center point of the framed area in the search area image, re-frame the image to be updated in the search area image based on the center.

[0125] Judging whether to update the dynamic template needs to be determined based on the tracking information corresponding to the search area image of the current frame.

[0126] Extract tracking information from the search area image of the current frame, including position information, discriminant information, and appearance information.

[0127] The position information is the best predicted bounding box output by the tracking head of the tracker in the current frame, which can reflect the geometric shape of the predicted target in the current frame and the position information in the entire search area image. The position information of the tracking target in the current frame can be obtained by using the predicted bounding box output by the tracking head.

[0128] The classification map matrix output by the tracking head can be used as discriminant information. The classification map can well reflect the degree to which the tracker distinguishes the target from the surrounding background in the current frame. The recognition accuracy of the target can be evaluated according to the confidence value in the classification map. If the confidence is high, it means that the tracker has better recognized the target; if the confidence is low, there may be misjudgment.

[0129] However, relying solely on the output of the tracker as the condition for judging the update of the dynamic template will inevitably introduce noise due to misjudgment. This incorrect template update may cause the tracker to drift. Therefore, the appearance information of the template is also referred to as an important objective judgment condition. The appearance information of the current frame can be compared with the appearance information in the dynamic template to evaluate whether the appearance of the current target has changed significantly.

[0130] After obtaining the position information, discrimination information, and appearance information, evaluate whether to update the dynamic template. For example, reasonable thresholds can be set to determine whether the confidence of the classification map exceeds the threshold; evaluate the similarity between the appearance information of the current frame and the appearance information of the dynamic template using a similarity metric.

[0131] If the position information, discrimination information, and appearance information all meet the update conditions, perform the update operation of the dynamic template; otherwise, retain the original dynamic template.

[0132] Reference Figure 2 , it is possible to determine whether to update the dynamic template through the template update module.

[0133] By adopting the embodiments of the present disclosure, by integrating the tracking information of the current frame, it is possible to dynamically adapt to changes in the tracking target, thereby ensuring that the tracker can more accurately reflect the state of the tracking target at different time points, thereby improving the overall tracking accuracy. Relying on the comprehensive evaluation of position information, discrimination information, and appearance information, the risk of misjudgment caused by a single information source is reduced. By comparing and verifying the mutual consistency of different information, the noise introduced by misjudgment can be effectively reduced, and incorrect template updates can be avoided. Incorrect template updates may cause the tracker to drift, affecting subsequent tracking effects. By comprehensively considering various tracking information, this situation can be effectively avoided, ensuring that the tracker can still maintain accurate tracking when the target appearance changes.

[0134] Reference Figure 2 , in order to provide an overall explanation of the target tracking method provided by the present disclosure, the working process of the method is now introduced, including Step 1 to Step 6.

[0135] Step 1: Perform data processing and embedding

[0136] For the incoming entire RGB or TIR search area image, first segment and expand it into a sequence of multiple patches according to the set patch size, then project it into a high-dimensional space through a convolutional network, and finally add it to the learnable position embedding to obtain the embedded feature sequence.

[0137] Step 2: Perform temporal bidirectional fusion

[0138] The bidirectional spatio-temporal fusion module receives the feature sequence from the dynamic template or the static template, extracts the dimensionality-reduced information of the feature information corresponding to the feature sequence to obtain the refined cross-time prompt information, and then re-ascends the dimension to the same size as the feature information of the template image, and adds it to the feature information of the other time to complete the temporal information fusion.

[0139] Step 3: Perform spatial bidirectional fusion

[0140] The two-way spatio-temporal fusion module receives a sequence of search regions of a certain modality, extracts dimensionality-reduced information from this feature sequence, and after obtaining the refined cross-modal prompt information, re-ascends the dimension to the same size as the feature information corresponding to the search region image, and adds it to the feature information in the sequence of search regions of another modality to complete the inter-modal information fusion.

[0141] Step 4: Perform calculations on the Transformer backbone network

[0142] The target feature sequence obtained by fusing the temporal information of the dynamic template and the static template is concatenated with the search region sequence and fed into the Transformer network. After passing through the calculations of normalization, multi-head self-attention, normalization, and multi-layer perceptron respectively, the output of this modality alone at the current layer is obtained. After adding it to the cross-modal prompt information provided by the other modality, the complete output at the current layer, that is, the fused feature sequence, is obtained.

[0143] Step 5: Classification and regression

[0144] After obtaining the fused feature sequences corresponding to the two modalities respectively, classification and regression are performed on the fused feature sequence corresponding to the RGB modality, the fused feature sequence corresponding to the TIR modality, and the weighted fused feature sequence corresponding to RGB-T. The best result among the three tracking results is selected as the target tracking result.

[0145] Step 6: Template update

[0146] The template update module receives and integrates the tracking information of the current frame to determine whether the dynamic template should be updated. The tracking information includes position information, discriminant information, and appearance information.

[0147] Based on the same technical concept, the present disclosure provides an object tracking device. Figure 3 It is a block diagram of an object tracking device shown in an embodiment of the present disclosure. According to Figure 3 shown, the device includes:

[0148] An acquisition module 310, configured to acquire a sequence of search regions corresponding to search region images of each modality, and acquire a first target sequence corresponding to each dynamic template and a second target sequence corresponding to each static template. The static template represents the initial features of the tracking target, and the dynamic template represents the motion features of the tracking target;

[0149] A fusion module 320, configured to input the sequences of search regions, the first target sequences, and the second target sequences of each modality into an encoding layer with shared weights to perform temporal information fusion and inter-modal information fusion, and obtain a fused feature sequence for each modality;

[0150] A tracking module 330, configured to demarcate the tracked target in the search area image according to the fusion feature sequences of each modality.

[0151] Optionally, the fusion module is specifically configured to perform:

[0152] Perform temporal information fusion on the first target sequence and the second target sequence to obtain a target feature sequence, where the target feature sequence is used to capture the behavior of the tracked target;

[0153] Based on the target feature sequence, perform feature enhancement on the search area sequence to obtain an enhanced feature sequence, where the feature enhancement is used to distinguish background information and tracked target information in the search area sequence;

[0154] Perform inter-modal information interaction according to search area sequences of different modalities to obtain cross-modal prompt information;

[0155] Fuse the enhanced feature sequence and the cross-modal prompt information to obtain a fusion feature sequence.

[0156] Optionally, the fusion module is specifically configured to perform:

[0157] Perform dimensionality reduction information extraction on the second target sequence to obtain cross-time prompt information;

[0158] Ascend the dimension of the cross-time prompt information to the same dimension size as the feature information in the first target sequence, and superimpose it with the feature information in the first target sequence to obtain a first target sequence with superimposed cross-time prompt information;

[0159] Determine the first target sequence with the superimposed cross-time prompt information as the target feature sequence.

[0160] Optionally, the fusion module is specifically configured to perform:

[0161] Perform dimensionality reduction information extraction on the search area sequences of each modality to obtain first cross-modal prompt information;

[0162] Ascend the dimension of the first cross-modal prompt information to the same dimension size as the feature information in the search area sequence to obtain a first dimension-ascended result; the first dimension-ascended result is used to be superimposed with the enhanced feature sequence of another modality. [[ID=z8]]

[0163] Optionally, the fusion module is specifically configured to perform:

[0164] Receive a second dimension-ascended result obtained from the search area sequence of another modality;

[0165] Superimpose the second dimension-ascended result with the enhanced feature sequence to obtain the fusion feature sequence after cross-modal fusion.

[0166] Optionally, the apparatus further includes:

[0167] A determination module, configured to determine an image to be updated through a boxed area in the search area image;

[0168] An evaluation module, configured to evaluate the image to be updated through position information, discrimination information, and appearance information, and determine whether the image to be updated passes the evaluation;

[0169] An update module, configured to update the dynamic template when the image to be updated passes the evaluation, and determine the image to be updated as the updated dynamic template; and not update the dynamic template when the image to be updated fails to pass the evaluation.

[0170] Optionally, the tracking module is specifically configured to perform:

[0171] Determine a classification map of the fusion feature sequence corresponding to each modality;

[0172] According to each of the classification maps, perform weighted fusion on the fusion feature sequences of each modality to obtain a weighted fusion feature sequence;

[0173] Respectively determine corresponding tracking results through the fusion feature sequence corresponding to each modality and the weighted fusion feature sequence;

[0174] Evaluate each of the tracking results through an evaluation index to obtain a corresponding evaluation result;

[0175] Determine the tracking result with the best evaluation result as the target tracking result;

[0176] Box out a tracking target in the search area image through the target tracking result.

[0177] Optionally, the acquisition module is specifically configured to perform:

[0178] Segment and expand an image into a sequence composed of multiple patches according to a set patch size; the image is a search area image, a static template image, or a dynamic template image;

[0179] Project the sequence composed of the multiple patches into a high-dimensional space through a convolutional network to obtain a feature sequence;

[0180] Wherein, when the image is a search area image, after projecting the sequence composed of the multiple patches into a high-dimensional space, add the projection result to the position information embedding to obtain a feature sequence; the position information is the position of the patch in the search area image.

[0181] Embodiments of the present disclosure also provide an electronic device. Refer to Figure 4 , Figure 4 which is a schematic diagram of an electronic device shown in embodiments of the present disclosure. As Figure 4 shown, the electronic device 400 includes: a memory 410 and a processor 420. The memory 410 is communicatively connected to the processor 420 via a bus. A computer program is stored in the memory 410, and the computer program can run on the processor 420, thereby implementing the steps in the object tracking method disclosed in the embodiments of the present disclosure.

[0182] Embodiments of the present disclosure also provide a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the steps in the object tracking method disclosed in the embodiments of the present disclosure are implemented.

[0183] Embodiments of the present disclosure also provide a computer program product, including a computer program. When the computer program is executed by a processor, the steps in the object tracking method disclosed in the embodiments of the present disclosure are implemented.

[0184] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other.

[0185] Those skilled in the art should understand that the embodiments of the present disclosure can be provided as a method, an apparatus, or a computer program product. Therefore, the embodiments of the present disclosure can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0186] The embodiments of the present disclosure are described with reference to the flowcharts and / or block diagrams of methods, apparatuses, electronic devices, and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of processes and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing terminal devices generate a device for implementing the specified functions in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0187] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction device that implements the functions specified in one process or multiple processes and / or one block or multiple blocks in the process Figure 1 one process or multiple processes and / or Figure 1 one block or multiple blocks

[0188] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, such that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, and thus the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one process or multiple processes and / or one block or multiple blocks in the process Figure 1 one process or multiple processes and / or Figure 1 one block or multiple blocks

[0189] Although some embodiments of the embodiments of the present disclosure have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concepts. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present disclosure

[0190] The above has introduced in detail a target tracking method, system and device provided by the present disclosure. Specific examples are used herein to elaborate on the principles and implementation manners of the present disclosure. The description of the above embodiments is only used to help understand the method and its core idea of the present disclosure; at the same time, for those of ordinary skill in the art, according to the idea of the present disclosure, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present disclosure

Claims

1. A target tracking method, characterized in that, Including: Obtaining search area sequences corresponding to search area images of each modality, and obtaining a first target sequence corresponding to each dynamic template and a second target sequence corresponding to each static template, where the static template represents the initial features of the tracking target, and the dynamic template represents the motion features of the tracking target; Inputting the search area sequences, the first target sequence, and the second target sequence of each modality into an encoding layer with shared weights to perform temporal information fusion and cross-modal information fusion, and obtaining a fused feature sequence for each modality; Based on the fused feature sequences of each modality, bounding the tracking target in the search area image; The step of inputting the search area sequences, the first target sequence, and the second target sequence of each modality into an encoding layer with shared weights to perform temporal information fusion and cross-modal information fusion, and obtaining a fused feature sequence for each modality includes: Performing temporal information fusion on the first target sequence and the second target sequence to obtain a target feature sequence, where the target feature sequence is used to capture the behavior of the tracking target; Based on the target feature sequence, enhancing the features of the search area sequence to obtain an enhanced feature sequence, where the feature enhancement is used to distinguish background information and tracking target information in the search area sequence; Performing cross-modal information interaction according to the search area sequences of different modalities to obtain cross-modal prompt information; Fusing the enhanced feature sequence and the cross-modal prompt information to obtain a fused feature sequence for each modality.

2. The method according to claim 1, wherein Performing temporal information fusion on the first target sequence and the second target sequence to obtain a target feature sequence, including: Performing dimensionality reduction information extraction on the second target sequence to obtain cross-time prompt information; Ascending the dimensionality of the cross-time prompt information to the same dimension as the feature information in the first target sequence, and superimposing it with the feature information in the first target sequence to obtain a first target sequence with superimposed cross-time prompt information; Determining the first target sequence with superimposed cross-time prompt information as the target feature sequence.

3. The method according to claim 1, characterized in that, Performing cross-modal information interaction according to the search area sequences of different modalities to obtain cross-modal prompt information, including: Performing dimensionality reduction information extraction on the search area sequences of each modality to obtain first cross-modal prompt information; Ascending the dimensionality of the first cross-modal prompt information to the same dimension as the feature information in the search area sequence to obtain a first dimensionality-ascended result; the first dimensionality-ascended result is used to be superimposed with the enhanced feature sequence of another modality.

4. The method according to claim 3, wherein After enhancing the features of the search area sequence based on the target feature sequence to obtain an enhanced feature sequence, it further includes: Receiving a second dimensionality-ascended result obtained from the search area sequence of another modality; Superimposing the second dimensionality-ascended result with the enhanced feature sequence to obtain the fused feature sequence after cross-modal fusion.

5. The method according to claim 1, characterized in that, After bounding the tracking target in the search area image, it further includes: Determining an image to be updated through the bounding area in the search area image; Evaluating the image to be updated through position information, discriminant information, and appearance information, and determining whether the image to be updated passes the evaluation; When the evaluation of the image to be updated passes, update the dynamic template and determine the image to be updated as the updated dynamic template; When the evaluation of the image to be updated fails, do not update the dynamic template.

6. The method according to claim 1, characterized in that According to the fusion feature sequences of each modality, the tracking target is framed in the search area image, including: Determine the classification map of the fusion feature sequence corresponding to each modality; According to each of the classification maps, perform weighted fusion on the fusion feature sequences of each modality to obtain a weighted fusion feature sequence; Respectively determine the corresponding tracking results through the fusion feature sequences corresponding to each modality and the weighted fusion feature sequence; Evaluate each tracking result through an evaluation index to obtain the corresponding evaluation result; Determine the tracking result with the best evaluation result as the target tracking result; Frame the tracking target in the search area image through the target tracking result.

7. The method according to any one of claims 1-6, characterized in that, The search area sequence, the first target sequence, and the second target sequence are determined by the following methods, including: According to the set patch size, segment and expand the image into a sequence composed of multiple patches; the image is a search area image, a static template image, or a dynamic template image; Project the sequence composed of the multiple patches into a high-dimensional space through a convolutional network to obtain a feature sequence; Among them, when the image is a search area image, after projecting the sequence composed of the multiple patches into a high-dimensional space, add the projection result to the position information embedding to obtain a feature sequence; the position information is the position of the patch in the search area image.

8. A target tracking device, characterized in that, Including: An acquisition module for acquiring the search area sequence corresponding to the search area image of each modality, and acquiring the first target sequence corresponding to each dynamic template and the second target sequence corresponding to each static template, where the static template represents the initial features of the tracking target, and the dynamic template represents the motion features of the tracking target; A fusion module for inputting the search area sequences, the first target sequences, and the second target sequences of each modality into an encoding layer with shared weights for temporal information fusion and inter-modal information fusion to obtain the fusion feature sequence of each modality; including: performing temporal information fusion on the first target sequence and the second target sequence to obtain a target feature sequence, where the target feature sequence is used to capture the behavior of the tracking target; based on the target feature sequence, perform feature enhancement on the search area sequence to obtain an enhanced feature sequence, where the feature enhancement is used to distinguish the background information and the tracking target information in the search area sequence; perform inter-modal information interaction according to the search area sequences of different modalities to obtain cross-modal prompt information; fuse the enhanced feature sequence and the cross-modal prompt information to obtain the fusion feature sequence of each modality; A tracking module for framing the tracking target in the search area image according to the fusion feature sequences of each modality.

9. An electronic device, characterized in that, Including: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the target tracking method according to any one of claims 1-7 are implemented.

Images