Visible light and infrared image fusion target tracking method based on visual prompt learning

Abstract

The invention discloses a visible light and infrared image fusion target tracking method based on visual cue learning, and relates to the field of extended target tracking. Comprising the steps of inputting video image data, converting an image through a block embedding module, extracting multi-scale information through scale perception attention, extracting each modal feature through a double-branch backbone network, performing specific modal visual prompt learning and inter-modal visual prompt interaction, and performing cross fusion of an adapter and a prediction head. According to the method, a more reliable and efficient target tracking system is realized by introducing a multi-scale information extraction mechanism, enhancing visual prompt learning and interaction capability and developing a more advanced multi-modal fusion strategy, and through the improvements, the performance of target tracking in a complex environment can be remarkably improved, and the requirements of practical application are met.

Description

Technical Field

[0001] The present invention mainly relates to the technical field of extended target tracking, and specifically to a visible light and infrared image fusion target tracking method based on visual cue learning. Background Art

[0002] With the rapid development of computer vision technology, target tracking, as an important research field, has shown great potential and value in many practical applications such as autonomous driving, intelligent security, and human-computer interaction. The core task of target tracking is to accurately and stably locate and track a given target in a continuous video sequence. However, due to changes in environmental factors and imaging conditions, traditional target tracking methods based on visible light images perform poorly in some complex scenarios.

[0003] Most traditional target tracking methods rely on visible light images. Such methods usually identify and locate targets in video sequences by means of feature extraction and matching. However, trackers based on visible light are highly dependent on ambient light. They perform well under good lighting conditions, but the information quality of images will be significantly reduced under extreme lighting conditions (such as direct strong light or low illumination at night) and bad weather conditions (such as fog, rain, and snow), which directly affects the performance of the tracker. For example, at night or in dark environments, due to insufficient light, the images captured by visible light cameras are often blurred, making it difficult to accurately identify and track targets. In this case, the tracker is prone to target loss or mistracking, resulting in reduced system reliability.

[0004] In order to overcome the limitations of visible light tracking, researchers began to try to use infrared imaging technology to assist target tracking. Infrared images capture the thermal radiation information of objects and are not affected by changes in ambient light, especially in low light and night conditions. Since infrared imaging technology can provide additional information, such as the location of the human body or other heat-generating objects, it performs well in heat source monitoring. However, relying solely on infrared images also has certain limitations. For example, when the temperature difference is not large, it is difficult for infrared images to distinguish between background and foreground targets, especially when there is a "heat cross" phenomenon, the mutual interference between different temperature areas will lead to a decrease in target tracking performance.

[0005] Given the inherent deficiencies of single-modality images in target tracking, multimodal fusion technology has gradually become a key means to improve the robustness and accuracy of target tracking. By combining the advantages of visible light and infrared images, researchers can capture target information more comprehensively in complex environments, thereby improving the overall performance of the tracking system. For example, during the day, visible light images can provide high-resolution texture details, while at night or in bad weather, infrared images can make up for the shortcomings of visible light imaging. Through the complementary fusion of the two, it is possible to better cope with complex and changing tracking environments.

[0006] Although multimodal fusion technology has shown great potential, existing target tracking methods based on visible light (RGB) and infrared (Thermal, T) still face many challenges in practical applications. Existing methods can be roughly divided into two categories: traditional methods and deep learning-based methods. Traditional methods rely on manually designed features, such as HOG, SIFT, etc., and perform similarity calculations through classical techniques such as correlation filters. Although these methods have certain effects in simple scenes, they are easily affected by noise when facing complex scenes, and have poor adaptability to target deformation and partial occlusion.

[0007] In recent years, with the rapid development of convolutional neural networks (CNNs) and Visual Transformer (ViT), RGBT object tracking methods based on deep learning have become a hot research area. Some current studies attempt to use pre-trained unimodal models, especially models trained on large-scale RGB datasets, to adapt them to multimodal tracking tasks by fine-tuning. This method has significant advantages in alleviating the lack of labeled data, while also making full use of the powerful ability of unimodal models in feature extraction to further improve the performance of multimodal tracking. In addition, inspired by the concept of prompt learning in the field of natural language processing, some studies introduce visual prompts into the object tracking task. By adding specific visual prompts to the input, the model can better focus on the task-related feature information, thereby enhancing its performance in downstream tasks. This prompt mechanism not only helps the basic model adapt to different tracking scenarios, but also provides a new idea for solving the problem of multimodal fusion in complex environments.

[0008] Although the above methods have made some progress, they still have the following shortcomings:

[0009] Insufficient multi-scale information extraction capabilities: Most ViT-based trackers fail to fully consider the multi-scale information of the target, especially in scenes where the target size varies greatly or the shape varies, and show low robustness. For example, when the target gradually approaches the camera from a distance, the target size changes significantly, and existing methods often cannot effectively cope with such changes, resulting in tracking failure.

[0010] Imperfect visual cue mechanism: Although some studies have introduced visual cue learning mechanisms to enhance the adaptability of the model, these methods usually only focus on the cue adjustment at the input end, ignoring the dynamic learning and enhancement of visual cues during feature extraction, and lack effective cross-modal cue interaction mechanisms. This means that when the model processes complex modal information, it cannot fully utilize the role of multimodal visual cues, limiting the improvement of tracking performance.

[0011] Simplification of multimodal fusion strategies: Currently, most fusion strategies, such as simple concatenation and weighted sum cross attention mechanisms, do not fully consider the differences in feature distribution between visible light and infrared images. This simple fusion method has poor adaptability to each modality when dealing with scenes with significant differences between modalities, often showing low adaptability and efficiency, and failing to fully utilize the advantages of each modality. Summary of the invention

[0012] Based on this, the purpose of the present invention is to provide a visible light and infrared image fusion target tracking method based on visual cue learning to solve the technical problems raised in the above background technology.

[0013] To achieve the above object, the present invention provides the following technical solutions:

[0014] The method for target tracking based on visual cue learning by fusing visible light and infrared images includes the following steps:

[0015] S1. Input video image data:

[0016] The system first receives video images as input data, including visible light template images, infrared template images, visible light search images, and infrared search images. These images will serve as the basis for subsequent processing, where the size of the search area image is adjusted to 256*256, and the size of the template image is adjusted to 128*128;

[0017] S2, transform the image through the block embedding module:

[0018] The input visible light and infrared image data are first input into the Patch Embedding module respectively and converted into a series of Tokens. This process not only divides the image into multiple small blocks, but also maps each block to a high-dimensional feature space, thus preparing for subsequent feature extraction.

[0019] S3, scale-aware attention extracts multi-scale information:

[0020] Next, the tokens obtained from different images are processed by the scale-aware attention module. This module can capture multi-scale information and enhance the detection ability of objects of different sizes and shapes by introducing dilated convolutions with different dilation rates. This step ensures that the model can maintain high-precision tracking even when the target size varies greatly:

[0021] Specifically, we take the scale-aware attention module in the visible light image branch as an example and assume that the input feature map is F rgb ; First, the feature map F rgbAfter linear layer mapping, query vector Q, key vector K and value vector V are generated; then, three dilated convolutions with different dilation rates (r=1, 2, 3) are designed to process K and V to obtain key K with different receptive fields. r and V r , the process is expressed as follows:

[0022] Q,K,V=Linear(F rgb ) (1)

[0023]

[0024] Among them, Linear represents the linear layer, DiConv represents the dilated convolution, and r represents the dilation rate;

[0025] Then, the multi-scale features F under different receptive fields are calculated through the self-attention mechanism Att,r ; means as follows:

[0026]

[0027] Among them, Softmax represents the Softmax function, T represents the transposition operation, and d is the dimension of K;

[0028] Next, in order to make full use of information at different scales, the enhanced features F at each expansion rate are Att,r Concatenated and mapped into a single attention-enhanced feature F through a linear layer Att ; Then, this feature is combined with the original input feature F rgb Add them element by element and normalize them through layer normalization; then, use a feedforward neural network to perform nonlinear transformation on the standardized features, and perform addition and normalization operations again to finally generate the output feature F rgb,out ; The whole process is formalized as:

[0029] F Att =Linear(cat[F Att,1 ,F Att,2 ,F Att,3 ]) (4)

[0030] F rgb =LN(F rgb +F Att )(5)

[0031] F rgb,out =LN(F rgb +FFN(F rgb )) (6)

[0032] Among them, cat represents concatenation operation, LN represents normalization, and FFN represents feedforward neural network;

[0033] S4, dual-branch backbone network extracts features of each modality:

[0034] The features of each modality are input into a dual-branch backbone network composed of Transformer encoders to extract visible light image features and infrared image features respectively. The input of each layer of encoder includes the output of the previous layer of encoder and the output of the multimodal prompt interaction learning module at the current stage. This dual-branch structure allows the model to process the information of the two modalities independently and perform effective fusion in subsequent steps.

[0035] S5. Modality-specific visual cue learning and inter-modality visual cue interaction:

[0036] Between each layer of Transformer encoder, a multimodal cue interaction learning module consisting of global token adaptive attention and spatial attention is embedded. The main function of this module is to learn visual cues of a specific modality and complete cue interaction between modalities. Specifically, the input of this module includes the output of the previous layer of Transformer encoders in two branches and the output of the previous layer of multimodal cue interaction learning module. In this way, the model can dynamically adjust the attention weights during feature extraction and improve its adaptability to complex scenes.

[0037] S6, Cross-fusion adapter:

[0038] After feature extraction by the dual-branch backbone network, the visible light image features and infrared image features are obtained respectively; then, the features of these two modalities are input into the cross-fusion adapter for fusion; this module introduces the adapter mechanism to adaptively adjust the adaptability of different modal features in the existing cross-attention, thus achieving more flexible and efficient multimodal information integration; this can make full use of the advantages of the two modalities and overcome the shortcomings of a single modality;

[0039] The following is a detailed description of the working principle of the CFA module; taking a certain CFA instance as an example, assuming that the feature output of the RGB branch is X q , the feature output by the TIR branch is X kv ; First, we use the linear layer to transform X q and X kv Mapping is performed to obtain the query vector q1, key vector k2 and value vector v2, as shown in formula (15):

[0040]

[0041] To further improve the adaptability of the model to modal characteristics, we use the query vector q 1An adapter is introduced on the vector; the adapter consists of a 1x1 dimensionality reduction convolution layer, a GELU activation function, and a 1x1 dimensionality increase convolution layer. The generated adjustment feature is recorded as adpt1; then, q1 is processed with adpt1, k2, and v2 through a multi-head cross attention mechanism to generate the feature F out1 , as shown in formulas (16) and (17):

[0042] Among them, downconv represents dimensionality reduction convolution, gelu represents GELU activation function, and upconv represents dimensionality increase convolution;

[0043] adpt1=upconv(gelu(downconv(q1))) (16)

[0044]

[0045] Next, we will F out1 With the original feature X q Add and normalize through layer normalization to generate the updated feature F out1 ; Then, we again out1 Apply the same adapter structure to generate feature adpt2, as shown in formulas (18) and (19):

[0046] F out1 =LN(F out1 +X q ) (18)

[0047] Finally, F out1 , the features processed by the feedforward neural network are added to adpt2, and the final output features are obtained through layer normalization, as shown in formula (20):

[0048] adpt2=up(gelu(down(F out1 ))) (19)

[0049] F out =LN(F out1 +FFN(F out1 )+adpt2) (20)

[0050] S7, prediction head:

[0051] Finally, the generated multimodal fusion features are input into the prediction head for classification and regression; the prediction head is responsible for mapping the fused features back to the original image coordinate system and predicting the precise position of the target in the image, that is, the bounding box containing the target. Through this process, the system can accurately locate and track the target and provide stable target tracking results.

[0052] In summary, the present invention mainly has the following beneficial effects:

[0053] This application introduces a multi-scale information extraction mechanism, enhances visual cue learning and interaction capabilities, and develops a more advanced multimodal fusion strategy to achieve a more reliable and efficient target tracking system. Through these improvements, we expect to significantly improve the performance of target tracking in complex environments and meet the needs of practical applications;

[0054] Enhanced ability to capture multi-scale information: By introducing scale-aware attention (SAA), the model of the present invention can more effectively capture target features of different scales and shapes. This mechanism uses different expansion rates to perceive the target area at multiple scales, ensuring that the model can still provide stable tracking effects even when the target changes in shape and size, significantly improving the robustness and accuracy in real complex scenes.

[0055] Enhanced learning and interaction of multimodal visual cues: This paper embeds a multimodal prompt interaction learning module (MPIL) between each layer of encoders in the dual-branch visual Transformer (ViT) backbone network to achieve prompt learning and information interaction for visible light and infrared images; this module combines global token adaptive attention and spatial attention mechanisms, which not only promotes prompt interaction between different modalities, but also further strengthens the exclusive prompt learning for each modality, so that the model can make better use of multimodal prompt information in complex and changing environments, thereby improving the robustness of the tracking system;

[0056] Improve the adaptability of the fusion module to the features of each modality: In view of the problem that the existing fusion methods are not adaptable enough to the modal features, the present invention proposes a new cross-fusion adapter (CFA). This module introduces an adaptive adjustment mechanism, which can dynamically adjust the weight distribution in the feature fusion process according to the characteristics of different modalities, ensuring that the information advantages of each modality can be fully utilized in the fusion process; this not only enhances the integration effect of multimodal information, but also makes the tracker more adaptable and robust in complex environments.

[0057] Through the above innovations, the present invention aims to develop a multimodal target tracking system that can perform well under a variety of environmental conditions (such as extreme lighting, severe weather, complex occlusions, etc.), ensuring the high reliability and accuracy of the tracking task, thereby providing a better solution for practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a flow chart of the present invention;

[0059] Figure 2 It is the overall framework diagram of the multi-modal target tracking method of the present invention;

[0060] Figure 3 This is a diagram of the multimodal prompt interactive learning module of the present invention. DETAILED DESCRIPTION

[0061] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0062] The following describes an embodiment of the present invention based on its overall structure.

[0063] Example

[0064] Please refer to the attached Figure 1-3 As shown, the visible light and infrared image fusion target tracking method based on visual cue learning includes the following steps:

[0065] S1. Input video image data

[0066] The system first receives video images as input data, including visible light template images, infrared template images, visible light search images and infrared search images. These images will serve as the basis for subsequent processing, where the size of the search area image is adjusted to 256*256 and the size of the template image is adjusted to 128*128.

[0067] S2. Transform the image through the block embedding module

[0068] The input visible light and infrared image data are first input into the Patch Embedding module respectively and converted into a series of Tokens. This process not only divides the image into multiple small blocks, but also maps each block to a high-dimensional feature space, thus preparing for subsequent feature extraction.

[0069] By embedding the Multi-modal Prompts Interaction Learning module, the model's ability to learn specific modal features is enhanced, and prompt interaction between modalities is realized; this module combines global token adaptive attention and spatial attention, enabling the model to dynamically adjust attention weights during feature extraction, thereby better understanding target features;

[0070] This module not only improves the feature representation ability of a single modality, but also promotes information exchange between different modalities, improving the robustness and accuracy of the system; compared with the traditional cross-attention mechanism, it improves the adaptive adjustment ability of specific modality features.

[0071] S3. Scale-aware attention extracts multi-scale information

[0072] Next, the tokens obtained from different images are processed by the scale-aware attention module. This module can capture multi-scale information and enhance the detection ability of objects of different sizes and shapes by introducing dilated convolutions with different dilation rates. This step ensures that the model can maintain high-precision tracking even when the target size varies greatly:

[0073] Specifically, we take the scale-aware attention module in the visible light image branch as an example and assume that the input feature map is F rgb ; First, the feature map F rgb After linear layer mapping, query vector Q, key vector K and value vector V are generated; then, three dilated convolutions with different dilation rates (r=1, 2, 3) are designed to process K and V to obtain key K with different receptive fields. r and V r , the process is expressed as follows:

[0074] Q,K,V=Linear(F rgb ) (1)

[0075]

[0076] Among them, Linear represents the linear layer, DiConv represents the dilated convolution, and r represents the dilation rate;

[0077] Then, the multi-scale features F under different receptive fields are calculated through the self-attention mechanism Att,r ; means as follows:

[0078]

[0079] Among them, Softmax represents the Softmax function, T represents the transposition operation, and d is the dimension of K;

[0080] Next, in order to make full use of information at different scales, the enhanced features F at each expansion rate are Att,r Concatenated and mapped into a single attention-enhanced feature F through a linear layer Att ; Then, this feature is combined with the original input feature F rgbAdd them element by element and normalize them through layer normalization; then, use a feedforward neural network to perform nonlinear transformation on the standardized features, and perform addition and normalization operations again to finally generate the output feature F rgb,out ; The whole process is formalized as:

[0081] F Att =Linear(cat[F Att,1 ,F Att,2 ,F Att,3 ]) (4)

[0082] F rgb =LN(F rgb +F Att )(5)

[0083] F rgb,out =LN(F rgb +FFN(F rgb )) (6)

[0084] Among them, cat represents concatenation operation, LN represents normalization, and FFN represents feedforward neural network.

[0085] The ability to extract objects of different sizes and shapes is enhanced by introducing the scale-aware attention (SAA) module. The SAA module uses dilated convolutions with different dilation rates to dynamically adjust the receptive field, thereby effectively capturing multi-scale information in the multi-head self-attention mechanism. This enables the model to maintain high-precision tracking even when the target size varies greatly or there is a complex background.

[0086] Compared with traditional convolutional neural networks, the SAA module can more flexibly adapt to different receptive field requirements, thereby better handling small-scale and large-scale targets, ensuring high detection accuracy under various conditions.

[0087] S4, dual-branch backbone network extracts features of each modality

[0088] The features of each modality are input into a dual-branch backbone network composed of Transformer encoders to extract visible light image features and infrared image features respectively. The input of each layer of encoder includes the output of the previous layer of encoder and the output of the multimodal prompt interaction learning module in the current stage. This dual-branch structure allows the model to process the information of the two modalities independently and effectively fuse them in subsequent steps.

[0089] S5. Modality-specific visual cue learning and inter-modality visual cue interaction

[0090] Between each layer of Transformer encoder, a multimodal cue interaction learning module consisting of global token adaptive attention and spatial attention is embedded. The main function of this module is to learn visual cues of a specific modality and complete cue interaction between modalities. Specifically, the input of this module includes the output of the previous layer of Transformer encoders in two branches and the output of the previous layer of multimodal cue interaction learning module. In this way, the model can dynamically adjust the attention weights during feature extraction and improve its adaptability to complex scenes.

[0091] like Figure 3 As shown:

[0092] Taking the multimodal prompt interactive learning module of the lth layer as an example, assuming that the two modal features of the input are M1 and M2, and the prompt of the MPIL module in the previous layer is represented as P l-1 ; First, three parallel global token adaptive attention (GTAA) modules are used to process the input features (Formula 7), and then the spatial attention (SA) module and dimensionality reduction convolution are used for further processing (Formula 8); Then, we perform fovea operation on M1, add it with M2 and Pl-1, and then perform dimensionality increase convolution as the prompt of the l-th layer mode, that is, P l The specific structure and implementation process of GTAA and SA modules will be described in detail in the following sections:

[0093]

[0094] P l =upconv{fovea(M1)+M2+P l-1} (9)

[0095] For the GTAA module; assuming the input feature is x, first perform average pooling and maximum pooling operations on it, and add the two to generate feature x a ; To capture global information, we will a Multiply it by its transpose to get the feature x a1 , while x a Perform one-dimensional convolution to obtain feature x a2 ; Then the feature x a1 and x a2 Add, after one-dimensional convolution and Sigmoid function processing, generate the output of GTAA; the whole process can be described by the following formula:

[0096] x a =maxpool(x)+avgpool(x) (10)

[0097]

[0098] GTAA(x)=Sigmoid(conv1d(x) a1 +x a2 )) (12)

[0099] Among them, maxpool represents the maximum pooling operation, avgpool represents the average pooling operation, T is the transpose operation, and conv1d represents the one-dimensional convolution;

[0100] For the SA module; assuming the input feature is y, first perform maximum pooling and average pooling operations on the feature, and concatenate the results to generate feature y ma ; Then, a convolution kernel of size 7*7 is used for processing, and the weight is adjusted by the Sigmoid function to obtain the spatial attention corresponding to the feature, which is multiplied by the original input feature to obtain the feature of spatial attention enhancement;

[0101] y ma =cat[maxpool(y),avgpool(y)] (13)

[0102] SA(y)=Sigmoid(conv(y ma ))(14)

[0103] A multimodal cue interaction learning module consisting of global token adaptive attention and spatial attention is embedded to learn visual cues of a specific modality and realize cue interaction between modalities.

[0104] The multimodal cue interaction learning module not only enhances the model’s ability to learn cues from a specific modality, but also facilitates the exchange of visual cues between different modalities; this improves the robustness and accuracy of the overall system, especially in target tracking tasks in complex environments; specifically, the module is able to dynamically adjust attention weights during feature extraction, allowing the model to better understand the characteristics of the target;

[0105] The embedded multimodal prompt interaction learning module enhances the model's ability to learn specific modal features and realizes prompt interaction between modalities. This module combines global token adaptive attention and spatial attention, enabling the model to dynamically adjust attention weights during feature extraction, thereby better understanding target features.

[0106] This module not only improves the feature representation ability of a single modality, but also promotes information exchange between different modalities, improves the robustness and accuracy of the system, and improves the adaptive adjustment ability of specific modality features compared to the traditional cross-attention mechanism.

[0107] S6, Cross-fusion adapter

[0108] After feature extraction by the dual-branch backbone network, the visible light image features and infrared image features are obtained respectively; then, the features of these two modalities are input into the cross-fusion adapter for fusion; this module introduces the adapter mechanism to adaptively adjust the adaptability of different modal features in the existing cross-attention, thus achieving more flexible and efficient multimodal information integration; this can make full use of the advantages of the two modalities and overcome the shortcomings of a single modality;

[0109] The following is a detailed description of the working principle of the CFA module; taking a certain CFA instance as an example, assuming that the feature output of the RGB branch is X q , the feature output by the TIR branch is X kv ; First, we use the linear layer to transform X q and X kv Mapping is performed to obtain the query vector q1, key vector k2 and value vector v2, as shown in formula (15):

[0110]

[0111] To further improve the adaptability of the model to modal characteristics, we use the query vector q 1 An adapter is introduced on the vector; the adapter consists of a 1x1 dimensionality reduction convolution layer, a GELU activation function, and a 1x1 dimensionality increase convolution layer. The generated adjustment feature is recorded as adpt1; then, q1 is processed with adpt1, k2, and v2 through a multi-head cross attention mechanism to generate the feature F out1 , as shown in formulas (16) and (17):

[0112] Among them, downconv represents dimensionality reduction convolution, gelu represents GELU activation function, and upconv represents dimensionality increase convolution;

[0113] adpt1=upconv(gelu(downconv(q1))) (16)

[0114]

[0115] Next, we will F out1 With the original feature X q Add and normalize through layer normalization to generate the updated feature F out1 ; Then, we again out1 Apply the same adapter structure to generate feature adpt2, as shown in formulas (18) and (19):

[0116] F out1 =LN(F out1 +X q) (18)

[0117] Finally, F out1 , the features processed by the feedforward neural network are added to adpt2, and the final output features are obtained through layer normalization, as shown in formula (20):

[0118] adpt2=up(gelu(down(F out1 ))) (19)

[0119] F out =LN(F out1 +FFN(F out1 )+adpt2) (20)

[0120] An adapter component is introduced into the existing cross-attention fusion mechanism to improve the network's adaptability to different modal data streams;

[0121] This module can adaptively adjust the importance of different modal features according to the characteristics of the input data, realizing flexible and efficient multimodal information integration; this fusion strategy overcomes the limitations of existing simple fusion methods and ensures the best tracking performance in various environments; specifically, this module can make full use of the advantages of visible light and infrared images to provide more accurate and stable target tracking results.

[0122] S7, prediction head

[0123] Finally, the generated multimodal fusion features are input into the prediction head for classification and regression; the prediction head is responsible for mapping the fused features back to the original image coordinate system and predicting the precise position of the target in the image, that is, the bounding box containing the target. Through this process, the system can accurately locate and track the target and provide stable target tracking results.

[0124] The present application discloses a target tracking method based on the fusion of visible light and infrared images based on visual cue learning, including a complete process from inputting video images to finally predicting the target position. The method covers the entire process from image preprocessing, feature extraction, multimodal fusion to target positioning, forming a complete end-to-end target tracking system. The entire method constructs a clearly structured and powerful target tracking framework through key technologies such as block embedding module, dual-branch backbone network, multimodal cue interactive learning module and cross-fusion adapter, which enables the system to achieve high-precision and robust target tracking in complex environments.

[0125] Although an embodiment of the present invention has been shown and described, this specific embodiment is merely an explanation of the present invention and is not a limitation of the invention. The specific features, structures, materials or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. After reading this specification, those skilled in the art may make modifications, substitutions and variations to the embodiments without creative contributions as needed without departing from the principles and purpose of the present invention. However, as long as they are within the scope of the claims of the present invention, they are protected by patent law.

Claims

1. A target tracking method based on visual cue learning by fusing visible light and infrared images, characterized in that: The following steps are involved: S1. Input video image data: The system first receives video images as input data, including visible light template images, infrared template images, visible light search images, and infrared search images. These images will serve as the basis for subsequent processing, where the size of the search area image is adjusted to 256*256, and the size of the template image is adjusted to 128*128; S2, transform the image through the block embedding module: The input visible light and infrared image data are first input into the Patch Embedding module respectively and converted into a series of Tokens. This process not only divides the image into multiple small blocks, but also maps each block to a high-dimensional feature space, thus preparing for subsequent feature extraction. S3, scale-aware attention extracts multi-scale information: Next, the tokens obtained from different images are processed by the scale-aware attention module. This module can capture multi-scale information and enhance the detection ability of objects of different sizes and shapes by introducing dilated convolutions with different dilation rates. This step ensures that the model can maintain high-precision tracking even when the target size varies greatly: Specifically, we take the scale-aware attention module in the visible light image branch as an example and assume that the input feature map is F rgb ; First, the feature map F rgb After linear layer mapping, query vector Q, key vector K and value vector V are generated; then, three dilated convolutions with different dilation rates (r=1, 2, 3) are designed to process K and V to obtain key K with different receptive fields. r and V r , the process is expressed as follows: Q,K,V=Linear(F rgb ) (1) Among them, Linear represents the linear layer, DiConv represents the dilated convolution, and r represents the dilation rate; Then, the multi-scale features F under different receptive fields are calculated through the self-attention mechanism Att,r ; means as follows: Among them, Softmax represents the Softmax function, T represents the transposition operation, and d is the dimension of K; Next, in order to make full use of information at different scales, the enhanced features F at each expansion rate are Att,r Concatenated and mapped into a single attention-enhanced feature F through a linear layer Att ; Then, this feature is combined with the original input feature F rgb Add them element by element and normalize them through layer normalization; then, use the feedforward neural network to perform nonlinear transformation on the standardized features, and perform addition and normalization operations again to finally generate the output feature F rgb,out ; The whole process is formalized as: F Att =Linear(cat[F Att,1 ,F Att,2 ,F Att,3 ]) (4) F rgb =LN(F rgb +F Att )(5) F rgb,out =LN(F rgb +FFN(F rgb )) (6) Among them, cat represents concatenation operation, LN represents normalization, and FFN represents feedforward neural network; S4, dual-branch backbone network extracts features of each modality: The features of each modality are input into a dual-branch backbone network composed of Transformer encoders to extract visible light image features and infrared image features respectively. The input of each layer of encoder includes the output of the previous layer of encoder and the output of the multimodal prompt interaction learning module at the current stage. This dual-branch structure allows the model to process the information of the two modalities independently and perform effective fusion in subsequent steps. S5. Modality-specific visual cue learning and inter-modality visual cue interaction: Between each layer of Transformer encoder, a multimodal cue interaction learning module consisting of global token adaptive attention and spatial attention is embedded. The main function of this module is to learn visual cues of a specific modality and complete cue interaction between modalities. Specifically, the input of this module includes the output of the previous layer of Transformer encoders in two branches and the output of the previous layer of multimodal cue interaction learning module. In this way, the model can dynamically adjust the attention weights during feature extraction and improve its adaptability to complex scenes. S6, Cross-fusion adapter: After feature extraction by the dual-branch backbone network, the visible light image features and infrared image features are obtained respectively; then, the features of these two modalities are input into the cross-fusion adapter for fusion; this module introduces the adapter mechanism to adaptively adjust the adaptability of different modal features in the existing cross-attention, thus achieving more flexible and efficient multimodal information integration; this can make full use of the advantages of the two modalities and overcome the shortcomings of a single modality; The following is a detailed description of the working principle of the CFA module; taking a certain CFA instance as an example, assuming that the feature output of the RGB branch is X q , the feature output by the TIR branch is X kv ; First, we use the linear layer to transform X q and X kv Mapping is performed to obtain the query vector q1, key vector k2 and value vector v2, as shown in formula (15): To further improve the adaptability of the model to modal characteristics, we use the query vector q 1 An adapter is introduced on the vector; the adapter consists of a 1x1 dimensionality reduction convolution layer, a GELU activation function, and a 1x1 dimensionality increase convolution layer. The generated adjustment feature is recorded as adpt1; then, q1 is processed with adpt1, k2, and v2 through a multi-head cross attention mechanism to generate the feature F out1 , as shown in formulas (16) and (17): Among them, downconv represents dimensionality reduction convolution, gelu represents GELU activation function, and upconv represents dimensionality increase convolution; adpt1=upconv(gelu(downconv(q1))) (16) Next, we will F out1 With the original feature X q Add and normalize through layer normalization to generate the updated feature F out1 ; Then, we again out1 Apply the same adapter structure to generate feature adpt2, as shown in formulas (18) and (19): F out1 =LN(F out1 +X q ) (18) Finally, F out1 , the features processed by the feedforward neural network are added to adpt2, and the final output features are obtained through layer normalization, as shown in formula (20): adpt2=up(gelu(down(F out1 ))) (19) F out =LN(F out1 +FFN(F out1 )+adpt2) (20) S7, prediction head: Finally, the generated multimodal fusion features are input into the prediction head for classification and regression; the prediction head is responsible for mapping the fused features back to the original image coordinate system and predicting the precise position of the target in the image, that is, the bounding box containing the target. Through this process, the system can accurately locate and track the target and provide stable target tracking results.

Images