Infrared tracking method based on feature local correction and multi-modal channel sparse selection cues

Abstract

The application discloses an infrared tracking method based on feature local correction and multimodal channel sparse selection prompt, comprising the following steps: step (1): a multimodal video is intercepted into multiple frames of still images in chronological order to obtain a data set, and different operations are performed for a training stage and a tracking stage; step (2): the data set is used to train fine-tuning network weight parameters to obtain algorithm network weights, and the network weight parameters are used for reasoning when network weights are loaded in the tracking stage; step (3): the network weight parameters obtained in the training stage are loaded, and then the position of a target in a first frame of a video is labeled, the target is continuously tracked in future frames, and position information of the target is output. The application considers the internal connection and complementarity of different modal features from two angles of a semantic space through a prompt module, and the effectiveness of prompt information is improved. Through a feature correction module, the feature deviation problem caused by spatial misalignment is effectively solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of target tracking, and particularly relates to an infrared tracking method based on feature local correction and multi-modal channel sparse selection prompt. BACKGROUND

[0002] The method paradigm based on prompt fine-tuning has become a hot spot for current researchers. In the field of vision, some researches add learnable visual prompts to the frozen base model, and research shows that visual prompt learning is expected to become an alternative method to complete fine-tuning. Some recent works add modal prompts to the frozen visible light tracker for visible-infrared multi-modal tracking, and achieve performance comparable to complete fine-tuning.

[0003] Although modal prompts improve the representation ability of visible light pre-trained large models in the thermal infrared domain in a simple and effective way, there are still some deficiencies. First, 1) they often ignore the unique properties of data, such as the spatial alignment problem between modalities, which may lead to inaccurate information processing and affect the subsequent accurate feature expression of the model; 2) the prompt module they design only considers spatial information and fails to fully consider in-depth information fusion and interaction at the semantic level. Therefore, these two deficiencies result in insufficient fusion and complementarity of modal information, thereby limiting the tracking performance.

[0004] The target tracking system and method fusing multi-modal features disclosed in CN117522923A introduce a target tracking system and method fusing multi-modal features, which uses deep learning technology to extract RGB features and thermal infrared features from the RGB image and thermal infrared image of target tracking respectively, and uses residual attention mechanism to adaptively adjust the feature weights in the spatial dimension and channel dimension after feature fusion of the RGB features and thermal infrared features through linear combination, and finally uses a classifier to determine whether the target object is out of the tracking range. However, since this technology still uses the method based on attention mechanism, there may still be information redundancy and inaccurate features after the fusion of modal information, so there is still a lot of room for improvement in the results of this method.

[0005] Under the existing technical framework, the processing of multi-modal data sets mainly focuses on directly extracting and processing the feature information of these data. This method usually uses modal fusion technology to integrate the features of different data sources, but it often ignores the unique properties of data, such as the spatial alignment problem between modalities. Specifically, these technologies assume that the spatial alignment between different modalities is complete, but this assumption often does not hold in actual applications, which may lead to inaccurate information processing and affect the subsequent accurate feature expression of the model.

[0006] In addition, although the current method considers the modal interaction at the feature level, this processing is still limited to relatively superficial feature concatenation or simple fusion, lacking exploration of deep semantic relationships between modalities. In the design of the prompt information generator, although efforts are made to integrate and interact spatial location information, this method fails to fully consider in-depth information fusion and interaction at the semantic level. This neglect of semantic levels can result in generated prompt information that fails to accurately reflect the internal connections and complexity of different modal data. SUMMARY

[0007] In order to overcome the deficiencies of the prior art, the purpose of the present application is to provide an infrared tracking method based on feature local correction and multi-modal channel sparse selection prompts, which considers the internal relationship and complementarity of different modal features from two angles of semantic space through the prompt module, improving the effectiveness of the prompt information. Through the feature correction module, the feature deviation problem caused by spatial misalignment is effectively solved.

[0008] In order to achieve the above purpose, the technical scheme adopted by the present application is:

[0009] The infrared tracking method based on feature local correction and multi-modal channel sparse selection prompts comprises the following steps:

[0010] Step (1): The multi-modal video, i.e. the infrared image video and the visible light image video aligned in time, is cut into multiple frames of still images in time sequence to obtain a data set. For a certain time, the data set contains paired visible light and thermal infrared images. Different operations are performed for the training phase and the tracking phase;

[0011] Step (2): For the training phase: the network includes a feature embedding layer, a local feature correction module, a base network, a prompt module and a tracking head. The multi-modal data is input from the feature embedding layer, and the tracking result is finally output from the tracking head. In the training process, the data set is input into the network, the final output is calculated with the real label to obtain the gradient to update the fine-tuning network weight parameters, and the algorithm network weight is obtained. When the network is loaded with weight in the tracking phase, the network weight parameters are used for inference;

[0012] Step (3): For the tracking phase: load the algorithm network weight obtained in the training phase, then mark the location of the target in the first frame of the video, and continuously track the target in the future frames to output the location information of the target.

[0013] The step (2) is specifically:

[0014] (2.1) Obtain training image pairs: Obtain paired image frames from the labeled multi-modal video dataset by extracting a multi-modal template centered on the target and a multi-modal search image, both of which contain the target and are at most T frames apart, the two frames are the template image and the search image of the visible light, the template image and the search image of the infrared, among the two frames, the visible light and infrared image of one frame is used as the search image, and the visible light and infrared image of the other frame is used as the template image; At this time, the network input is a pair of visible light template search images and a pair of corresponding thermal infrared template search images; At the same time, the class of the target is ignored during training, and the proportion of the target in each image is normalized, so as to maintain the aspect ratio of the image;

[0015] (2.2) Preprocess image pairs: During training, adjust and scale the two multi-modal template and multi-modal search images to make the boundary box plus the additional blank of the context have a fixed area;

[0016] (2.3) PatchEmbedding extracts image Patch features: Two pairs of multi-modal template search images are subjected to two-dimensional convolution, and visible light template image features and visible light search image features, thermal infrared template image features and thermal infrared template image features are obtained;

[0017] (2.4) Modal local feature correction: taking the thermal infrared feature as the query Q and the visible light feature as K and V, the V is weighted by using the cross-attention mechanism weight, and the auxiliary image feature is obtained in turn, and finally the auxiliary image and the thermal infrared feature are added to obtain the corrected feature, and the visible light feature remains unchanged;

[0018] (2.5) Generate prompt information: use the modal sparse channel selection prompt module, for the first layer prompt module, take the visible light feature and the corrected thermal infrared feature as input, and for the subsequent prompt module, take the feature output by the basic network and the output of the last layer prompt module as input, pool the two features and add them together, then use different linear changes to decouple the channel features specific to the features, and then horizontally sum the channel weights of the two inputs, in addition, for the feature of visible light (input by the basic network), the feature needs to be weighted again using the spatial attention mechanism after being weighted by the channel weight, and then the two weighted inputs are added together to obtain the prompt information;

[0019] (2.6) Basic network forward propagation: add the prompt information obtained in (2.5) to the output of the corresponding layer encoder to form the input of the next layer encoder; obtain the final feature map;

[0020] (2.7) Input the final feature map into the center tracking head, use the center tracking head to obtain the target center position, center offset and target size, obtain the response in the feature map through the center branch, i.e. the response feature map, obtain the offset of the center position through the offset branch, further fine-tune the target position, and finally obtain the length and width of the center position through the size branch.

[0021] (2.8) Calculate the distribution difference loss between the response feature map and the ground truth feature map, and the loss values ​​of the center position and IOU between the predicted box and the ground truth box, and then backpropagate.

[0022] In step (2.1), the labeled multimodal video datasets are two publicly available visible infrared target tracking datasets: RGBT234 and GTOT.

[0023] The visible light images from different frames (at different times) are used as the search image and the template image, respectively, and the infrared images from different frames (at different times) are used as the search image and the template image, respectively, thus obtaining the visible light template image and the search image, and the corresponding thermal infrared template image and the search image.

[0024] In step (2.2), if the bounding box size is (w, h), the context boundary is p, and the scaling factor is s, the area of ​​the rectangle selected to make the scale-adjusted rectangle is a constant A = s(w + 2p) × s(h + 2p). The area of ​​the template image is A = 128 × 128, and the number of context elements is set to half the average dimension p = (w + h) / 4. The template and search image for each frame are processed offline to avoid resizing the images during training.

[0025] In step (2.4), the thermal infrared features are search map features and template map features; the visible light features are search map features and template map features.

[0026] The design of the modal local feature adaptive correction module consists of two main parts. The first part uses a cross-attention mechanism to generate an auxiliary feature map. The second part is the correction process, which adds the auxiliary feature map to the original infrared features to obtain the corrected features. This structure is used to calibrate the infrared local offset features and reduce the possibility of information accumulation deviation during the chain transmission process.

[0027] Derived from the visible light mode and the thermal infrared mode, respectively denoted as and The labeled sequence constitutes the input of this module. First, the two initial layer inputs, visible light features... and infrared features Split into {Z RGB ,X RGB} and {Z TIR ,X TIR}, Z, X represent the template graph features and search graph features, respectively, and then {Z RGB ,X RGB} pass through Gate K (·) and Gate V (·) to obtain:

[0028]

[0029] {Z TIR ,X TIR} pass through Gate Q (·) to obtain:

[0030]

[0031] Gate K (·), Gate V (·), and Gate Q (·) here are 1x1 convolutions, which pass through Q to query similar pixels in K, construct a modal feature pixel similarity matrix, and then add weights to V to obtain an auxiliary feature map;

[0032]

[0033] Finally, add and splice it with {Z TIR ,X TIR} to obtain the corrected infrared feature:

[0034] H TIR ′=Concat(Z TIR +AuxMap Z ,X TIR +AuxMap X )。

[0035] In the step (2.5), the prompt module 12 layers, the input of each layer is the sum of the output of the previous layer network and the output of the corresponding layer of the prompt module;

[0036] The modal prompt module is inserted into the multi-stage backbone network, and the initial input mark sequence of the two modalities is and a frozen base encoder containing L layers The designed modal sparse channel selection prompt module learns prompt information from the two input streams, and the process is written as:

[0037]

[0038] Where MCSP l (·) represents the l-th layer sparse modal channel selection prompter, P l+1(l = 0, 1,..., L-1) represent the prompt information of different levels, especially when l = 0, P 0 = H TIR ', that is, the output of the feature correction module in (2.4).

[0039] The modal sparse channel selection prompt module includes four steps:

[0040] (i) Feature pooling and fusion decoupling;

[0041] (ii) Use sparse Top-k to select modal significant channels;

[0042] (iii) Weight the original features with lateral channel weights;

[0043] (iv) The base network features are added to the prompt features after spatial attention to generate multimodal prompts.

[0044] In step (2.6), the encoder is also the base network mentioned above, and the encoding is the role of the base network. The encoder is composed of 12 identical transformer blocks stacked together. Each block contains a self-attention calculation and a feedforward neural network. The encoder has a total of 12 layers, and the process needs to be repeated 12 times.

[0045] In step (2.8), first, use the Focal Loss loss function to calculate the loss value between the response feature map and the high-frequency feature map generated by the real label position information, then use L1 Loss to calculate the loss value of the predicted value (x, y, w, h) and the real value (gtx, gty, gtw, gth), and finally use giou Loss to calculate the IOU loss value of the predicted value (x, y, w, h) and the real value (gtx, gty, gtw, gth). Add the weighted sum of the above loss values, use the calculated loss value to update the network parameters, and loop the training multiple times until the network converges. During the update process, the base model weights are frozen and do not participate in the update.

[0046] The Focal Loss is calculated as follows:

[0047]

[0048] FL(p t )=-(1-p t ) γ log(p t )

[0049]

[0050] where w, h are the width and height of the response map, y is the high-dimensional feature map generated by the real label position information, and gamma is a constant.

[0051] The L1 Loss is specifically calculated as follows:

[0052]

[0053] where n is the number of total samples.

[0054] The giou Loss is specifically calculated as follows:

[0055]

[0056] where A represents the predicted frame (x, y, w, h), B represents the real frame (gtx, gty, gtw, gth), and C represents the minimum bounding rectangle of A and B.

[0057] The step (3) is specifically:

[0058] (3.1) Obtain a template image: process the first frame image of the multi-modal video to be tracked as the template image for this tracking according to the method of processing the template image in step (2.2);

[0059] (3.2) Input the final output feature to the tracking head: input the cropped multi-modal search image and the multi-modal template image to the entire algorithm network, which is composed of a basic network, a feature correction module and a prompt module, to obtain a search image feature map prompted by thermal infrared information, and finally obtain the position of the maximum response score through the tracking head, take the predicted value of the offset branch with the highest corresponding score response to refine the target center position, and take the predicted value of the size branch with the highest corresponding score response as the final predicted value of the target length and width;

[0060] (3.3) Crop the next frame search image: after each frame of tracking is completed, the current predicted position is taken as the center to crop the search image for the next frame of tracking, until the last frame is tracked.

[0061] The beneficial effects of the present application are:

[0062] (1) The present application proposes a sparse channel selection prompt module, which decouples the feature-specific channel features using different linear changes after pooling the sum of the two modal information, and then performs adaptive parallel sparse channel selection, which can fully utilize the most significant channel representation of different modalities to generate robust and accurate prompt information and fully utilize the representation advantage of the basic model.

[0063] (2) The application designs a local feature correction module, which uses thermal infrared information to query visible light highlight information, generates thermal infrared local correction features, and adds them to the original features. It can fully utilize the significant features of the visible light that are accurately surrounded by the labeled box to adaptively learn the correction relationship and obtain more accurate thermal infrared features. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 is the implementation flowchart of the application.

[0065] Figure 2 is the model framework diagram of the application.

[0066] Figure 3 is the feature local correction module schematic diagram of the application.

[0067] Figure 4 is the schematic diagram of the multi-modal channel sparse selection prompt module of the application. DETAILED DESCRIPTION

[0068] The application will be further described in detail below in combination with the drawings.

[0069] The thermal infrared target tracking method based on feature local correction and multi-modal channel sparse selection prompt module, the specific implementation includes the following steps:

[0070] (1) Frame interception of multi-modal images: the multi-modal video is intercepted into multiple frames of still images in time sequence, and different operations are performed for the training stage and the tracking stage;

[0071] (2) For the training stage:

[0072] (2.1) Obtain a pair of training images: obtain a pair of image frames from the labeled multi-modal video dataset by extracting a multi-modal template centered on the target and a multi-modal search image, both of which contain the target and are at most T frames apart. At this time, the network input is a pair of visible light template search images and a pair of corresponding thermal infrared template search images. At the same time, the class of the target is ignored during training, and the proportion of the target in each image is normalized, so as to maintain the aspect ratio of the image;

[0073] (2.2) Preprocessing image pairs: During training, both template images are resized to 128x128 pixels and both search images are resized to 256x256 pixels for both modalities. The images are scaled so that the bounding box plus the additional blank of context has a fixed area. More precisely, if the size of the bounding box is (w, h), the size of the context is p, and the scaling factor is s, chosen so that the area of the scaled rectangle is a constant A = s(w + 2p)xs(h + 2p). We use the area A = 128x128 for the template images. The amount of context is set to half of the average dimension p = (w + h) / 4. The template and search images of each frame are preprocessed offline to avoid resizing during training;

[0074] (2.3) Patch Embedding: Two-dimensional convolution is performed on the two pairs of template and search images, and a convolution kernel of size 16*16 is used. This results in visible light template image features [768x8x8], visible light search image features [768x16x16], thermal infrared template image features [768x8x8], and thermal infrared template image features [768x16x16].

[0075] (2.4) Modal local feature correction: The thermal infrared features are used as the query Q, and the visible light features are used as K and V. The cross-attention mechanism is used to weight V, and the auxiliary image features are obtained. Finally, the corrected features are obtained by adding the auxiliary image and the thermal infrared features, and the visible light features remain unchanged.

[0076] As shown in Figure 3 , the network can explicitly utilize infrared information only in the first layer of the base network. If the initial input features are inaccurate, cumulative bias will occur in the subsequent chain transmission process, so the accuracy of the first layer input infrared information is crucial to the performance of the tracker. Since there is a certain spatial position offset between the infrared image and the visible light image, a modal local feature adaptive correction module is designed. This structure mainly consists of two parts. The first part uses the cross-attention mechanism to generate auxiliary feature maps, and the second part is the correction process, which adds the auxiliary feature maps to the original infrared features to obtain the corrected features. Through this structure, the infrared local offset features are calibrated, and the possibility of information cumulative bias in the chain transmission process is reduced.

[0077] Specifically, the labeled sequences from the visible light modality and the thermal infrared modality (denoted as and ) constitute the input of this module. First, the two first-layer inputs, the visible light features and the infrared features are split into {Z RGB ,X RGB} and {Z TIR ,XTIR}, Z, X represent template graph features and search graph features respectively, then {Z RGB ,X RGB} pass through Gate K (·) and Gate V (·) to obtain:

[0078]

[0079] {Z TIR ,X TIR} pass through Gate Q (·) to obtain:

[0080]

[0081] Gate K (·) here, Gate V (·) and Gate Q (·) are 1x1 convolution. Similar pixels in K are queried by Q to construct a modal feature pixel similarity matrix, and then weighted with V to obtain an auxiliary feature map

[0082]

[0083] Finally, it is added and spliced with {Z TIR ,X TIR} to obtain the corrected infrared feature:

[0084] H TIR ' = Concat (Z TIR + AuxMap Z ,X TIR + AuxMap X )

[0085] (2.5) generate prompt information: here mainly use modal sparse channel selection prompt module, for the first layer prompt module, visible light feature and corrected thermal infrared feature are used as input, and for subsequent prompt module, the feature output by the basic network and the output of the last layer prompt module are used as input, the two features are pooled and added, then different linear changes are used to decouple channel-specific features, the channel weights of the two inputs are spliced and transversely calculated, in addition, for the feature of visible light (input by the basic network), the spatial attention mechanism is used again after weighting by the channel weight, then the two weighted inputs are added to obtain the prompt information;

[0086] In this way, the modal sparse channel selection prompt module fully extracts the prompt feature representation of different semantic levels, while learning the semantic and spatial complementarity between the two modalities, and generates more robust prompt information. Sparse channel selection helps to adaptively balance the importance between intermediate base features and learned prompts, as well as the complementarity between the visible light modality and the thermal infrared modality. It provides more accurate and robust target representation for downstream tracking tasks.

[0087] As shown in the accompanying Figure 2 , the proposed modal prompt module is inserted into the multi-stage backbone network. Denoted by the formula, the initial input token sequence of the two modalities is and a frozen base encoder containing L layers , the designed modal sparse channel selection prompt module learns the prompt information from the two input streams, and the process is written as:

[0088]

[0089] where MCSP l (·) represents the l-th layer sparse modal channel selection prompter, P l+1 (l = 0, 1,..., L-1) represents the prompt information at different levels. In particular, when l = 0, P 0 = H TIR ', which is the output of the feature correction module in (2.4).

[0090] The detailed design of the modal sparse channel selection prompt module is shown in the accompanying Figure 4 , which mainly consists of three parts: 1. Convolutional pooling, which is used to reduce the number of operation parameters and obtain initial feature channel information; 2. Fusion decoupling, which is used to fuse the dual modal channel information, and then decouple the modal specific channel features through different fully connected; 3. Sparse Top-K, which is used to eliminate redundant channel features and retain significant channel features. The modal sparse channel selection prompt module has two input branches, which are the token sequence of the base network output stream and the prompt information stream P l .

[0091] The modal sparse channel selection prompt module mainly includes four steps: (i) feature pooling fusion decoupling; (ii) using sparse Top-k to select modal significant channels; (iii) using lateral channel weights to weight the original features; (iv) the base network features are added to the prompt features after spatial attention to generate multimodal prompts. Specifically, considering that the single modal stream has certain redundant features, the prompt block should focus on the most significant channel information in different feature streams, and select the channel weighted information with strong modal specificity.

[0092] To control the amount of network learnable parameters, the two feature streams are first reduced dimensionally to [16×H×W] :

[0093] M base = g1(H l-1 ), M prompt = g2(P l-1 )

[0094] where the projection functions g1(·) and g2(·) are simple 1x1 convolutional layers. Then the channel features of the two streams are obtained using GAP, and the fused feature M shared

[0095] M shared = c shared (GAP(M base )+GAP(M prompt ))

[0096] Then the information-specific linear transformation and sparsification are applied respectively:

[0097] W base = TopK(Fc base (M shared ))

[0098] W prompt = TopK(Fc prompt (M shared ))

[0099] where TopK(·) is the operation of selecting the top-k largest channel features (k = 8) and setting the rest to 0, and and are concatenated to obtain The channel weights of the two feature streams are obtained by SoftMax computation respectively:

[0100] w base ,w prompt = softmax(Concat(W base ,W prompt ))

[0101] Finally, the spatial attention operation is performed on the base embedding M base , which first applies the lambda-smooth spatial SoftMax over all spatial dimensions, and then generates the enhanced embedding M base by applying the channel- spatial attention mask M fovea on M base

[0102]

[0103] where i = 1, 2, …, H, j = 1, 2, …, W, λ is the learnable weighting parameter of each block. Then, the mixed modality embedding is obtained by additive binding, and the learned hint is obtained by the following way,

[0104]

[0105] (2.6) Base network forward propagation: add the hint information obtained in (2.5) to the output of the corresponding layer of the encoder to form the input of the next layer of the encoder. The encoder has 12 layers, and this process needs to be repeated 12 times.

[0106] (2.7) Obtain the target center position, center offset and target size using the center tracking head: obtain the response in the feature map through the center branch, obtain the offset of the center position through the offset branch, further fine-tune the target position, and finally obtain the length and width of the center position through the size branch;

[0107] (2.8) Calculate the loss value and back propagation: first, use the Focal Loss loss function to calculate the loss value between the response map and the high-strength feature map generated by the real label position information, then use L1 Loss to calculate the loss value of the predicted value (x, y, w, h) and the real value (gtx, gty, gtw, gth), and finally use giou Loss to calculate the IOU loss value of the predicted value (x, y, w, h) and the real value (gtx, gty, gtw, gth), add the above loss values by weighting, use the calculated loss value to update the network parameters by back propagation, and loop training multiple times until the network converges. Where (x, y, w, h) represents the center coordinates (x, y) and the width and height (w, h) of the prediction box, and (gtx, gty, gtw, gth) represents the center coordinates (gtx, gty) and the width and height (gtw, gth) of the real box.

[0108] The Focal Loss is calculated as follows:

[0109]

[0110] FL(p t )=-(1-p t ) γ log(p t )

[0111]

[0112] where w, h are the width and height of the response map, y is the high-strength feature map generated by the real label position information, and γ is a constant.

[0113] The L1 Loss is calculated as follows:

[0114]

[0115] wherein n is the number of total samples;

[0116] giou Loss is specifically calculated as follows:

[0117]

[0118]

[0119] wherein A represents a predicted frame (x, y, w, h), B represents a real frame (gtx, gty, gtw, gth), and C represents the minimum bounding box of A and B.

[0120] In the updating process, the basic model weight is frozen and does not participate in the updating;

[0121] (3) For the test phase:

[0122] (3.1) Obtain a template image: process the first frame image of the multi-modal video to be tracked as the template image of this tracking according to the method of processing the template image in (2.2);

[0123] (3.2) Input the final output feature into the tracking head: input the cropped multi-modal search image and the multi-modal template image into the entire algorithm network to obtain the search image feature map prompted by the thermal infrared information, and finally obtain the position of the maximum response score through the tracking head, take the prediction value of the offset branch with the highest corresponding score response to refine the target center position, and take the prediction value of the size branch with the highest corresponding score response as the final prediction value of the target length and width;

[0124] (3.3) Crop the next frame search image: after each frame tracking is completed, the current predicted position is taken as the center to crop the search image of the next frame tracking, and this is continued until the last frame is tracked.

Claims

1. An infrared tracking method based on feature local correction and multimodal channel sparse selection cues, characterized in that, Includes the following steps; Step (1): Extract multiple still images from the multimodal video, i.e., the time-aligned infrared image video and visible light image video, in chronological order to obtain the dataset. For a certain moment, the dataset contains pairs of visible light and thermal infrared images. Different operations are performed for the training and tracking phases. Step (2): For the training phase: During the training process, the dataset is input into the network, and the final output is compared with the real label to calculate the loss, thereby obtaining the gradient to update the fine-tuned network weight parameters and obtain the algorithm network weights. When loading the network weights in the tracking phase, the network weight parameters are used for inference. Step (3): For the tracking stage: Load the network weight parameters obtained in the training stage, then mark the location of the target in the first frame of the video, and continue to track the target in future frames, outputting the location information of the target; Step (2) specifically involves: (2.1) Obtaining training image pairs: By extracting multimodal templates and multimodal search images centered on the target, pairs of image frames are obtained from the labeled multimodal video dataset. Both frames contain the target and are spaced at most T frames apart. These two frames are the visible light template image and search image, and the infrared template image and search image required for subsequent network training. In these two frames, the visible light and infrared images of one frame are used as the search image, and the visible light and infrared images of the other frame are used as the template image. At this time, the network input is a pair of visible light template search images and a pair of corresponding thermal infrared template search images. Meanwhile, the target category is ignored during training, and the target proportion in each image is normalized to maintain the aspect ratio of the image. (2.2) Preprocessing image pairs: During training, the two multimodal templates and the multimodal search image are adjusted and scaled so that the bounding boxes plus the additional blank space of the context have a fixed area; (2.3) Patch Embedding Extraction of Image Patch Features: Two-dimensional convolution of two pairs of multimodal template search maps will yield visible light template map features and visible light search map features, thermal infrared template map features and thermal infrared template map features; (2.4) Modal local feature correction: The thermal infrared feature is used as the query Q, and the visible light feature is used as K and V. The V is weighted using the cross attention mechanism to obtain the auxiliary map features in turn. Finally, the auxiliary map and the thermal infrared feature are summed to obtain the corrected features, while the visible light feature remains unchanged. (2.5) Generating prompt information: The prompt module is selected using modal sparse channel selection. For the first-layer prompt module, the visible light features and the corrected thermal infrared features are used as inputs. For subsequent prompt modules, the features output by the base network and the output of the previous-layer prompt module are used as inputs. The two features are pooled and summed, and different linear transformations are used to decouple the feature-specific channel features. After concatenation, the channel weights of the two inputs are calculated horizontally. In addition, for the visible light features, after being weighted by the channel weights, a spatial attention mechanism is used to weight them again. Then, the two weighted inputs are summed to obtain the prompt information. (2.6) Forward propagation of the basic network: The prompt information obtained in step (2.5) is added to the output of the encoder of the corresponding layer to form the input of the encoder of the next layer; the final feature map is obtained. (2.7) Input the final feature map into the center tracking head, use the center tracking head to obtain the target center position, center offset and target size, obtain the response in the feature map through the center branch, i.e. the response feature map, obtain the offset of the center position through the offset branch, further fine-tune the target position, and finally obtain the length and width of the center position through the size branch. (2.8) Calculate the distribution difference loss between the response feature map and the ground truth feature map, and the loss value between the center position and IOU between the predicted box and the ground truth box, and then backpropagate. In step (2.4), the thermal infrared features are search map features and template map features; Visible light features include search map features and template map features; The design of the modal local feature adaptive correction module consists of two main parts. The first part uses a cross-attention mechanism to generate an auxiliary feature map. The second part is the correction process, which adds the auxiliary feature map to the original infrared features to obtain the corrected features. This structure is used to calibrate the infrared local offset features and reduce the possibility of information accumulation deviation during the chain transmission process. Derived from the visible light mode and the thermal infrared mode, respectively denoted as and The labeled sequence constitutes the input of this module. First, the two initial layer inputs, visible light features... and infrared features Split into {Z RGB ,X RGB } and {Z TIR ,X TIR }, Z and X represent the template graph features and search graph features, respectively, and then {Z RGB ,X RGB } via Gate K (·) and Gate V (·) yields: {Z TIR ,X TIR } via Gate Q (·) yields: Gate here K (·), Gate V (·), Gate Q (·) is a 1×1 convolution. It uses Q to query similar pixels in K, constructs a modal feature pixel similarity matrix, and then weights it with V to obtain an auxiliary feature map. Finally, it is combined with {Z} TIR ,X TIR The corrected infrared features are obtained after summing and splicing: H TIR ′=Concat(Z TIR +AuxMap Z ,X TIR +AuxMap X ); In step (2.5), the prompting module has 12 layers, and the input of each layer is the sum of the output of the previous layer network and the output of the prompting module of the corresponding layer; The modal cueing module is inserted into the multi-stage backbone network, and the initial input label sequences for the two modalities are as follows: And a layer containing L The frozen base encoder, and the designed modal sparse channel selection cue module, learns cue information using these two input streams. This process is written as follows: Among them, MCSP l (·) represents the l-th layer sparse modal channel selection prompt, P l+1 (l = 0, 1, ..., L-1) represent different levels of prompt information. Specifically, when l = 0, P... 0 =H TIR ′, that is, the output of the feature correction module in (2.4); The modal sparse channel selection cue module includes four steps: (i) Feature pooling followed by fusion and decoupling; (ii) Use sparse Top-k selection to select modally significant channels; (iii) Weight the original features using horizontal channel weights; (iv) Basic network features are combined with cue features after spatial attention to generate multimodal cues.

2. The infrared tracking method based on feature local correction and multimodal channel sparse selection cues according to claim 1, characterized in that, In step (2.1), the labeled multimodal video datasets are two publicly available visible infrared target tracking datasets: RGBT234 and GTOT. The visible light images of different frames are used as the search image and template image, respectively, and the infrared images of different frames are used as the search image and template image, respectively, thus obtaining the visible light template image and search image, and the corresponding thermal infrared template image and search image.

3. The infrared tracking method based on feature local correction and multimodal channel sparse selection cues according to claim 1, characterized in that, In step (2.2), if the bounding box size is (w, h), the context boundary is p, and the scaling factor is s, the area of ​​the rectangle selected to make the scale-adjusted rectangle is a constant A = s(w+2p) × s(h+2p); the area of ​​the template image is A = 128 × 128, the number of contexts is set to half of the average dimension p = (w+h) / 4, and the template and search image for each frame are processed offline to avoid resizing the images during training.

4. The infrared tracking method based on feature local correction and multimodal channel sparse selection cues according to claim 1, characterized in that, In step (2.6), the encoder is also the basic network mentioned above. Encoding is the function of the basic network. The encoder is composed of 12 identical transformer blocks stacked together. Each block contains a self-attention computation and a feedforward neural network. The encoder has a total of 12 layers, and this process needs to be repeated 12 times.

5. The infrared tracking method based on feature local correction and multimodal channel sparse selection cueing according to claim 1, characterized in that, In step (2.8), the FocalLoss loss function is first used to calculate the loss value between the response feature map and the Gaussian feature map generated from the real label location information. Then, L1 Loss is used to calculate the loss value between the predicted value (x,y,w,h) and the real value (gtx,gty,gtw,gth). Finally, Giou Loss is used to calculate the IOU loss value between the predicted value (x,y,w,h) and the real value (gtx,gty,gtw,gth). The above loss values ​​are weighted and summed. The calculated loss value is used for backpropagation to update the network parameters. The training is repeated multiple times until the network converges. During the update process, the weights of the basic model are frozen and do not participate in the update.

6. The infrared tracking method based on feature local correction and multimodal channel sparse selection cues according to claim 5, characterized in that, The Focal Loss is calculated in detail as follows: FL(p t )=-(1-p t ) γ log(p t ) Where w and h are the width and height of the response map, y is a Gaussian feature map generated from the real label location information, and γ is a constant; The L1Loss is calculated as follows: Where n is the total number of samples to be calculated; The specific calculation of giou Loss is as follows: Where A represents the predicted bounding box (x, y, w, h), B represents the ground truth bounding box (gtx, gty, gtw, gth), and C represents the minimum bounding rectangle of A and B.

7. The infrared tracking method based on feature local correction and multimodal channel sparse selection cues according to claim 1, characterized in that, Step (3) specifically involves: (3.1) Obtain the template image: Process the first frame of the multimodal video to be tracked according to the method of processing the template image in step (2.2) and use it as the template image for this tracking; (3.2) Input the final output features into the tracking head: Input the cropped multimodal search map and multimodal template map into the entire algorithm network. The algorithm network consists of a basic network, a feature correction module and a prompting module to obtain the search image feature map prompted by thermal infrared information. Finally, the position of the maximum response score is obtained through the tracking head. The predicted value of the offset branch with the highest corresponding score response is taken to refine the target center position. Then, the predicted value of the size branch with the highest corresponding score response is taken as the predicted value of the final target length and width. (3.3) Crop the search image for the next frame: After each frame of tracking is completed, the search image for the next frame of tracking will be cropped with the current predicted position as the center until the last frame is tracked.

Images