Infrared tracking method based on feature local correction and multi-modal channel sparse selection prompt

Through feature local correction and multimodal channel sparse selection prompt module, the problem of spatial alignment between modes in infrared tracking is solved, semantic-level information fusion is realized, robust prompt information is generated, and infrared tracking performance is improved.

CN119068217AActive Publication Date: 2024-12-03XIDIAN UNIV

Patent Information

Application Number
CN202411122743.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-12-03
Estimated Expiration
2044-08-15

AI Technical Summary

Technical Problem

The existing infrared tracking technology ignores the spatial alignment problem between modes in multimodal data processing, resulting in inaccurate information processing and lack of exploration of deep semantic relationships, which limits the full fusion and complementarity of modal information, affecting tracking. performance.

Method used

Using the method of feature local correction and multimodal channel sparse selection prompt, the feature correction and information fusion of semantic space are performed, and the feature channel is decoupled to generate robust prompts through the cross attention mechanism and the modal sparse channel selection prompt module. information.

Benefits of technology

It effectively solves the feature deviation problem caused by spatial misalignment, makes full use of semantic complementarity between modes, generates accurate prompt information, and improves infrared tracking performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068217A_ABST
    Figure CN119068217A_ABST
Patent Text Reader

Abstract

The invention discloses an infrared tracking method based on feature local correction and multi-modal channel sparse selection prompt. The method comprises the following steps: step (1), intercepting a multi-modal video into multiple frames of still images according to a time sequence to obtain a data set, and performing different operations on a training stage and a tracking stage; (2) training and finely adjusting network weight parameters by using a data set to obtain an algorithm network weight, and reasoning by using the network weight parameters when the network loads the weight in the tracking stage; and (3) the network weight parameters obtained in the training stage are loaded, then the position of the target is marked in the first frame of the video, the target is continuously tracked in a future frame, and the position information of the target is output. According to the method, the internal relation and complementarity of different modal features are considered from two perspectives of semantic space through the prompt module, and the effectiveness of prompt information is improved. And through the feature correction module, the problem of feature deviation caused by space misalignment is effectively solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of target tracking, and in particular relates to an infrared tracking method based on feature local correction and multi-modal channel sparse selection prompts. Background Art

[0002] The method paradigm based on cue fine-tuning has become a hot topic among current researchers. In the field of vision, some studies have added learnable visual cues to the frozen base model, and the research shows that visual cue learning is expected to become an alternative to full fine-tuning. Some recent works have added modal cues to the frozen visible light tracker for visible light infrared multimodal tracking, achieving performance comparable to full fine-tuning.

[0003] Although modal cues improve the representation ability of large visible light pre-trained models in the thermal infrared domain in a simple and effective way, their shortcomings are: 1) they often ignore the unique properties of the data, such as the spatial alignment problem between modalities, which may lead to inaccurate information processing, thus affecting the subsequent accurate feature expression of the model; 2) the cue module they designed only considers spatial information, and fails to fully consider in-depth information fusion and interaction at the semantic level. Therefore, these two shortcomings lead to the failure of modal information to be fully integrated and complementary, thus limiting the tracking performance.

[0004] The target tracking system and method integrating multimodal features with publication number CN117522923A introduces a target tracking system and method integrating multimodal features, which uses deep learning technology to extract RGB features and thermal infrared features from the RGB image and thermal infrared image of target tracking, respectively, and then uses the residual attention mechanism to adaptively adjust the feature weights in the spatial dimension and channel dimension after fusion of RGB features and thermal infrared features by linear combination, and finally uses the classifier to determine whether the target object is out of the tracking range. However, since this technology still uses a method based on the attention mechanism, there may still be information redundancy and inaccurate features after the modal information is fused, so the results of this method still have a lot of room for improvement.

[0005] Under the existing technical framework, the processing of multimodal datasets mainly focuses on directly extracting and processing the feature information of these data. This method usually adopts the technology of modal fusion to integrate the features of different data sources, but they often ignore the unique properties of the data, such as the spatial alignment problem between modalities. Specifically, these technologies assume that the spatial alignment between different modalities is complete, but this assumption is often not true in practical applications, which may lead to inaccurate information processing, thereby affecting the subsequent accurate feature expression of the model.

[0006] In addition, although current methods consider modal interactions at the feature level, this processing is still limited to relatively superficial feature cascading or simple fusion, lacking the exploration of deep semantic relationships between modalities. In the design of the prompt information generator, despite efforts to integrate and interact spatial location information, this approach fails to fully consider in-depth information fusion and interaction at the semantic level. This neglect of the semantic level may result in the generated prompt information failing to accurately reflect the inherent connection and complexity of different modal data. Summary of the invention

[0007] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide an infrared tracking method based on feature local correction and multi-modal channel sparse selection prompts, which considers the inherent connection and complementarity of different modal features from two perspectives of semantic space through a prompt module, thereby improving the effectiveness of prompt information. Through the feature correction module, the feature deviation problem caused by spatial misalignment is effectively solved.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is:

[0009] The infrared tracking method based on feature local correction and multi-modal channel sparse selection prompts includes the following steps:

[0010] Step (1): The multimodal video, i.e., the temporally aligned infrared image video and the visible light image video, is intercepted into multiple frames of still images in chronological order to obtain a data set. For a certain moment, the data set contains pairs of visible light and thermal infrared images, and different operations are performed for the training phase and the tracking phase;

[0011] Step (2): For the training phase: the network includes a feature embedding layer, a local feature correction module, a base network, a prompt module and a tracking head. Multimodal data is input from the feature embedding layer, and finally the tracking result is output from the tracking head. During the training process, the data set is input into the network, and the loss is calculated by comparing the final output with the true label to obtain the gradient to update the fine-tuning network weight parameters and obtain the algorithm network weights. When the network loads the weights in the tracking phase, the network weight parameters are used for inference.

[0012] Step (3): For the tracking phase: load the algorithm network weights obtained in the training phase, then mark the location of the target in the first frame of the video, continue to track the target in future frames, and output the location information of the target.

[0013] The step (2) is specifically:

[0014] (2.1) Obtaining training image pairs: By extracting a multimodal template and a multimodal search image centered on the target, pairs of image frames are obtained from a labeled multimodal video dataset. Both frames contain the target and are separated by at most T frames. The two frames are the visible light template map and search map, infrared template map and search map required for subsequent network training. Among these two frames, one frame of visible light and infrared images is used as the search map, and the other frame of visible light and infrared images is used as the template map. At this time, the network input is a pair of visible light template search maps and a pair of corresponding thermal infrared template search maps. At the same time, the target category is ignored during training, and the target ratio in each image is normalized to maintain the aspect ratio of the image.

[0015] (2.2) Preprocessing image pairs: During the training process, the two multimodal templates and the multimodal search image are adjusted and scaled so that the bounding box plus the additional blank space of the context has a fixed area;

[0016] (2.3) PatchEmbedding extracts image patch features: Performing two-dimensional convolution on two pairs of multimodal template search images will obtain visible light template image features and visible light search image features, thermal infrared template image features and thermal infrared template image features;

[0017] (2.4) Modal local feature correction: The thermal infrared feature is used as the query Q, and the visible light feature is used as K, V. V is weighted using the cross-attention mechanism weights to obtain the auxiliary image features in turn. Finally, the auxiliary image and the thermal infrared feature are added to obtain the corrected feature. The visible light feature remains unchanged.

[0018] (2.5) Generate prompt information: Use the modal sparse channel selection prompt module. For the first-layer prompt module, the visible light features and the corrected thermal infrared features are used as inputs, and for the subsequent prompt modules, the features output by the basic network and the output of the previous layer of prompt modules are used as inputs. The two features are pooled and added together, and the feature-specific channel features are decoupled using different linear changes. After splicing, the channel weights of the two inputs are calculated horizontally. In addition, for the features of visible light (input from the basic network), they need to be weighted again using the spatial attention mechanism after being weighted by the channel weights. Then, the two weighted inputs are added together to obtain the prompt information.

[0019] (2.6) Forward propagation of the basic network: add the prompt information obtained in (2.5) to the output of the encoder of the corresponding layer to form the input of the encoder of the next layer; obtain the final feature map;

[0020] (2.7) Inputting the final feature map into the center tracking head, using the center tracking head to obtain the target center position, center offset and target size, obtaining the response in the feature map through the center branch, that is, the response feature map, obtaining the offset of the center position through the offset branch, further fine-tuning the target position, and finally obtaining the length and width of the center position through the size branch;

[0021] (2.8) Calculate the distribution difference loss value between the response feature map and the true feature map and the loss value of the center position and IOU between the predicted box and the true box and back-propagate.

[0022] In the step (2.1), the annotated multimodal video datasets are two visible light infrared target tracking datasets, RGBT234 and GTOT, which are publicly available online;

[0023] The visible light images of different frames (at different times) are used as search images and template images respectively, and the infrared images of different frames (at different times) are used as search images and template images respectively, so that the visible light template image and search image, and the corresponding thermal infrared template image and search image are obtained.

[0024] In step (2.2), if the size of the bounding box is (w, h), the context boundary is p, and the scaling factor is s, the area of ​​the rescaled rectangle is selected to be a constant A = s(w+2p) × s(h+2p). Using the area of ​​the template image A = 128 × 128, the number of contexts is set to half the average dimension p = (w+h) / 4, and the template and search image of each frame are processed offline to avoid resizing the image during training.

[0025] In the step (2.4), the thermal infrared features are search image features and template image features; the visible light features are search image features and template image features;

[0026] Design a modal local feature adaptive correction module. The structure is mainly divided into two parts. The first part uses the cross-attention mechanism to generate auxiliary feature maps. The second part is the correction process, which adds the auxiliary feature maps to the original infrared features to obtain the corrected features. This structure is used to calibrate the infrared local offset features to reduce the possibility of information accumulation deviation in the chain transmission process.

[0027] From the visible light mode and thermal infrared mode, they are expressed as and The tag sequence of is the input of this module. First, the two first layers are input, the visible light feature and infrared characteristics Split into {Z RGB ,X RGB} and {Z TIR ,X TIR}, Z, X represent the template graph features and search graph features respectively, then {Z RGB ,X RGB}Through Gate K (·) and Gate V (·) and we get:

[0028]

[0029] {Z TIR ,X TIR}Through Gate Q (·) and we get:

[0030]

[0031] Gate here K (·),Gate V (·),Gate Q (·) is a 1×1 convolution, which queries similar pixels in K through Q, constructs a modal feature pixel similarity matrix, and then weights it with V to obtain an auxiliary feature map;

[0032]

[0033] Finally, combine it with {Z TIR ,X TIR}After adding and splicing, we get the corrected infrared features:

[0034] H TIR ′=Concat(Z TIR +AuxMap Z ,X TIR +AuxMap X ).

[0035] In the step (2.5), the prompt module has 12 layers, and the input of each layer is the sum of the output of the previous layer network and the output of the prompt module of the corresponding layer;

[0036] The modality hint module is inserted into the multi-stage backbone network, and the initial input token sequences of the two modalities are and a layer containing L The frozen base encoder is designed to learn the hint information from these two input streams using the modal sparse channel selection hint module. The process is written as:

[0037]

[0038] Among them, MCSP l (·) represents the l-th layer sparse modality channel selection cue, P l+1(l=0,1,...,L-1) represents different levels of prompt information. In particular, when l=0, P 0 =H TIR ′, which is the output of the feature correction module in (2.4).

[0039] The modal sparse channel selection prompt module consists of four steps:

[0040] (i) Feature pooling followed by fusion and decoupling;

[0041] (ii) Use sparse Top-k to select modality-significant channels;

[0042] (iii) weighting the original features using lateral channel weights;

[0043] (iv) The base network features are summed with the cue features after spatial attention to generate multimodal cues.

[0044] In the step (2.6), the encoder is also the basic network mentioned above. Encoding is the function of the basic network. The encoder is composed of 12 identical layers of transformer blocks. Each block contains self-attention calculation and feedforward neural network. The encoder has 12 layers in total, and the process needs to be repeated 12 times.

[0045] In the step (2.8), the Focal Loss loss function is first used to calculate the loss value between the response feature map and the Gaussian feature map generated by the real label position information, and then the L1 Loss is used to calculate the loss value of the predicted value (x, y, w, h) and the real value (gtx, gty, gtw, gth). Finally, the IOU loss value of the predicted value (x, y, w, h) and the real value (gtx, gty, gtw, gth) is calculated using the giou Loss. The above loss values ​​are weighted added, and the calculated loss values ​​are used to back-propagate and update the network parameters. The training is repeated multiple times until the network converges. During the update process, the basic model weights are frozen and do not participate in the update.

[0046] Focal Loss is calculated as follows:

[0047]

[0048] FL(p t )=-(1-p t ) γ log(p t )

[0049]

[0050] Where w and h are the width and height of the response map, y is the Gaussian feature map generated by the true label position information, and γ is a constant;

[0051] The specific calculation of L1 Loss is as follows:

[0052]

[0053] Where n is the number of total samples to be calculated.

[0054] The specific calculation of Giou Loss is as follows:

[0055]

[0056] Where A represents the predicted box (x, y, w, h), B represents the real box (gtx, gty, gtw, gth), and C represents the minimum enclosing rectangular box of A and B.

[0057] The step (3) is specifically:

[0058] (3.1) Obtaining a template image: Processing the first frame of the multimodal video to be tracked as the template image for this tracking according to the method for processing the template image in step (2.2);

[0059] (3.2) Input the final output features into the tracking head: Input the cropped multimodal search image and the multimodal template image into the entire algorithm network. The algorithm network consists of a basic network, a feature correction module, and a prompt module. The search image feature map prompted by the thermal infrared information is obtained. Finally, the position of the maximum response score is obtained through the tracking head. The prediction value of the offset branch with the highest corresponding score response is taken to refine the target center position. The prediction value of the size branch with the highest corresponding score response is taken as the final prediction value of the length and width of the target.

[0060] (3.3) Cropping the next frame search image: After each frame of tracking is completed, the search image of the next frame of tracking will be cropped with the current predicted position as the center until the last frame is tracked.

[0061] Beneficial effects of the present invention:

[0062] (1) The present invention proposes a sparse channel selection prompt module, which pools and adds the two-modal information and uses different linear changes to decouple the feature-specific channel features, and then adaptively parallelizes the sparse channel selection. It can make full use of the most significant channel representations of different modalities to generate robust and accurate prompt information, giving full play to the representation advantages of the basic model.

[0063] (2) The present invention designs a local feature correction module, which uses thermal infrared information to query visible light salient information, generates thermal infrared local correction features, and adds them to the original features. It can make full use of the salient features of visible light that the target is accurately surrounded by the annotation box to adaptively learn the correction relationship and obtain more accurate thermal infrared features. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is an implementation flow chart of the present invention.

[0065] Figure 2 It is a model framework diagram of the present invention.

[0066] Figure 3 It is a schematic diagram of the characteristic local correction module of the present invention.

[0067] Figure 4 It is a schematic diagram of the multimodal channel sparse selection prompt module of the present invention. DETAILED DESCRIPTION

[0068] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0069] The thermal infrared target tracking method based on feature local correction and multi-modal channel sparse selection prompt module includes the following steps:

[0070] (1) Capturing multimodal images by frame: Capturing multimodal videos into multiple still images in chronological order, and performing different operations for the training phase and the tracking phase;

[0071] (2) For the training phase:

[0072] (2.1) Obtain training image pairs: By extracting target-centered multimodal templates and multimodal search images, pairs of image frames are obtained from the annotated multimodal video dataset. Both frames contain the target and are separated by at most T frames. At this time, the network input is a pair of visible light template search images and a pair of corresponding thermal infrared template search images. At the same time, the target category is ignored during training, and the target ratio in each image is normalized to maintain the aspect ratio of the image;

[0073] (2.2) Preprocess image pairs: During training, for both modal images, the template image needs to be resized to 128×128 pixels and the search image is resized to 256×256 pixels. Scale the image so that the bounding box plus the additional padding of the context has a fixed area. More precisely, if the dimensions of the bounding box are (w,h), the bounds of the context are p, and the scaling factor is s, it is chosen so that the area of ​​the rescaled rectangle is a constant A=s(w+2p)×s(h+2p). We use the area of ​​the template image A=128×128. Set the number of contexts to half the average dimension p=(w+h) / 4. The template and search images of each frame are processed offline to avoid resizing the images during training;

[0074] (2.3) Patch Embedding Extracts image patch features: Perform two-dimensional convolution on the two pairs of template search images with a convolution kernel size of 16*16, and you will get visible light template image features [768×8×8], visible light search image features [768×16×16], thermal infrared template image features [768×8×8], and thermal infrared template image features [768×16×16].

[0075] (2.4) Modal local feature correction: The thermal infrared feature is used as the query Q, and the visible light feature is used as K, V. V is weighted using the cross-attention mechanism weight to obtain the auxiliary image feature. Finally, the auxiliary image and the thermal infrared feature are added to obtain the corrected feature. The visible light feature remains unchanged.

[0076] like Figure 3 As shown in the figure, the network can only explicitly use infrared information in the first layer of the basic network. If the initial input features are inaccurate, cumulative deviations will occur in the subsequent chain transmission process. Therefore, the accuracy of the first-layer input infrared information is crucial to the performance of the tracker. Since infrared images and visible light images have a certain spatial position offset, a modal local feature adaptive correction module is designed. The structure is mainly divided into two parts. The first part uses the cross-attention mechanism to generate auxiliary feature maps. The second part is the correction process, which adds the auxiliary feature map to the original infrared feature to obtain the corrected feature. The infrared local offset feature is calibrated through this structure to reduce the possibility of information cumulative deviation in the chain transmission process.

[0077] Specifically, the visible light mode and thermal infrared mode (respectively and ) constitutes the input of this module. First, the two first layers are input, the visible light feature and infrared characteristics Split into {Z RGB ,X RGB} and {Z TIR ,XTIR}, Z, X represent the template graph features and search graph features respectively, then {Z RGB ,X RGB}Through Gate K (·) and Gate V (·) and we get:

[0078]

[0079] {Z TIR ,X TIR}Through Gate Q (·) and we get:

[0080]

[0081] Gate here K (·),Gate V (·),Gate Q (·) is a 1×1 convolution. By querying similar pixels in K through Q, the modal feature pixel similarity matrix is ​​constructed, and then weighted with V to obtain the auxiliary feature map

[0082]

[0083] Finally, combine it with {Z TIR ,X TIR}After adding and splicing, we get the corrected infrared features:

[0084] H TIR ′=Concat(Z TIR +AuxMap Z ,X TIR +AuxMap X )

[0085] (2.5) Generate prompt information: Here, the modal sparse channel selection prompt module is mainly used. For the first-layer prompt module, the visible light features and the corrected thermal infrared features are used as inputs, and for the subsequent prompt modules, the features output by the basic network and the output of the previous layer of prompt modules are used as inputs. The two features are pooled and added, and different linear changes are used to decouple the feature-specific channel features. After splicing, the channel weights of the two inputs are calculated horizontally. In addition, for the features of visible light (input by the basic network), after being weighted by the channel weight, they need to be weighted again using the spatial attention mechanism. Then, the two weighted inputs are added to obtain the prompt information;

[0086] In this way, the modality sparse channel selection cue module fully extracts cue feature representations at different semantic levels, learns the semantic and spatial complementarity between the two modalities, and generates more robust cue information. Sparse channel selection helps to adaptively balance the importance between intermediate basic features and learned cues, as well as the complementarity between visible light and thermal infrared modalities, providing more accurate and robust target representation for downstream tracking tasks.

[0087] As attached Figure 2 As shown in Figure 2, the proposed modality hint module is inserted into the multi-stage backbone network. In terms of formula, the initial input token sequence of the two modalities is and a layer containing L The frozen base encoder is designed to learn the hint information from these two input streams using the modal sparse channel selection hint module. The process is written as:

[0088]

[0089] Among them, MCSP l (·) represents the l-th layer sparse modality channel selection cue, P l+1 (l=0,1,...,L-1) represents different levels of prompt information. In particular, when l=0, P 0 =H TIR ′, which is the output of the feature correction module in (2.4).

[0090] The detailed design of the modal sparse channel selection prompt module is shown in the attached Figure 4 As shown in the figure, the structure is mainly divided into three parts: 1. Convolution pooling, which is used to reduce the number of operation parameters and obtain the initial characteristic channel information; 2. Fusion decoupling, which is used to fuse the dual-modal channel information and then adaptively decouple the modality-specific channel features through different full connections; 3. Sparse Top-K, which is used to remove redundant channel features and retain significant channel features. The modality sparse channel selection prompt module has two input branches, namely the basic network output stream and prompt information flow P l The token sequence.

[0091] The modality sparse channel selection prompt module mainly includes four steps: (i) feature pooling and then fusion decoupling; (ii) using sparse Top-k to select modality-significant channels; (iii) using lateral channel weights to weight the original features; (iv) adding the basic network features to the prompt features after spatial attention to generate multimodal prompts. Specifically, considering that the single-modality stream has certain redundant characteristics, the prompt block should focus on the most significant channel information in different feature streams and select the channel weighted information with strong modality specificity.

[0092] In order to control the amount of learnable parameters of the network, we first reduce the dimensions of the two feature flows to [16×H×W] :

[0093] M base =g1(H l-1 ),M prompt =g2(P l-1 )

[0094] Among them, the projection functions g1(·) and g2(·) are simple 1×1 convolutional layers. Then GAP is used to obtain the channel features of the two streams, and after adding them, a weight-sharing linear layer is used to obtain the fusion feature M. shared

[0095] M shared =c shared (GAP(M base )+GAP(M prompt ))

[0096] Then, information-specific linear changes and sparse operations are performed respectively:

[0097] W base =TopK(Fc base (M shared ))

[0098] W prompt =TopK(Fc prompt (M shared ))

[0099] Among them, TopK(·) is the channel feature with the largest Top-k before sparse selection (k=8), and the rest of the positions are set to 0. and After splicing, we get After performing SoftMax calculation horizontally, the weights of the two feature stream channels are obtained respectively.

[0100] w base ,w prompt =softmax(Concat(W base ,W prompt ))

[0101] Finally, the base embedding M base Perform a spatial attention operation that first applies a λ-smoothed spatial SoftMax across all spatial dimensions and then base Apply channel-space attention class mask M on fovea , generating enhanced embeddings

[0102]

[0103] where i = 1, 2, ..., H, j = 1, 2, ..., W, and λ are learnable weight parameters for each block. Then, the mixed modal embedding is obtained by additive binding, and the learned hint is obtained by,

[0104]

[0105] (2.6) Forward propagation of the basic network: add the hint information obtained in (2.5) to the output of the encoder of the corresponding layer to form the input of the encoder of the next layer. The encoder has 12 layers in total, and this process needs to be repeated 12 times.

[0106] (2.7) Use the center tracking head to obtain the target center position, center offset and target size: obtain the response in the feature map through the center branch, obtain the offset of the center position through the offset branch, further fine-tune the target position, and finally obtain the length and width of the center position through the size branch;

[0107] (2.8) Calculate the loss value and back propagate: First, use the Focal Loss loss function to calculate the loss value between the response map and the Gaussian feature map generated by the real label position information, then use L1 Loss to calculate the loss value of the predicted value (x, y, w, h) and the real value (gtx, gty, gtw, gth), and finally use giou Loss to calculate the IOU loss value of the predicted value (x, y, w, h) and the real value (gtx, gty, gtw, gth). Add the above loss values ​​weightedly, use the calculated loss value to back propagate and update the network parameters, and repeat the training cycle for multiple times until the network converges. Among them, (x, y, w, h) represents the center coordinates (x, y) and width and height (w, h) of the predicted box, and (gtx, gty, gtw, gth) represents the center coordinates (gtx, gty) and width and height (gtw, gth) of the real box.

[0108] Focal Loss is calculated as follows:

[0109]

[0110] FL(p t )=-(1-p t ) γ log(p t )

[0111]

[0112] Where w and h are the width and height of the response map, y is the Gaussian feature map generated by the true label position information, and γ is a constant.

[0113] The specific calculation of L1 Loss is as follows:

[0114]

[0115] Where n is the number of total samples calculated;

[0116] The specific calculation of Giou Loss is as follows:

[0117]

[0118]

[0119] Where A represents the predicted box (x, y, w, h), B represents the real box (gtx, gty, gtw, gth), and C represents the minimum enclosing rectangular box of A and B.

[0120] During the update process, the basic model weights are frozen and do not participate in the update;

[0121] (3) For the testing phase:

[0122] (3.1) Obtaining a template image: Processing the first frame of the multimodal video to be tracked as the template image for this tracking according to the method for processing the template image in (2.2);

[0123] (3.2) Input the final output features into the tracking head: Input the cropped multimodal search image and the multimodal template image into the entire algorithm network to obtain the search image feature map prompted by the thermal infrared information. Finally, the position of the maximum response score is obtained through the tracking head. The prediction value of the offset branch with the highest corresponding score response is taken to refine the target center position. Then, the prediction value of the size branch with the highest corresponding score response is taken as the final prediction value of the length and width of the target.

[0124] (3.3) Cropping the next frame search image: After each frame of tracking is completed, the search image of the next frame of tracking will be cropped with the current predicted position as the center until the last frame is tracked.

Claims

1. An infrared tracking method based on feature local correction and multimodal channel sparse selection prompts, characterized in that: The steps include: Step (1): The multimodal video, i.e., the temporally aligned infrared image video and the visible light image video, is intercepted into multiple frames of still images in chronological order to obtain a data set. For a certain moment, the data set contains pairs of visible light and thermal infrared images, and different operations are performed for the training phase and the tracking phase; Step (2): For the training phase: During the training process, the data set is input into the network, and the loss is calculated by comparing the final output with the true label, so as to obtain the gradient to update the fine-tuning network weight parameters and obtain the algorithm network weight. When the network loads the weight in the tracking phase, the network weight parameters are used for inference; Step (3): For the tracking phase: load the network weight parameters obtained in the training phase, then mark the location of the target in the first frame of the video, continue to track the target in future frames, and output the location information of the target.

2. The infrared tracking method based on feature local correction and multi-modal channel sparse selection prompt according to claim 1, characterized in that: The step (2) is specifically: (2.1) Obtaining training image pairs: By extracting a multimodal template and a multimodal search image centered on the target, pairs of image frames are obtained from a labeled multimodal video dataset. Both frames contain the target and are separated by at most T frames. The two frames are the visible light template map and search map, infrared template map and search map required for subsequent network training. Among the two frames, one frame of visible light and infrared images is used as the search map, and the other frame of visible light and infrared images is used as the template map. At this time, the network input is a pair of visible light template search maps and a pair of corresponding thermal infrared template search maps. At the same time, the target category is ignored during training, and the target ratio in each image is normalized to maintain the aspect ratio of the image. (2.2) Preprocessing image pairs: During the training process, the two multimodal templates and the multimodal search image are adjusted and scaled so that the bounding box plus the additional blank space of the context has a fixed area; (2.3) Patch Embedding Extracts image patch features: Performing two-dimensional convolution on two pairs of multimodal template search images will obtain visible light template image features and visible light search image features, thermal infrared template image features and thermal infrared template image features; (2.4) Modal local feature correction: The thermal infrared feature is used as the query Q, and the visible light feature is used as K, V. V is weighted using the cross-attention mechanism weights to obtain the auxiliary image features in turn. Finally, the auxiliary image and the thermal infrared feature are added to obtain the corrected feature. The visible light feature remains unchanged. (2.5) Generate prompt information: Use the modal sparse channel selection prompt module. For the first-layer prompt module, the visible light features and the corrected thermal infrared features are used as inputs, and for the subsequent prompt modules, the features output by the basic network and the output of the previous layer prompt module are used as inputs. The two features are pooled and added, and different linear changes are used to decouple the feature-specific channel features. After splicing, the channel weights of the two inputs are calculated horizontally. In addition, for the visible light features, after being weighted by the channel weights, they need to be weighted again using the spatial attention mechanism. Then, the two weighted inputs are added to obtain the prompt information. (2.6) Forward propagation of the basic network: add the prompt information obtained in step (2.5) to the output of the encoder of the corresponding layer to form the input of the encoder of the next layer; obtain the final feature map; (2.7) Inputting the final feature map into the center tracking head, using the center tracking head to obtain the target center position, center offset and target size, obtaining the response in the feature map through the center branch, that is, the response feature map, obtaining the offset of the center position through the offset branch, further fine-tuning the target position, and finally obtaining the length and width of the center position through the size branch; (2.8) Calculate the distribution difference loss value between the response feature map and the true feature map and the loss value of the center position and IOU between the predicted box and the true box and back-propagate.

3. The infrared tracking method based on feature local correction and multi-modal channel sparse selection prompt according to claim 2, characterized in that: In the step (2.1), the annotated multimodal video datasets are two visible light infrared target tracking datasets, RGBT234 and GTOT, which are publicly available online; Visible light images of different frames are used as search images and template images respectively, and infrared images of different frames are used as search images and template images respectively, so as to obtain visible light template images and search images, and corresponding thermal infrared template images and search images.

4. The infrared tracking method based on feature local correction and multi-modal channel sparse selection prompt according to claim 2, characterized in that: In step (2.2), if the size of the bounding box is (w, h), the context boundary is p, and the scaling factor is s, the area of ​​the rescaled rectangle is selected to be a constant A = s(w+2p) × s(h+2p); the area of ​​the template image is used A = 128 × 128 , the number of contexts is set to half of the average dimension p = (w + h) / 4, and the template and search image of each frame are processed offline to avoid resizing the image during training.

5. The infrared tracking method based on feature local correction and multi-modal channel sparse selection prompt according to claim 2, characterized in that: In the step (2.4), the thermal infrared features are search image features and template image features; The visible light features are search graph features and template graph features; Design a modal local feature adaptive correction module. The structure is mainly divided into two parts. The first part uses the cross-attention mechanism to generate auxiliary feature maps. The second part is the correction process, which adds the auxiliary feature maps to the original infrared features to obtain the corrected features. This structure is used to calibrate the infrared local offset features to reduce the possibility of information accumulation deviation in the chain transmission process. From the visible light mode and thermal infrared mode, they are expressed as and The tag sequence of is the input of this module. First, the two first layers are input, the visible light feature and infrared characteristics Split into {Z RGB ,X RGB } and {Z TIR ,X TIR }, Z, X represent the template graph features and search graph features respectively, then {Z RGB ,X RGB }Through Gate K (·) and Gate V (·) and we get: I∈{X,Z} I∈{X,Z} {Z TIR ,X TIR }Through Gate Q (·) and we get: Gate here K (·),Gate V (·),Gate Q (·) is a 1×1 convolution, which queries similar pixels in K through Q, constructs a modal feature pixel similarity matrix, and then weights it with V to obtain an auxiliary feature map; Finally, combine it with {Z TIR ,X TIR }After adding and splicing, we get the corrected infrared features: H TIR ′=Concat(Z TIR +AuxMap Z ,X TIR +AuxMap X )。 6. The infrared tracking method based on feature local correction and multi-modal channel sparse selection prompt according to claim 5, characterized in that: In the step (2.5), the prompt module has 12 layers, and the input of each layer is the sum of the output of the previous layer network and the output of the prompt module of the corresponding layer; The modality hint module is inserted into the multi-stage backbone network, and the initial input token sequences of the two modalities are and a layer containing L The frozen base encoder is designed to learn the hint information from these two input streams using the modal sparse channel selection hint module. The process is written as: l=0,1,...,L-1 Among them, MCSP l (·) represents the l-th layer sparse modality channel selection cue, P l+1 (l=0,1,...,L-1) represents different levels of prompt information. In particular, when l=0, P 0 =H TIR ′, which is the output of the feature correction module in (2.4); The modal sparse channel selection prompt module consists of four steps: (i) Feature pooling followed by fusion and decoupling; (ii) Use sparse Top-k to select modality-significant channels; (iii) weighting the original features using lateral channel weights; (iv) The base network features are summed with the cue features after spatial attention to generate multimodal cues.

7. The infrared tracking method based on feature local correction and multi-modal channel sparse selection prompt according to claim 6, characterized in that: In the step (2.6), the encoder is also the basic network mentioned above. Encoding is the function of the basic network. The encoder is composed of 12 identical layers of transformer blocks. Each block contains self-attention calculation and feedforward neural network. The encoder has 12 layers in total, and the process needs to be repeated 12 times.

8. The infrared tracking method based on feature local correction and multi-modal channel sparse selection prompt according to claim 2, characterized in that: In the step (2.8), the FocalLoss loss function is first used to calculate the loss value between the response feature map and the Gaussian feature map generated by the real label position information, and then the L1Loss is used to calculate the loss value of the predicted value (x, y, w, h) and the real value (gtx, gty, gtw, gth). Finally, the IOU loss value of the predicted value (x, y, w, h) and the real value (gtx, gty, gtw, gth) is calculated using the giou Loss. The above loss values ​​are weighted added, and the calculated loss values ​​are used to back-propagate and update the network parameters. The training is repeated multiple times until the network converges. During the update process, the basic model weights are frozen and do not participate in the update.

9. The infrared tracking method based on feature local correction and multi-modal channel sparse selection prompt according to claim 8, characterized in that: Focal Loss is calculated as follows: FL(p t )=-(1-p t ) γ log(p t ) Where w and h are the width and height of the response map, y is the Gaussian feature map generated by the true label position information, and γ is a constant; The specific calculation of L1Loss is as follows: Where n is the number of total samples to be calculated. The specific calculation of Giou Loss is as follows: Where A represents the predicted box (x, y, w, h), B represents the real box (gtx, gty, gtw, gth), and C represents the minimum enclosing rectangular box of A and B.

10. The infrared tracking method based on feature local correction and multi-modal channel sparse selection prompt according to claim 2, characterized in that: The step (3) is specifically: (3.1) Obtaining a template image: Processing the first frame of the multimodal video to be tracked as the template image for this tracking according to the method for processing the template image in step (2.2); (3.2) Input the final output features into the tracking head: Input the cropped multimodal search image and the multimodal template image into the entire algorithm network. The algorithm network consists of a basic network, a feature correction module, and a prompt module. The search image feature map prompted by the thermal infrared information is obtained. Finally, the position of the maximum response score is obtained through the tracking head. The prediction value of the offset branch with the highest corresponding score response is taken to refine the target center position. The prediction value of the size branch with the highest corresponding score response is taken as the final prediction value of the length and width of the target. (3.3) Cropping the next frame search image: After each frame of tracking is completed, the search image of the next frame of tracking will be cropped with the current predicted position as the center until the last frame is tracked.

Citation Information

Patent Citations

  • Human face recognition activated voice navigation method and system, storage medium and equipment

    CN110908718A

  • RGBT unmanned aerial vehicle target tracking method and system based on multi-stage attention mechanism

    CN116188528A

  • Video target real-time tracking system and method based on template prompt

    CN118037778A

  • Natural language interaction for smart assistant

    IN201947029578A

Cited By

  • Target tracking algorithm based on feature complementary fusion and key frame template updating

    CN119380045A

  • Multi-modal target tracking method guided by sparse space-time mark

    CN120411166A

  • Multi-modal target tracking method and system based on time sequence modeling and prompt fine tuning

    CN120510480A

  • Visible light infrared target tracking method and system based on edge information dynamic enhancement

    CN120807561A

  • Agricultural disease hash retrieval method based on DV stabilization and adaptive feature enhancement

    CN121278128A