Progressive multi-modal target tracking method based on adaptive channel exchange

Abstract

The invention discloses a progressive multi-modal target tracking method based on adaptive channel exchange, which is used for solving the problem of modal difference between an infrared image and a visible light image. According to the method, a modal mixing layer design is adopted, and efficient fusion of cross-modal features is realized through a progressive interaction strategy. Specifically, firstly, a cross-modal channel correlation matrix is calculated, a multi-layer perceptron is utilized to predict a channel exchange probability, and a probability re-parameterization technology is adopted in a training stage to realize adaptive channel selection; and then, inputting the mixed multi-modal features into a parameter-shared double-flow Transform network, and carrying out joint feature extraction and relation modeling on the mixed multi-modal features and the parameter-shared double-flow Transform network. According to the method, through progressive feature interaction, the distribution difference between modes is effectively reduced, and the feature fusion effect is improved. According to the method, the accuracy and robustness of multi-modal target tracking can be remarkably improved in a complex environment, and the method is suitable for a multi-sensor visual system in the fields of intelligent monitoring, automatic driving and the like.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and artificial intelligence, and particularly relates to a progressive multimodal target tracking method and system based on adaptive channel exchange. Background Art

[0002] Multimodal target tracking technology, by fusing complementary information from different sensors, has become a key means of improving tracking performance in complex environments. In practical applications such as security monitoring and autonomous driving, single-modality tracking systems often struggle to cope with challenges such as illumination changes and target occlusion. The coordinated use of visible light and infrared modalities can compensate for each other's respective perception limitations. However, due to the fundamental differences between the two modalities in imaging principles and feature expression, existing fusion methods often struggle to achieve truly effective cross-modal interaction. Infrared images reflect the thermal radiation characteristics of the target and are insensitive to illumination changes, but are weak in representing texture details. Visible light images, on the other hand, can provide rich texture and color information but are prone to failure in low-light conditions. This fundamental difference means that direct feature fusion will introduce a large amount of noise and even cause semantic confusion.

[0003] Currently, there are two typical network architectures in the field of multimodal object tracking: single-stream networks and dual-stream networks. However, both architectures suffer from significant deficiencies in modal interaction. Single-stream networks directly concatenate the inputs of the visible light and thermal infrared modalities and feed them into a shared network for processing. While this crude fusion approach achieves rigid interaction between modalities, because the currently used backbone networks are pre-trained on visible light datasets, they are prone to modality bias during feature extraction. Specifically, the network tends to rely on features from the visible light modality while ignoring the useful information from the thermal infrared modality, especially in shallow layers. In contrast, dual-stream networks use two independent branches to process the visible light and thermal infrared modalities, respectively. While this avoids modal interference during feature extraction, modal fusion is typically performed at deeper layers, completely severing intermodal correlations during early feature extraction. This interaction makes it difficult for the network to establish effective cross-modal correspondences early on, missing the optimal opportunity to leverage shallow features for modal complementarity.

[0004] The fundamental flaw of these two architectures is that they fail to establish a mechanism for progressive, adaptive cross-modal interaction. Single-stream networks perform indiscriminate fusion too early, while dual-stream networks implement inter-modal interaction too late. This paper addresses these issues by proposing a new progressive interaction paradigm that achieves adaptive interaction and complementary enhancement of cross-modal features through continuous fusion at different network depths. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention proposes a progressive multimodal target tracking method based on adaptive channel exchange, which realizes the adaptive fusion of features of visible light and thermal infrared modalities through an innovative modal mixing layer design. This method breaks through the inherent limitations of traditional single-stream and dual-stream networks, and establishes an effective cross-modal interaction mechanism in the initial stage of feature extraction. The adaptive channel exchange unit dynamically selects feature channels with strong complementarity for exchange by calculating the cross-modal channel correlation. Specifically, the probabilistic reparameterization technology is used to achieve differentiable channel selection in the training phase, and deterministic channel exchange is performed directly based on the learned exchange probability in the testing phase. This mechanism can realize modal interaction at the shallow level of the network, effectively alleviating the modal preference problem of the single-stream network and the interaction lag problem of the dual-stream network.

[0006] The technical solution adopted by the present invention is: a progressive multimodal target tracking method based on adaptive channel switching, which has the following steps:

[0007] Step 1: Input the template images and search area images of the visible light and thermal infrared modalities, convert them into image sequences respectively, and embed them into the shared feature space to obtain the initial features of the templates and the initial features of the search area;

[0008] Step 2: The initial features of the visible light and thermal infrared templates and the initial features of the search area are gradually interacted through a multi-level modal mixing layer to output the visible light search area features and the infrared search area features;

[0009] Step 3: After fusing the visible light search area features and the infrared search area features, they are input into the prediction head network for target positioning prediction.

[0010] The advantages of the present invention compared with the prior art are:

[0011] (1) The modal mixing layer proposed in this paper achieves continuous cross-modal feature interaction through adaptive channel exchange, effectively alleviating the modal preference of single-stream networks and the insufficient interaction of dual-stream networks. Furthermore, by dynamically adjusting the exchange ratio, an optimal interaction paradigm is achieved, which mitigates modal differences in the early stage and achieves deep fusion in the late stage.

[0012] (2) Compared with traditional fixed-ratio or heuristic channel selection methods, the adaptive exchange unit of the present invention uses a reparameterization technique during the training phase to achieve discretization decisions while maintaining gradient backpropagation. At the same time, the introduced exchange ratio constraint loss effectively prevents the problem of modal characteristics being lost due to excessive exchange.

[0013] (3) The present invention can maintain its performance advantages in typical challenging scenarios, such as low-light conditions, thermal cross-talk scenarios, and fast motion. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Flowchart of the progressive multimodal target tracking method based on adaptive channel switching of the present invention;

[0015] Figure 2 This is a structural diagram of the modal mixing layer proposed in the present invention;

[0016] Figure 3 This is a structural diagram of the adaptive channel switching unit proposed in the present invention. DETAILED DESCRIPTION

[0017] In order to make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other. To achieve the above-mentioned objectives, the present invention adopts the following technical solutions.

[0018] The specific flow chart of the progressive multimodal target tracking method based on adaptive channel switching of the present invention is as follows: Figure 1 As shown. The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It includes the following steps:

[0019] Step 1: Input the template images and search area images of the visible light and thermal infrared modalities, convert them into image sequences respectively, and embed them into the shared feature space to obtain the initial features of the templates and the initial features of the search area;

[0020] Step 2: The initial features of the visible light and thermal infrared templates and the initial features of the search area are gradually interacted through a multi-level modal mixing layer to output the visible light search area features and the infrared search area features;

[0021] Step 3: After fusing the visible light search area features and the infrared search area features, they are input into the prediction head network for target positioning prediction.

[0022] More specifically, step 1: obtain the search and template images of the visible and infrared modalities, and crop the template images of the two modalities according to the size and position of the target to ensure that the target body and context information are completely preserved; then divide the four images into non-overlapping image blocks, where ; Subsequently, the image sequence is embedded into dimensional shared feature space, usually , and add a learnable positional encoding to get the initial features of the input. The specific calculation formula is as follows:

[0023] (1)

[0024] The meanings of the symbols in formula (1) are:

[0025] : The first frame of original visible light input image;

[0026] : original thermal infrared first frame input image;

[0027] : visible light template image;

[0028] : Thermal infrared template image;

[0029] : visible light search image;

[0030] : Thermal infrared search image;

[0031] : The coordinates of the target's real annotation box;

[0032] : Template cropping area expansion coefficient;

[0033] : visible light template image block;

[0034] : Thermal infrared template image block;

[0035] : visible light search area image block;

[0036] : Thermal infrared search area image block;

[0037] : Initial features of visible light template;

[0038] : Initial features of thermal infrared template;

[0039] : Initial features of the visible light search area;

[0040] : Initial characteristics of thermal infrared search area;

[0041] : Learnable embedding weight matrix for visible light modality;

[0042] : Learnable embedding weight matrix for thermal infrared modality;

[0043] : Position encoding of the template;

[0044] : Position code of the search area;

[0045] : Non-overlapping image block operation;

[0046] : A function that crops an image proportionally based on a labeled box;

[0047] Step 2: The initial features are progressively interacted through multiple layers of modal mixing. Figure 2 As shown in the figure, the modal mixing layer consists of two parts, namely the adaptive channel exchange unit and the joint feature extraction unit. The template and the search area features are spliced to obtain the joint features of the two modalities. and , input mode mixing layer, the calculation process is as follows:

[0048] Step 2a: If Figure 3 As shown in the figure, in the adaptive channel exchange unit, the cross-modal channel correlation of the joint features is calculated to quantify the contribution of each channel to the complementary information. The output of this step is the probability distribution of channel exchange, which provides the basis for the subsequent decision-making mechanism. The calculation formula is as follows:

[0049] (2)

[0050] The meanings of the symbols in formula (2) are:

[0051] : No. Visible light joint features of layer modal mixing layer input;

[0052] : No. Thermal infrared joint features of layer mode mixing layer input;

[0053] : A learnable query matrix for computing channel correlations;

[0054] : A learnable bond matrix for computing channel correlations;

[0055] : probability distribution of channel switching;

[0056] : channel correlation vector;

[0057] : Take the diagonal vector of the matrix;

[0058] : Multilayer Perceptron;

[0059] Step 2b: During the training phase, differentiable reparameterization techniques are used to map the continuous exchange probability vector into an approximately differentiable exchange decision vector. During the testing phase, a hard decision is made based on the trained probability distribution to determine whether each channel performs modal information exchange. This step generates a decision vector indicating whether each channel performs the exchange operation. The calculation formulas for the two stages are as follows:

[0060] During the training phase:

[0061] (3)

[0062] The meanings of the symbols in formula (3) are:

[0063] : No. The probability distribution of whether channels are swapped;

[0064] : independent and identically distributed Gumbel noise;

[0065] : binary exchange decision flag during training phase;

[0066] : A function that returns the index of the maximum value;

[0067] : one-hot encoding;

[0068] During the testing phase:

[0069] (4)

[0070] Step 2c: Use the decision vector to perform feature update and output the bimodal feature representation after channel swapping, as shown below:

[0071] (5)

[0072] The meanings of the symbols in formula (5) are:

[0073] : No. The first thermal infrared feature of the modal mixing layer input channels;

[0074] : No. The first layer of visible light features input to the modal mixing layer channels;

[0075] : No. The first visible light feature after channel exchange of the layer mode mixing layer channels;

[0076] Step 2d: In the feature extraction unit, a parameter-sharing multi-head attention mechanism is used to deeply fuse the bimodal features after channel exchange. The number of multi-head attention heads used is set to 12 to introduce rich multi-view information. At the same time, residual connections and layer normalization are introduced to ensure training stability. The specific calculation process is as follows:

[0077] (6)

[0078] The meanings of the symbols in formula (6) are:

[0079] : No. Query features of the attention heads;

[0080] : No. The learnable bond matrix of the attention heads;

[0081] : No. The learnable value matrix of the attention heads;

[0082] : No. Output features of the attention head;

[0083] : Fusion features of multi-head attention output;

[0084] : Single-head attention dimension;

[0085] : Output projection matrix;

[0086] : layer normalization;

[0087] In step 2, the exchange ratio of the modal mixing layer is dynamically controlled by a constraint loss function to prevent excessive exchange from causing confusion of modal information. The loss function is defined as:

[0088] (7)

[0089] The meanings of the symbols in formula (7) are:

[0090] : No. The actual exchange ratio of the layer;

[0091] : The set exchange ratio threshold;

[0092] : total number of modal mixing layers;

[0093] : channel switching loss;

[0094] During the training process, the network monitors the channel exchange ratio of each modal mixing layer in real time, and accumulates the loss of each layer as the final total exchange ratio loss value. When it exceeds the threshold, a linear penalty is imposed. , to balance the retention of modality-specific information and the acquisition of cross-modal information.

[0095] Step 3: After the two search regions output by the last modal mixing layer are fused, they are input into the prediction head network to predict the target position. This can be divided into two steps:

[0096] Step 3a: Output the target position heat map, size map, and offset map through parallel branches:

[0097] (8)

[0098] The meanings of the symbols in formula (8) are:

[0099] : Visible light search area features output by the last modal mixing layer;

[0100] : Infrared search area features output by the last modal mixing layer;

[0101] : The search area features after fusion;

[0102] : Convolution used to fuse multimodal features;

[0103] : Target center heat map;

[0104] : Center point quantization error offset;

[0105] : target size map;

[0106] : Convolution of predicted heatmap;

[0107] : Convolution of predicted offset;

[0108] : Convolution of predicted target size;

[0109] Step 3b: Determine the center position based on the peak of the heat map, and generate the final bounding box based on the size and offset of the corresponding position:

[0110] (9)

[0111] The meanings of the symbols in formula (9) are:

[0112] : abscissa of the peak position of the heat map;

[0113] : ordinate of the peak position of the heat map;

[0114] : The horizontal coordinate offset of the center point;

[0115] : The vertical coordinate offset of the center point;

[0116] : step length from feature map to original image;

[0117] : Target width predicted by size map;

[0118] : Target height predicted by size map;

[0119] : Width size regression coefficient;

[0120] : Height size regression coefficient;

[0121] : The width of the final bounding box;

[0122] : The height of the final bounding box;

[0123] : The horizontal coordinate of the center of the final bounding box;

[0124] : The vertical coordinate of the center of the final bounding box;

[0125] The heat map prediction branch of the prediction head network in step 3 is optimized using Focal loss, and the size and offset prediction branches are optimized using L1 loss. Combined with the exchange ratio constraint function in the adaptive channel exchange unit, the overall optimization function is:

[0126] (10)

[0127] The meanings of the symbols in formula (10) are:

[0128] : total loss;

[0129] : classification loss;

[0130] : Localization loss for constrained offset prediction;

[0131] : Localization loss that constrains target size prediction;

[0132] : Constraining the loss of channel exchange ratio;

[0133] : The weight of each loss; This allows for an effective balance between coarse-grained and fine-grained localization during training, while avoiding the channel swap constraint from affecting the main task of the tracking model.

[0134] The specific embodiments of the present invention have been fully described and demonstrated through the aforementioned embodiments and accompanying drawings, but the scope of protection of the present invention should not be understood as being limited to these specific embodiments. Any person familiar with the technical field, based on the technical concepts and core innovations disclosed in the present invention, the multimodal target tracking solution realized by equivalent transformation or combined application of technical features, as long as its core technical principle does not deviate from the scope of protection defined in the claims of the present invention, should be regarded as an equivalent technical solution of the present invention and covered within the scope of protection of the present invention.

Claims

1. A progressive multimodal target tracking method based on adaptive channel switching, characterized in that: The following steps are involved: Step 1: Input the template images and search area images of the visible light and thermal infrared modalities, convert them into image sequences respectively, and embed them into the shared feature space to obtain the initial features of the templates and the initial features of the search area; Step 2: The initial features of the visible light and thermal infrared templates and the initial features of the search area are gradually interacted through a multi-level modal mixing layer to output the visible light search area features and the infrared search area features; Step 3: After fusing the visible light search area features and the infrared search area features, they are input into the prediction head network for target positioning prediction.

2. The progressive multimodal target tracking method based on adaptive channel switching according to claim 1, characterized in that: In step 1, the template images of the visible light and thermal infrared modalities are first cropped according to the size and position of the target to ensure that the target body and context information are completely preserved; The four images are then divided into non-overlapping image blocks of the same size, and the image sequence is embedded into dimensional shared feature space, and add a learnable position encoding to obtain the initial input features. The specific calculation formula is as follows: (1) The meanings of the symbols in formula (1) are: : The first frame of original visible light input image; : Original thermal infrared first frame input image; : visible light template image; : Thermal infrared template image; : visible light search image; : Thermal infrared search image; : The coordinates of the target's real annotation box; : Template cropping area expansion coefficient; : visible light template image block; : Thermal infrared template image block; : visible light search area image block; : Thermal infrared search area image block; : Initial features of visible light template; : Initial features of thermal infrared template; : Initial features of the visible light search area; : Initial characteristics of thermal infrared search area; : Learnable embedding weight matrix for visible light modality; : Learnable embedding weight matrix for thermal infrared modality; : Position encoding of the template; : Position code of the search area; : Non-overlapping image block operation; : A function that crops an image proportionally based on a bounding box.

3. The progressive multimodal target tracking method based on adaptive channel switching according to claim 1, characterized in that: The modal mixing layer in step 2 consists of two parts: the adaptive channel exchange unit and the joint feature extraction unit; the template and the search area features are spliced to obtain the joint features of the two modes. and , input mode mixing layer, the calculation process is as follows: Step 2a: In the adaptive channel exchange unit, the cross-modal channel correlation of the joint features is calculated to quantify the contribution of each channel to the complementary information and output the probability distribution of channel exchange; Step 2b: During the training phase, a differentiable reparameterization technique is used to map the continuous swap probability vector into an approximately differentiable swap decision vector. During the testing phase, a hard decision is made based on the trained probability distribution to determine whether each channel performs modal information swapping. This step generates a decision vector indicating whether each channel performs the swap operation. Step 2c: Use the decision vector to update the feature and output the bimodal feature representation after channel swapping; Step 2d: In the feature extraction unit, a parameter-sharing multi-head attention mechanism is used to deeply fuse the bimodal features after channel exchange.

4. The progressive multimodal target tracking method based on adaptive channel switching according to claim 3, characterized in that: In step 2a, the probability distribution of channel switching is calculated as follows: (2) The meanings of the symbols in formula (2) are: : No. Visible light joint features of layer modal mixing layer input; : No. Thermal infrared joint features of layer mode mixing layer input; : A learnable query matrix for computing channel correlations; : A learnable bond matrix for computing channel correlations; : channel correlation vector; : Take the diagonal vector of the matrix; : Multilayer Perceptron; : The probability distribution of channel switching.

5. The progressive multimodal target tracking method based on adaptive channel switching according to claim 3, characterized in that: In step 2b, the calculation formulas for the two stages are as follows: During the training phase: (3) The meanings of the symbols in formula (3) are: : No. The probability distribution of whether the channels are swapped; : independent and identically distributed Gumbel noise; : binary exchange decision flag during training phase; : A function that returns the index of the maximum value; : one-hot encoding; During the testing phase: (4)。 6. The progressive multimodal target tracking method based on adaptive channel switching according to claim 3, characterized in that: In step 2c, the feature update is as follows: (5) The meanings of the symbols in formula (5) are: : No. The thermal infrared joint features of the modal mixing layer input channels; : No. The first layer of visible light joint features input to the modal mixing layer channels; : No. The first visible light joint feature after channel exchange of the layer modal mixing layer channels.

7. The progressive multimodal target tracking method based on adaptive channel switching according to claim 3, characterized in that: In step 2d, the specific calculation process is as follows: (6) The meanings of the symbols in formula (6) are: : No. Query features of the attention heads; : No. The learnable bond matrix of the attention heads; : No. The learnable value matrix of the attention heads; : No. Output features of the attention head; : Fusion features of multi-head attention output; : Single-head attention dimension; : Output projection matrix; : Layer normalization.

8. The progressive multimodal target tracking method based on adaptive channel switching according to claim 1, characterized in that: In step 2, the modal mixing layer dynamically controls the number of channel exchanges through the exchange ratio constraint loss function, which is defined as: (7) The meanings of the symbols in formula (7) are: : No. The actual exchange ratio of the layer; : The set exchange ratio threshold; : total number of modal mixing layers; : Channel exchange loss.

9. The progressive multimodal target tracking method based on adaptive channel switching according to claim 1, characterized in that: In step 3, the fused multimodal features are input into the prediction head network to predict the target position, including: Step 3a: Output the target position heat map, size map, and offset map through the parallel branches of the prediction head network: (8) The meanings of the symbols in formula (8) are: : Visible light search area features output by the last modal mixing layer; : The infrared search area features output by the last modal mixing layer; : The search area features after fusion; : Convolution used to fuse multimodal features; : Target center heat map; : Center point quantization error offset; : target size map; : Convolution of predicted heatmap; : Convolution of predicted offset; : Convolution of predicted target size; Step 3b: Determine the center position based on the peak of the heat map, and generate the final bounding box based on the size and offset of the corresponding position: (9) The meanings of the symbols in formula (9) are: : abscissa of the peak position of the heat map; : ordinate of the peak position of the heat map; : The horizontal coordinate offset of the center point; : The vertical coordinate offset of the center point; : step length from feature map to original image; : Target width predicted by size map; : Target height predicted by size map; : Width size regression coefficient; : Height size regression coefficient; : The width of the final bounding box; : The height of the final bounding box; : The horizontal coordinate of the center of the final bounding box; : The vertical coordinate of the center of the final bounding box.

10. The progressive multimodal target tracking method based on adaptive channel switching according to claim 1, characterized in that: In step 3, the heat map prediction branch of the prediction head network is optimized using Focal loss, and the size and offset prediction branches are optimized using L1 loss. Therefore, the overall optimization function is: (10) The meanings of the symbols in formula (10) are: : total loss; : classification loss; : Localization loss for constrained offset prediction; : Localization loss that constrains target size prediction; : Constraining the loss of channel exchange ratio; : The weight of each loss.

Images