A visible-infrared target tracking method and system based on modal mass perception and multi-level interaction

Abstract

The application discloses a visible light-infrared target tracking method and system based on modal quality perception and multi-level interaction, and belongs to the field of computer vision and multi-modal perception. The method is based on a double-stream encoder-single decoder network (DESDNet) and comprises: a pixel-level cross-modal fusion module for modal quality perception of shallow features to generate a fusion enhanced template; a feature-level cross-modal fusion module for extracting modal specificity and shared features by using an efficient channel attention and a gated feature reorganization unit; and a multi-branch decision-level fusion module for reliability evaluation and correction of multiple prediction heads. The application constructs a full-link multi-level interactive fusion mechanism of "pixel level-feature level-decision level", avoids the introduction of noise, adaptively mines deep effective information of the disturbed modal, and significantly improves the accuracy and robustness of target tracking in complex environments.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a visible light-infrared target tracking method and system based on modal quality perception and multi-level interaction. Background Technology

[0002] Traditional RGBT multimodal fusion tracking algorithms typically perform information fusion at a single level (such as feature level or decision level only), which can easily lead to the loss of detailed information during feature extraction.

[0003] In complex real-world environments, when a particular mode (such as visible light or infrared) is interfered with by strong light, rain, fog, or other factors, its shallow information is significantly reduced or overwhelmed by noise. Existing technologies often employ strategies such as simply discarding low-quality modes or performing indiscriminate weighted fusion. This approach not only fails to fully leverage the complementary advantages of multimodal sensor information and fails to extract the deeper, effective information remaining in the interfered modes, but also easily introduces interference noise into subsequent tracking networks, leading to decreased tracking accuracy, easy model drift, and over-reliance on ideal imaging conditions. Summary of the Invention

[0004] The purpose of this invention is to address the technical pain points of insufficient multimodal information mining capabilities and low utilization rate of effective information from disturbed modes in existing RGBT tracking algorithms. By constructing a multi-level (pixel-level-feature-decision-optimization) interactive and progressive fusion mechanism of "perception-evaluation-optimization", this invention can adaptively extract and retain fine-grained details and deep auxiliary information across modes while avoiding the introduction of low-quality noise, thereby significantly improving the tracking accuracy and robustness of multimodal weak targets in highly dynamic and strongly disturbed complex scenes.

[0005] The technical solution of the present invention: According to a first aspect of the present invention, a visible light-infrared target tracking method based on modal quality perception and multi-level interaction is proposed, comprising the following steps: Step S1: Input the template images of the visible light mode and the infrared mode and the search image into the dual-stream backbone network extractor respectively to obtain the shallow feature map of the first stage.

[0006] The dual-stream backbone network adopts a weight-sharing twin network structure based on Swin-Transformer. In the first stage, the input image size is converted from H×W×3 to H / 4×W / 4×48 through two SPD-Conv modules.

[0007] Step S2: The shallow feature map is input to the pixel-level cross-modal fusion module. Distribution difference processing is performed by equipping each modality with an independent modality-specific normalization layer. A dynamic modality quality mask is generated using a lightweight convolutional network. The dynamic modality quality mask is multiplied pixel by pixel with the original image of the corresponding modality and weighted and added to obtain a pixel-level fusion enhancement template that retains fine-grained information.

[0008] The shallow feature map is input to a pixel-level cross-modal fusion module. Distribution differences are processed by equipping each modality with an independent Modality-Specific Normalization (MSBN) layer, and a dynamic modality quality mask is generated using a lightweight convolutional network. This lightweight convolutional network uses 1×1 convolution for dimensionality reduction, ReLU activation, and then another 1×1 convolution to compress the channels to 1 or 3. Finally, it is normalized to the [0,1] interval using a sigmoid activation function. The quality mask is upsampled to the original image size and then multiplied pixel-by-pixel with the original image of the corresponding modality and weighted summed to obtain a pixel-level fusion enhancement template that retains fine-grained information. The upsampling operation employs a continuous mask generation method based on implicit neural representation. Fourier position encoding maps spatial coordinates to a high-dimensional feature space, and combined with local feature vectors in the low-resolution feature map, a multilayer perceptron (MLP) is used to fit the continuous mask distribution.

[0009] Step S3: Input the feature maps of the last three stages of the backbone network into the feature-level cross-modal fusion module.

[0010] First, the efficient channel attention mechanism (ECA) is used to perform global average pooling on the feature maps of each stage. Channel weights are generated by one-dimensional convolution and sigmoid activation function, and then multiplied with the elements of the original feature map to obtain the feature vectors with modality-specific enhancement.

[0011] Secondly, using the gated feature recombination unit (GFRU), the pixel-level fusion enhancement template is used as the common feature basis, and the basis features are extracted through depthwise separable convolution. The modality-specific information vector is used to generate a spatial adaptive gated signal through 1×1 convolution and 3×3 convolution. The spatial gated feature map is multiplied element-wise with the common feature basis and then added element-wise to obtain the modality shared feature vector.

[0012] Step S4: The visible light specific information, infrared specific information, and modality shared feature vector are respectively input into the prediction heads of three parallel Region Candidate Networks (RPNs) in the multi-branch decision-level fusion module, and the initial target bounding boxes and confidence scores corresponding to the modality tendencies are output.

[0013] Based on the pixel-level fusion enhancement template, a lightweight evaluation network is used to score the reliability of the initial target bounding box. The target region is cropped from the pixel-level fusion enhancement template according to the position of each candidate bounding box, encoded using 3×3 convolutions, ReLU activation, and global average pooling, and then passed through a fully connected layer and a sigmoid activation function to generate a reliability score. The initial confidence score is multiplied by the reliability score to obtain a corrected confidence score, and the modality with the highest corrected confidence score is selected. Finally, a region centered on this result is cropped from the pixel-level fusion enhancement template with an area twice the size of the original bounding box. A lightweight regression network with three layers of 3×3 convolutions and one ×1 convolution outputs the offset, which is added to the initial target bounding box to obtain the final target position.

[0014] In one possible embodiment, in step S2, the mode-specific normalization layer (MSBN) performs batch normalization on the first-stage output feature maps of the visible light branch and the infrared branch respectively to adapt to the differences in data distribution of different modes.

[0015] In one possible embodiment, in step S2, the process of generating a dynamic modal quality mask using a lightweight convolutional network includes: using a 1×1 convolution to reduce the dimensionality of the feature map processed by MSBN, using the ReLU activation function, then using a 1×1 convolution to compress the channels to 1 or 3, and finally using the sigmoid activation function to normalize the output value to the [0,1] interval to obtain the quality response distribution mask of the corresponding modality.

[0016] In one possible embodiment, in step S2, the upsampling operation of upsampling the quality mask to the original image size is a continuous mask generation method based on implicit neural representation: using Fourier position encoding to map spatial coordinates to a high-dimensional feature space, combining local feature vectors in the low-resolution feature map, and fitting a continuous mask distribution through a multilayer perceptron (MLP).

[0017] In one possible embodiment, the efficient channel attention mechanism (ECA) in step S3 is implemented as follows: global average pooling is performed on the feature maps of each stage, channel weights are generated by one-dimensional convolution and sigmoid activation function, and then multiplied with the elements of the original feature map to obtain the feature vectors with modality-specific enhancement.

[0018] In one possible embodiment, in step S3, the gated feature recombination unit (GFRU) is implemented as follows: common feature basis is extracted from the pixel-level fusion enhancement template using depthwise separable convolution; spatial gated feature map is generated by using modality-specific information vector through 1×1 convolution and 3×3 convolution; the spatial gated feature map is multiplied element-wise with the common feature basis and then added element-wise to obtain the modality shared feature vector.

[0019] In one possible embodiment, in step S4, the process by which the lightweight evaluation network scores the reliability of the initial target bounding box includes: cropping the target region from the pixel-level fusion enhancement template according to the position of each candidate bounding box, encoding it through 3×3 convolution, ReLU activation function and global average pooling, and then generating a reliability score through a fully connected layer and sigmoid activation function.

[0020] In one possible embodiment, in step S4, the local fine-tuning regression is implemented as follows: taking the initial target bounding box with the highest corrected confidence as the center, a region with a coverage area twice that of the original bounding box is cropped in the pixel-level fusion enhancement template, and the offset is output through a lightweight regression network with 3 layers of 3×3 convolution and 1×1 convolution. The offset is added to the initial target bounding box to obtain the final target position.

[0021] According to a second aspect of the present invention, a visible-infrared target tracking system based on modal quality perception and multi-level interaction is proposed to implement the aforementioned visible-infrared target tracking method based on modal quality perception and multi-level interaction, comprising: A pixel-level cross-modal fusion module is used to input template images of visible light mode and infrared mode and search image into a dual-stream backbone network extractor to obtain shallow feature maps in the first stage. The shallow feature maps are then input into the pixel-level cross-modal fusion module, where distribution difference processing is performed by equipping each modality with an independent modality-specific normalization layer (MSBN), and a dynamic modality quality mask is generated using a lightweight convolutional network. The quality mask is then multiplied pixel by pixel with the original image of the corresponding modality and weighted and summed to obtain a pixel-level fusion enhancement template that retains fine-grained information. The feature-level cross-modal fusion module is used to input the feature maps of the last three stages of the backbone network into the feature-level cross-modal fusion module, use the efficient channel attention mechanism (ECA) to enhance the specific information of each modality, and use the gated feature recombination unit (GFRU) with the pixel-level fusion enhancement template as the common feature base, combine the modality-specific information to generate a spatial adaptive gated signal, and dynamically fuse to obtain the modality shared feature vector; The multi-branch decision-level fusion module is used to input visible light-specific information, infrared-specific information, and modality-shared feature vectors into the prediction heads of three parallel Region Candidate Networks (RPNs) in the multi-branch decision-level fusion module, and output the initial target bounding box and confidence score corresponding to the modality tendency. Based on the pixel-level fusion enhancement template, the initial target bounding box is scored for reliability by a lightweight evaluation network, and the target bounding box with the highest corrected confidence score is selected for local fine-tuning regression to output the final target position.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Multi-layered progressive fusion architecture: It breaks through the limitations of traditional single fusion layer and establishes a full-link multimodal interaction channel of "pixel level - feature level - decision level". Through the innovative pixel-level "jump connection" mechanism, it effectively overcomes the problem of loss of fine-grained texture information in deep networks.

[0023] 2. Adaptive extraction based on modality quality awareness: Instead of simply discarding disturbed modes, we use independent and specific normalization layers and quality masks to filter out noise while deeply mining their auxiliary discriminative features.

[0024] 3. Decision-level inversion correction mechanism: Using the pixel-level fused image with the richest original information as a global arbiter, the outputs of the three independent high-dimensional prediction heads are evaluated and spatially offset corrected to achieve closed-loop optimization. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the DESDNet network structure; Figure 2 This is a schematic diagram of the Spd-conv convolution operation process. Detailed Implementation

[0026] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0027] A visible-infrared target tracking method based on modal quality perception and multi-level interaction includes the following steps: Step S1: Input the template images of the visible light mode and the infrared mode and the search image into the dual-stream backbone network extractor respectively to obtain the shallow feature map of the first stage; Step S2: Input the shallow feature map into the pixel-level cross-modal fusion module. By equipping each modality with an independent modality-specific normalization layer for distribution difference processing, and using a lightweight convolutional network to generate a dynamic modality quality mask, the dynamic modality quality mask is multiplied pixel by pixel with the original image of the corresponding modality and weighted summed to obtain a pixel-level fusion enhancement template that retains fine-grained information. Step S3: Input the feature maps of the last three stages of the backbone network into the feature-level cross-modal fusion module, use the efficient channel attention mechanism to enhance the specific information of each modality, and use the gated feature recombination unit with the pixel-level fusion enhancement template as the common feature base, combine the modality-specific information to generate a spatial adaptive gated signal, and dynamically fuse to obtain the modality-shared feature vector. Step S4: The visible light-specific information, infrared-specific information, and modality-shared feature vector are input into the prediction heads of three parallel region candidate networks in the multi-branch decision-level fusion module, respectively, and the initial target bounding boxes and confidence scores corresponding to the modality tendencies are output. Based on the pixel-level fusion enhancement template, the reliability of the initial target bounding boxes is scored by a lightweight evaluation network, and the target bounding box with the highest corrected confidence score is selected for local fine-tuning regression to output the final target position.

[0028] In one embodiment, the scenario is summarized as follows: This embodiment is designed for continuous target tracking by a drone under complex weather conditions.

[0029] System hardware configuration: edge computing platforms such as NVIDIA Jetson Orin Nano / Rockchip RK3588.

[0030] System software framework: Ubuntu 20.04, PyTorch 1.12, Python 3.10 The specific implementation steps are as follows: S1: Model Input Data Processing DESDNet is based on the Swing Transformer as its backbone network, such as Figure 1 As shown, it includes four stages, from stage 1 to stage 4.

[0031] The overall network structure adopts a twin network structure, meaning that the visible light branch and the infrared branch use the same structure. When the network is used, it receives two video streams—one visible light and one infrared—as input, which are consecutive image sequence pairs.

[0032] In each loop, only one frame of visible light and one frame of infrared image search maps are processed. After processing, the tracking result is given. Then, the next loop begins, with the next pair of visible light and infrared search maps input for processing. This process is repeated cyclically.

[0033] For each pair of visible light and infrared image inputs, the image input template processing method is as follows: Given the search graph and template coordinates [x1, y1, w1, h1] of the visible light image, the template is extracted and fed into the visible light branch. Given the search graph and template coordinates [x2, y2, w2, h2] of the infrared image, the template is extracted and fed into the infrared branch. Then, the subsequent calculation process is carried out.

[0034] Unlike the search graph, the template image does not need to be processed every time; it is only updated when necessary, such as when the target changes significantly. The search graph, on the other hand, is processed every frame.

[0035] like Figure 1As shown, a pair of visible light and infrared images are input and then sent to their respective branches. In the original SwinTransformer structure, the Patch Partition module before the first stage converts the original input image size from H×W×3 to H / 4×W / 4×48 using a convolution with a 4×4 kernel, a stride of 4, and 48 channels. Infrared images are typically single-channel, i.e., H×W×1. To parallelize the operation, it is copied three times and concatenated along the channel direction to obtain H×W×3.

[0036] However, in DESDNet, the regular convolutional patch partitioning is replaced by two SPD-Conv (space-to-depth convolutions). The specific operation is as follows: The original input image has dimensions H×W×3, such as... Figure 2 As shown, after the first SPD-Conv, it is converted to H / 2×W / 2×12, and after the second SPD-Conv, it is converted to H / 4×W / 4×48.

[0037] The purpose of this replacement is that SPD-Conv is a lossless process compared to direct convolution, without discarding any information.

[0038] S2: Modality-Specific Information Extraction and Pixel-Level Fusion The full name of the PCFM module is Pixel-Level Cross-Modal Fusion Module (PCFM).

[0039] High-resolution shallow feature maps output from the first stage (stage 1) of the backbone network and The input is fed into the MSBN layer. The Linear embedding and Swin TransformerBlock in the first stage of the backbone network both adopt the standard methods in Swin Transformer, which are existing technologies in the industry and will not be elaborated here.

[0040] and The dimension is H / 4×W / 4×C. The number of channels C here varies depending on the backbone network model. For the Swin-Tiny model, C=96, and for larger models such as Swin-Base, C=128.

[0041] In PCFM, each mode is equipped with an independent adaptive Modality-Specific Batch Norm (MSBN) layer, enabling the visible light and infrared network branches to handle the differences in their respective modal distributions.

[0042] and After normalization by the MSBN layer, each modal quality distribution is obtained by passing it through a small convolutional network.

[0043] The steps to implement this small convolutional network are as follows: (1) Using 1×1 convolution, and Perform dimensionality reduction; (2) Then, the ReLU activation function is used, followed by a 1×1 convolution to compress the channels to 1 or 3. For color visible light, the input is usually RGB three channels, in which case the visible light branch needs to be compressed to 3 channels. For infrared images, it is usually a single channel, in which case the infrared branch needs to be compressed to 1 channel. This invention can also be applied to single-channel grayscale visible light images, in which case it needs to be compressed to 1 channel; (3) Finally, the sigmoid activation function is used to normalize the final output value to the [0,1] interval, resulting in a mask for the quality response distribution of each modality. The above process can be represented as:

[0044]

[0045] and These represent the effective information distribution weights for the visible light mode and the infrared mode, respectively, with dimensions H / 4×W / 4×1. After being mapped back to the original image size (H×W) by the upsampling module (UP), the fused image is obtained by element-wise multiplication with the original image and then element-wise addition. The above process can be represented as:

[0046] in, This represents an upsampling operation.

[0047] The upsampling operation can be performed using the following three methods: (1) Bilinear interpolation H×W can be obtained by direct interpolation of H / 4×W / 4. This method has low computational cost and is fast. However, the checkerboard effect is obvious.

[0048] (2) Subpixel convolution First, a 3×3 convolution is used to restore the feature map from H / 4×W / 4×1 to H / 4×W / 4×16. Then, pixel shuffle is performed, and 16 pixels corresponding to each feature point are stitched together to form a 4×4 image, replacing the original point. This process is repeated to replace H / 4×W / 4 points, resulting in the restored image H / 4×W / 4×1. This method is computationally faster and has learning capabilities.

[0049] (3) MLP-based upsampling of implicit neural representations To overcome the limitations of traditional discrete upsampling methods in recovering high-frequency spatial details, we propose a continuous mask generation module based on implicit neural representation for upsampling.

[0050] Unlike directly interpolating feature maps, we model mask generation as a coordinate-to-value mapping function. Specifically, we utilize Fourier position encoding to map spatial coordinates to a high-dimensional feature space and combine it with low-resolution semantic features, fitting a continuous mask distribution using a lightweight multilayer perceptron (MLP). This method not only achieves resolution-independent and flexible reconstruction but also effectively suppresses mesh artifacts, generating attention maps with sharper edges.

[0051] The entire upsampling process: given a low-resolution feature map Our goal is to reconstruct a high-resolution mask. For high-resolution masks arbitrary spatial coordinates (in Normalized to The mask value at this point It is predicted by a multilayer perceptron (MLP). Specific implementation method: Step 1: Positional Encoding and Feature Query. To enable the MLP to capture high-frequency details in the image, Fourier Positional Encoding is first applied to the spatial coordinates, as shown in the following equation. A high-resolution mask is then applied. Low- and medium-dimensional coordinates Mapping to a high-dimensional feature space:

[0052] in L represents the frequency layer number; γ(p) represents the feature vector after Fourier position encoding; p represents the spatial coordinates (x, y); L represents the frequency layer number; T represents the transpose sign.

[0053] Meanwhile, in order to inject semantic information into the image, a grid sampling mechanism is used to extract semantic information from the input low-resolution feature map. Querying high-resolution masks arbitrary spatial coordinates Local feature vectors at corresponding positions .

[0054] The specific method is as follows: (1) Calculate coordinates In low-resolution feature maps The floating-point coordinates (i, j) in the equation.

[0055] Where H represents the height of the feature map, W represents the width of the feature map, and r is the magnification factor, which is 4 in this embodiment.

[0056] (2) Obtaining high-resolution masks using bilinear interpolation arbitrary spatial coordinates In low-resolution feature maps Local feature vectors at corresponding positions in .

[0057]

[0058] Here, sample represents the bilinear interpolation operation.

[0059] Step 2: Map the Fourier position code obtained in Step 1 to high-dimensional coordinate space. and the local eigenvectors obtained through bilinear interpolation After stitching, the data is input into a multilayer perceptron for calculation. The final result is a high-resolution mask. arbitrary spatial coordinates The mask value at this point Repeat the above steps to complete the entire high-resolution mask. The construction of the upsampled image is completed.

[0060] S3: Multi-level Feature Mining and Feature-Level Fusion The full name of the FCMF module is Feature-Level Cross-Modal Fusion Module (FCFM). FCFM is applied to the last three stages of the backbone network: Stage 2, Stage 3, and Stage 4. The visible light feature vectors of each of the last three stages of the backbone network are then processed. and infrared feature vectors As input. The specific implementation steps are as follows: Step 1: Input the information into the Efficient Channel Attention (ECA) module of each modality to enhance the specific information of each modality.

[0061] ECA performs global average pooling (GAP) on the feature vectors of the three stages (Stage 2, Stage 3, and Stage 4) of each modality, reducing the dimensions from [H / 8,W / 8,2C], [H / 16,W / 16,4C], and [H / 32,W / 32,8C] to [1,1,2C], [1,1,4C], and [1,1,8C], and then uses one-dimensional convolution and a sigmoid activation function to generate channel weights.

[0062] Step 2: Use the generated channel weights and their respective original feature vectors. and Element-wise multiplication yields feature vectors enhanced for each modality. The feature vectors from Stage 2, Stage 3, and Stage 4 are then concatenated to obtain feature vectors that retain and enhance the modality-specific information of each individual mode. and .

[0063] Steps 1 and 2 can be represented as follows:

[0064]

[0065] in, This indicates a concatenation operation along the channel dimension. This represents element-wise multiplication. The channel weighting function based on ECA is calculated using the following formula:

[0066] in, Indicates global average pooling. Represents one-dimensional convolution. This represents the sigmoid activation function.

[0067] In addition to multi-level mining and enhancement of modality-specific information, FCFM also enhances modality-shared information. The processing of shared information in FCFM is based on the fused image in PCFM. and their respective modal-specific feature vectors and Image fusion In shared information, it serves as a high-resolution prior knowledge foundation, supplementing the middle layers of FCFM with fine-grained information via jump connections from the lower layers of PCFM. Each modality-specific feature vector... and It then acts as a modulation signal with characteristic features.

[0068] Based on this structure, this section designs a Gated Feature Recomposition Unit (GFRU).

[0069] The module implementation steps in GFRU are as follows: Step 1: Extract the fused image using depthwise separable convolution (DSG). Based on the feature information, a common feature basis is constructed. The number of channels in the DSG convolution kernel and the modality-specific feature vectors are used to construct this basis. and Consistency, enabling fused images The dimensional modal specific information feature vector after DSG and Consistency is necessary for subsequent processing.

[0070] Step 2: Use modality-specific feature vectors and Each modality is constructed with a spatial adaptive gating system to select common information from the basis. Specifically, the modality-specific feature vectors are used to... and After compressing the number of channels to 1 dimension using 1×1 convolutions, spatially gated feature maps are generated using 3×3 convolutions and a sigmoid activation function. The 3×3 convolutions have a stride of 1 and padding of 1 to ensure the feature map size remains unchanged.

[0071] Step 3: Multiply the individual spatial gating points and common basis elements together and then add them together to obtain the modality-shared feature vector. .

[0072] The implementation process of the above GFRU can be represented as follows:

[0073] in, This indicates a concatenation operation along the channel dimension. This represents element-wise multiplication. This represents a depthwise separable convolution.

[0074] By controlling the enhancement strength of common features to specific information through gating mechanisms, adaptive fusion can be accurately achieved. Compared to direct addition, although this method has a slightly higher computational complexity, it can effectively avoid the suppression of common modal information by the modality-specific information acquired by each modality-specific pathway.

[0075] S4: Multi-branch decision-level fusion and correction The MDFM module is a multi-branch decision-level fusion module.

[0076] The input to MDFM includes: a fine-grained pixel-level fused image obtained in PCFM. The feature vectors of visible and infrared mode-specific information were obtained in FCFM. and and modal shared feature vectors .

[0077] In MDFM, feature vectors based on visible light and infrared modal-specific information and and modal shared feature vectors Three prediction heads are constructed using a Region Proposal Network (RPN) template, outputting three sets of target locations. These three prediction heads represent the target locations tending for the visible light mode, infrared mode, and fused mode, respectively. The reliability of the outputs from the three modes needs to be evaluated, and the fused image generated by pixel-level fusion... It possesses the richest fine-grained information, enabling the most accurate decisions. Ultimately, decision-level fusion and target box correction are performed based on the evaluation results. MDFM fully considers modality specificity and similarity, achieving decision-level information fusion of "perception-evaluation-optimization" through the aforementioned methods.

[0078] The specific implementation steps of MDFM are as follows: 1) Construct a three-branch tracking head to obtain preliminary localization results: Will , and After segmenting the template and search map, a cross-correlation operation is performed to obtain three sets of classification feature maps and regression feature maps. Based on these feature maps, foreground and background are classified, and the proximity to the center of the ground truth is determined, generating an offset factor to finally obtain the three sets of target boxes. and the corresponding confidence level The above operations are standard operations of the RPN network. The RPN network belongs to the standard structure of single-object tracking algorithms in the field of image processing and is considered existing technology.

[0079] 2) Based on fused images Conduct a reliability assessment: Confidence levels of visible light mode, infrared mode, and fused mode outputs While it can reflect the reliability of results from each modality to some extent, it lacks a global perspective for judgment. In contrast, fused images... This preserves the original high-quality information of the visible and infrared modes, enabling the evaluation of the reliability of the results output from the three modes. Specifically, firstly, based on each candidate box... The location of the representation from the fused image Crop the target area Then a small convolutional network is used to... For encoding, this method uses a 3×3 convolution, ReLU activation function, and global average pooling. Finally, a fully connected layer and sigmoid activation function are used to generate a reliability score. The above process can be described as follows:

[0080] in, and These are learnable parameters. It is global average pooling.

[0081] 3) Decision-level fusion and target bounding box correction based on evaluation results: Confidence level for each modal output And the reliability score is obtained through evaluation. Multiply to obtain the corrected confidence level Select the result of the mode with the highest corrected confidence level. This serves as the final tracker output. To further improve the accuracy of target tracking, the final output... Based on We will further refine this process. The specific implementation method is as follows: First, using... Centered on Cut out an area of area . The output channels are compressed to 4 channels using a 3×3 convolutional kernel and a 1×1 convolution. This lightweight regression network outputs a pair of... Correction offset Finally, the offset and The results are summed to obtain the final value. The entire network structure is end-to-end, and a unified tracking loss is used for training.

[0082] This embodiment designs a DESDNet with an end-to-end architecture, where the RPN part is trained separately using the target tracking dataset. The loss function of the RPN tracking head consists of classification loss, regression loss, and centrality loss. Classification loss... Use cross-entropy loss. Regression loss. GIoU is used for calculation. The calculation method for GIoU is as follows:

[0083] Where A is the predicted bounding box, B is the ground truth bounding box, and C is the minimum bounding rectangle of A and B.

[0084] Regression loss The calculation method is as follows:

[0085] in and These are the predicted value and the actual value, respectively.

[0086] Centrality loss The binary cross-entropy loss is used for calculation, and the calculation method is as follows:

[0087] in, This represents the predicted value output by the centrality branch. The actual value represents the data set during annotation, and its calculation method is as follows:

[0088] in, , , and Indicates respectively The distance of the point from the left, right, top, and bottom bounding boxes of the real target.

[0089] After completing the training of the RPN tracking heads, freeze the three RPN tracking heads and train the rest of the network. Only use the final output bounding boxes to calculate the GIoU loss. The total loss function is shown in the following formula:

[0090] Among them, the loss balance hyperparameters are obtained by searching for parameters through the network. and .

[0091] During training, DESDNet was implemented using Python 3.8 and PyTorch 1.12.0. Training was conducted on a server with four RTX A6000 GPUs, each with 48GB of VRAM. The server's operating system was Ubuntu 22.04, the CPU was an Intel Xeon W9-3495X, and the RAM was 512GB. The batch size was set to 32 during training. The template and search region sizes were set to 128×128 and 256×256, respectively. First, three RPN tracking heads were trained on datasets such as GOT10K and OTB100. After training, the three RPN tracking heads were frozen, and the entire network was trained using the RGBT dataset.

[0092] The model was optimized during training using an adaptive moment estimation optimizer with decoupled weight decay (AdamW). Momentum and weight decay were set to 0.9 and 10, respectively. -4 The initial learning rate for this chapter is set to 10. -4 After training for 200 epochs, the learning rate was adjusted to 10. -5 The training data used included the following datasets: GTOT, RGBT210, RGBT234, and LasHeR.

[0093] The inference process is as follows: First, an initial template is extracted from the first frame of the visible light and infrared images; then, a square search region is cropped at the center of the template, and the side length of the cropped square region is equal to 1.5 times the length of the template diagonal; finally, the template and search map are scaled to the specified size of the model and then input into the network for inference.

[0094] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A visible-infrared target tracking method based on modal quality perception and multi-level interaction, characterized in that, include: Step S1: Input the template images of the visible light mode and the infrared mode and the search image into the dual-stream backbone network extractor respectively to obtain the shallow feature map of the first stage; Step S2: Input the shallow feature map into the pixel-level cross-modal fusion module. By equipping each modality with an independent modality-specific normalization layer for distribution difference processing, and using a lightweight convolutional network to generate a dynamic modality quality mask, the dynamic modality quality mask is multiplied pixel by pixel with the original image of the corresponding modality and weighted summed to obtain a pixel-level fusion enhancement template that retains fine-grained information. Step S3: Input the feature maps of the last three stages of the backbone network into the feature-level cross-modal fusion module, use the efficient channel attention mechanism to enhance the specific information of each modality, and use the gated feature recombination unit with the pixel-level fusion enhancement template as the common feature base, combine the modality-specific information to generate a spatial adaptive gated signal, and dynamically fuse to obtain the modality-shared feature vector. Step S4: The visible light-specific information, infrared-specific information, and modality-shared feature vector are input into the prediction heads of three parallel region candidate networks in the multi-branch decision-level fusion module, respectively, and the initial target bounding boxes and confidence scores corresponding to the modality tendencies are output. Based on the pixel-level fusion enhancement template, the reliability of the initial target bounding boxes is scored by a lightweight evaluation network, and the target bounding box with the highest corrected confidence score is selected for local fine-tuning regression to output the final target position.

2. The visible-infrared target tracking method based on modal quality perception and multi-level interaction according to claim 1, characterized in that, In step S2, the modality-specific normalization layer performs batch normalization on the first-stage output feature maps of the visible light branch and the infrared branch respectively to adapt to the differences in data distribution of different modalities.

3. The visible-infrared target tracking method based on modal quality perception and multi-level interaction according to claim 1, characterized in that, In step S2, the process of generating a dynamic modal quality mask using a lightweight convolutional network includes: using a 1×1 convolution to reduce the dimensionality of the feature map processed by MSBN, using the ReLU activation function, then using a 1×1 convolution to compress the channels to 1 or 3, and finally using the sigmoid activation function to normalize the output value to the [0,1] interval to obtain the quality response distribution mask of the corresponding modality.

4. The visible-infrared target tracking method based on modal quality perception and multi-level interaction according to claim 1, characterized in that, In step S2, the upsampling operation of upsampling the quality mask to the original image size is a continuous mask generation method based on implicit neural representation. Specifically, it involves using Fourier position encoding to map spatial coordinates to a high-dimensional feature space, combining local feature vectors in the low-resolution feature map, and fitting a continuous mask distribution through a multilayer perceptron.

5. The visible-infrared target tracking method based on modal quality perception and multi-level interaction according to claim 1, characterized in that, In step S3, the efficient channel attention mechanism is implemented as follows: global average pooling is performed on the feature maps of each stage, channel weights are generated by one-dimensional convolution and sigmoid activation function, and then multiplied with the elements of the original feature map to obtain the feature vectors with modality-specific enhancement.

6. The visible-infrared target tracking method based on modal quality perception and multi-level interaction according to claim 1, characterized in that, In step S3, the gated feature recombination unit is implemented as follows: common feature bases are extracted from pixel-level fusion enhancement templates using depthwise separable convolution; spatial gated feature maps are generated by using modality-specific information vectors through 1×1 and 3×3 convolutions; the spatial gated feature maps are multiplied element-wise with the common feature bases and then added element-wise to obtain modality-shared feature vectors.

7. The visible-infrared target tracking method based on modal quality perception and multi-level interaction according to claim 1, characterized in that, In step S4, the process of the lightweight evaluation network scoring the reliability of the initial target bounding box includes: cropping the target region from the pixel-level fusion enhancement template according to the position of each candidate bounding box, encoding it through 3×3 convolution, ReLU activation function and global average pooling, and then generating a reliability score through a fully connected layer and sigmoid activation function.

8. The visible-infrared target tracking method based on modal quality perception and multi-level interaction according to claim 1, characterized in that, In step S4, the local fine-tuning regression is implemented as follows: taking the initial target bounding box with the highest corrected confidence as the center, a region with a coverage area twice that of the original bounding box is cropped in the pixel-level fusion enhancement template, and the offset is output through a lightweight regression network with 3 layers of 3×3 convolution and 1×1 convolution. The offset is added to the initial target bounding box to obtain the final target position.

9. The visible-infrared target tracking method based on modal quality perception and multi-level interaction according to claim 1, characterized in that, The dual-stream backbone network extractor is a weight-shared twin network based on Swin-Transformer, and its first-stage block segmentation module uses two SPD-Conv operations instead of conventional convolution.

10. A visible-infrared target tracking system based on modal quality perception and multi-level interaction, used to implement the visible-infrared target tracking method based on modal quality perception and multi-level interaction as described in any one of claims 1-9, characterized in that, include: A pixel-level cross-modal fusion module is used to input the shallow feature map into the pixel-level cross-modal fusion module. It performs distribution difference processing by equipping each modality with an independent modality-specific normalization layer, and uses a lightweight convolutional network to generate a dynamic modality quality mask. The quality mask is multiplied pixel by pixel with the original image of the corresponding modality and weighted and added to obtain a pixel-level fusion enhancement template that retains fine-grained information. The feature-level cross-modal fusion module is used to input the feature maps of the last three stages of the backbone network into the feature-level cross-modal fusion module, enhance the specific information of each modality by using an efficient channel attention mechanism, and use the pixel-level fusion enhancement template as a common feature base by using a gated feature recombination unit to generate a spatial adaptive gated signal by combining modality-specific information, and dynamically fuse to obtain a modality-shared feature vector. The multi-branch decision-level fusion module is used to input visible light-specific information, infrared-specific information, and modality-shared feature vectors into the prediction heads of three parallel region candidate networks in the multi-branch decision-level fusion module, and output the initial target bounding box and confidence score corresponding to the modality tendency. Based on the pixel-level fusion enhancement template, the initial target bounding box is scored for reliability by a lightweight evaluation network, and the target bounding box with the highest corrected confidence score is selected for local fine-tuning regression to output the final target position.

Images