Infrared and visible image registration and fusion method based on multi-dimensional attention

Abstract

This invention discloses a multi-dimensional attention-based method for infrared and visible light image registration and fusion. First, an infrared and visible light image dataset is constructed and preprocessed to simulate the generation of the image dataset required for registration and fusion. The generated dataset is then divided into training and testing sets. Next, a registration and fusion network model for infrared and visible light images based on multi-dimensional attention is constructed. The constructed registration and fusion network model is trained using supervised training, and the model parameters are jointly optimized through multiple loss functions until the model converges. Finally, the trained registration and fusion network model is used to register and fuse pairs of infrared and visible light images from the testing set, outputting the final registered and fused image. This invention solves the problems of lack of modality specificity in infrared and visible light image feature extraction, insufficient registration feature calibration, and poor synergy between fusion and registration in existing technologies, achieving high-precision image registration and high-quality image fusion.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method for infrared and visible light image registration and fusion based on multidimensional attention. Background Technology

[0002] Infrared and visible light images are key image data sources for fields such as machine vision, security monitoring, remote sensing, and target detection, and they have complementary advantages in image features. Infrared images are based on the thermal radiation characteristics of targets and are not limited by lighting conditions. They can clearly identify the outline and position of targets in complex environments such as night, fog, and rain. However, their spatial resolution is low, texture details are lacking, and the visual effect of the image is relatively blurry. Visible light images rely on the reflected light of objects and have rich spatial texture, color, and detail information. The edges of targets and background textures are finely depicted, but they are easily affected by external environmental factors such as lighting, weather, and occlusion. The imaging effect is significantly reduced in low-light or complex environments.

[0003] To fully integrate the target thermal radiation features of infrared images with the spatial detail features of visible light images, and to obtain a fused image that combines target recognition and rich detail, infrared-visible light image registration and fusion technology has emerged. Image registration is a prerequisite for fusion, referring to the process of matching images of the same target or scene acquired from different modalities, perspectives, and times to the same spatial coordinate system; image fusion, on the other hand, extracts the effective features of each modality image based on registration and performs complementary fusion to generate a high-quality fused image.

[0004] Existing infrared and visible light image registration and fusion methods suffer from several technical shortcomings: First, the feature fusion module in the registration stage only considers feature calibration in a single dimension, making it difficult to simultaneously differentiate the importance of features in the channel dimension and mine the spatial correlation of features in the spatial dimension. Insufficient feature calibration leads to insufficient registration accuracy and poor deformation compensation. Second, the attention mechanism design in the fusion stage is simplistic, often employing only a single mechanism of channel attention or spatial attention. This fails to accurately focus on the complementary features of infrared and visible light images from multiple dimensions, including channels, space, and windows, resulting in insufficient targeting and effectiveness of feature fusion. Third, the DW module of registration and fusion has poor integration. The deformation features after registration do not deeply interact with the multi-level attention features of the fusion network, causing the fused image to neither retain the spatial alignment characteristics after registration nor fully integrate the core features of the two modalities. Ultimately, it is difficult to balance the target sharpness and background detail richness of the fused image.

[0005] Deep learning-based registration and fusion methods have become the mainstream research approach. Convolutional Neural Networks (CNNs) are widely used for registration feature extraction due to their local feature extraction capabilities, but they suffer from limited receptive fields and insufficient global feature capture. Transformer-type attention mechanisms can effectively capture global features, but they are less efficient in extracting local detail features. Current technologies have not yet organically combined dual-domain deformation correction feature calibration modules with multi-dimensional attention-enhanced fusion modules, failing to achieve a synergistic improvement in registration accuracy and fusion effect, and thus failing to meet the practical application requirements of infrared and visible light image registration and fusion in complex environments. Summary of the Invention

[0006] The purpose of this invention is to provide an infrared and visible light image registration and fusion method based on multidimensional attention, which solves the problems of lack of modality specificity in infrared and visible light image feature extraction, insufficient registration feature calibration, and poor synergy between fusion and registration in the prior art, thereby achieving high-precision image registration and high-quality image fusion.

[0007] The technical solution adopted in this invention is an infrared and visible light image registration and fusion method based on multidimensional attention, which is implemented according to the following steps: Step 1: Construct and preprocess infrared and visible light image datasets to simulate the generation of image datasets required for registration and fusion. Then, divide the generated datasets into training and test sets. Step 2: Construct a registration and fusion network model for infrared and visible light images based on multidimensional attention; Step 3: Train the constructed registration and fusion network model using supervised training. Optimize the model parameters jointly using multiple loss functions until the model converges. Step 4: Use the trained registration and fusion network model to register and fuse the infrared and visible light image pairs in the test set, and output the final registered and fused image.

[0008] The invention is further characterized in that, Step 1 is implemented in the following steps: Step 1.1: The infrared and visible light image dataset contains image pairs of various scenes, including nighttime, foggy days, indoors, and outdoors. The raw infrared and visible light image data is preprocessed, including image denoising, size normalization, and grayscale stretching, and all images are uniformly adjusted to a preset size. Data augmentation operations such as random cropping, rotation, and flipping are performed on the normalized images. All corresponding infrared and visible light images after processing are combined in pairs to form infrared-visible light image pairs. Step 1.2: Select infrared-visible image pairs from the preprocessed image data, set the preset size area at the center of the image pair as the test set, and the rest outside the center area as the training area; perform zero-filling processing on the test area, and randomly crop image blocks of preset size from the training area as the training set to ensure that the training set and the test set do not overlap.

[0009] The specific structure of the registration and fusion network model in step 2 is as follows: The fusion registration network model consists of a registration sub-network and a fusion sub-network. The output of the registration sub-network serves as the input of the fusion sub-network, achieving collaborative optimization of registration and fusion. The registration sub-network is an improved MPDRNet network, including an infrared encoder, a decoder, a visible light encoder, a deformation site prediction module, a dual-domain deformation correction module, and a spatial transformation module. Both the encoder and decoder are four-level cascaded multi-scale downsampling feature extraction structures. The overall model starts with the input single-channel image and sequentially goes through four levels of feature extraction and downsampling processing. Each level uses a 3×3 convolutional layer with a stride of 2, combined with a batch normalization layer and a LeakyReLU activation layer as the basic downsampling unit, achieving feature transformation that halves the resolution and doubles the number of channels successively. The first level consists only of basic convolutional units. The second, third, and fourth levels all connect residual blocks after the basic convolutional units. Each residual block consists of two 3×3 convolutional layers, one LeakyReLU activation function, and residual skip connections. The final output resolution is the same as the original. Figure 1 Four sets of multi-scale feature maps with channel numbers of 16, 32, 64, and 128 respectively (1 / 2, 1 / 4, 1 / 8, 1 / 16, 16, 32, 64, 128) were used throughout the process. The overall structure was a pyramid-shaped downsampling model, providing rich multi-scale features from shallow to deep and local to global for subsequent image registration tasks. Residual blocks were used to mitigate gradient vanishing. The dual-domain deformation correction module was a feature refinement structure enhanced by multi-branch feature fusion and channel-space dual attention mechanisms. Starting with the three sets of input feature maps, a 1×1 convolutional layer was first used to compress the channel dimension and perform preliminary feature transformation, resulting in three sets of dimensionality-reduced features. These were then fed into a 3×3 convolutional layer for spatial feature extraction and fusion. A Softmax activation function was then used to generate three sets of spatial weight maps. After fine-tuning the weights using a 1×1 convolutional layer, these weights were added element-wise with the original input feature maps to complete residual feature fusion. A ReLU activation function was then used to enhance the non-linear expression. Finally, the weighted features were multiplied element-wise with the original input feature maps to obtain the weighted fused features. The fused features are then sequentially fed into the channel attention module and the spatial attention module for dual enhancement. In the channel attention branch, the features are first extracted for global channel information through parallel max pooling and average pooling, and then the inter-channel dependencies are modeled through a shared multilayer perceptron. After fusion through a 1×1 convolutional layer, the channel attention weights are generated by the Sigmoid activation function and multiplied element-wise with the input features to complete channel dimension weighting. In the spatial attention branch, the features are extracted for spatial global information through parallel max pooling and average pooling, and the spatial position dependencies are modeled through a convolutional layer. The spatial attention weights are generated by the Sigmoid activation function and multiplied element-wise with the input features to complete spatial dimension weighting. Finally, after two attention enhancements, refined features are output. Overall, adaptive weighted fusion of multi-source features and channel-space dual-dimensional attention enhancement are achieved to improve the discriminative and expressive power of the features.

[0010] The registration subnetwork in step 2 is implemented as follows: Step 2.1: The infrared image and the visible light image are input into the infrared encoder and the visible light encoder, respectively. The two encoders extract features at four scales and output the following features in sequence: an 8-channel 256×256 multi-scale feature map enc3, a 16-channel 128×128 multi-scale feature map enc2, a 32-channel 64×64 multi-scale feature map enc1, and a 64-channel 32×32 multi-scale feature map enc0. Step 2.2: The two enc0 feature maps of infrared and visible light are concatenated with 128 channels and input into the bottleneck layer of the decoder. After passing through two ConvBnLeakyRelu2d layers, the number of channels is reduced to 64 to obtain the initial feature dec0 of the decoder. Step 2.3: Perform three-level upsampling on dec0, with each level of upsampling introducing a dual-domain deformation correction module; First-level upsampling After transposing and upsampling to 64×64 with 32 channels, the upsampled decoded output is obtained. The deformation field prediction module predicts the deformation field based on the deformation field. For infrared The features are spatially deformed to obtain the deformation features. Upsampled decoding output Deformation characteristics Visible light Feature stitching, input to the dual-domain deformation correction module, and after dual calibration by channel attention and spatial attention, output the final deformation features. dec1 The second and third level upsampling processes are the same as the first level, sequentially processing multi-scale features. , The features are deformed and calibrated to output the final deformation features. dec2 , ; Step 2.4: The deformation field prediction module processes the initial features of the decoder. dec0、 dec1 dec2, dec3 Predict the deformation field separately flow0-flow3 The deformation field phi_4 is obtained by deformation fusion and vector integration through the dual-domain deformation correction module; the spatial transformation module performs spatial deformation on the original infrared image according to phi_4 to obtain the infrared registration image I_ir_reg that is registered with the visible light image.

[0011] The dual-domain deformation correction module is a feature refinement structure that integrates multi-branch feature adaptive weighting and channel-space dual attention mechanisms. The network starts with three sets of input feature maps. First, it compresses the channel dimension and performs preliminary feature transformation through a 1×1 convolutional layer, resulting in three sets of dimensionality-reduced features. These are then fed into a 3×3 convolutional layer for spatial feature extraction and fusion. A Softmax activation function generates three sets of spatial weight maps, which are then refined through a 1×1 convolutional layer. The adjusted weights are first added element-wise with the original input feature map to achieve residual feature fusion, then enhanced with a ReLU activation function for non-linear expression, and finally multiplied element-wise with the original input feature map to obtain the weighted fused features. These fused features are then enhanced by both the channel attention module and the spatial attention module. In the channel attention branch, global channel information is extracted in parallel through max pooling and average pooling, and then further enhanced through shared multi-channel attention. The layer perceptron models inter-channel dependencies. After fusion through 1×1 convolutional layers, channel attention weights are generated by the Sigmoid activation function and multiplied element-wise with the input features to achieve weighted enhancement of the channel dimension. In the spatial attention branch, features are extracted using max pooling and average pooling in parallel to extract global spatial information. Spatial location dependencies are modeled through convolutional layers, and spatial attention weights are generated by the Sigmoid activation function. These weights are then multiplied element-wise with the input features to achieve weighted enhancement of the spatial dimension. Finally, after two attention enhancements, the network outputs refined features. Overall, it achieves adaptive weighted fusion of multi-source features and dual-dimensional attention enhancement of channels and space, effectively improving the discriminative and expressive power of the features. The specific operation is as follows: Feature stitching: The decoder upsampled features, deformed infrared encoded features, and visible light encoded features are stitched together along the channel dimension; Channel attention calibration: Channel features are extracted through adaptive average pooling and max pooling, channel attention weights are generated through convolutional layers, and channel weights are assigned to the concatenated features; Spatial attention calibration: Spatial features are extracted through depthwise separable convolution, and then... Activate the generated spatial attention weights to perform spatial dimension calibration on the channel-calibrated features; Residual fusion: The spatially calibrated features are residually fused with the original spliced ​​features to improve the feature representation capability and output the fused features after dual calibration.

[0012] The feature interaction module consists of a registration sub-network, a feature stitching module, a 1×1 convolutional compression module, and a fusion sub-network. The registration sub-network takes infrared and visible light images as input and outputs infrared registration image features, visible light original image features, and deformation field features. The feature stitching module stitches the three types of features in the channel dimension to obtain high-dimensional stitched features. The 1×1 convolutional compression module compresses the number of stitched feature channels to 64, generating the registration fusion feature F_inter. The fusion sub-network takes F_inter as input and realizes cross-module interaction and deep fusion of registration features and fusion features, completing the core fusion task and realizing cross-module interaction between registration features and fusion features.

[0013] The structure of the fusion subnetwork is as follows: The fusion subnetwork is an improvement The network, with its two-branch structure, handles the data separately. The infrared registration features and visible light features in the model are used. The core of the model is a multi-dimensional attention enhancement module consisting of three branches. The first branch directly feeds the input features into the channel attention module, models the channel dimension dependency, and generates channel attention weight features. The second branch directly feeds the input features into the spatial attention module, models the spatial dimension position dependency, and generates spatial attention weight features. The third branch first passes the input features through a 1×1 convolutional layer to complete channel dimension compression and feature transformation, then splits the transformed features into three sets of features: Q, K, and V. These features are then processed sequentially through dot product, scaling, positional bias self-attention calculation, and the Softmax activation function to generate a window self-attention weight matrix Awin. This matrix is ​​multiplied with the V feature to complete self-attention weighting, and then integrated through a 1×1 convolutional layer to obtain the window self-attention enhanced features. The output features of the three branches are fed into a multi-weight fusion module, which uses three sets of learnable weights (W1, W2, W3) to complete the weighted fusion of channel attention, spatial attention, and window self-attention features. The fused features are then finally processed by a 1×1 convolutional layer. The convolutional layer completes the adjustment of channel dimension and feature integration, and outputs refined feature f. Overall, it realizes the parallel enhancement and adaptive fusion of three attention mechanisms: channel, spatial and window self-attention, which effectively improves the global and local modeling capabilities of features. The fusion subnetwork also includes a head convolutional layer and a dual-branch feature extraction module. Starting with the input feature X, it first undergoes a 1×1 convolutional layer for unified channel transformation, then is fed into a parallel dual-branch for feature enhancement. Finally, it outputs refined features through multiple residual fusion: the left convolutional branch passes through two consecutive convolutional layers, using short inter-layer connections to add the input feature residuals to extract local spatial features; the right branch first passes through a normalization layer, then connects to a window self-attention module to model global dependencies within a local window, followed by another normalization and a multilayer perceptron for nonlinear feature transformation. The output of this branch is also added to its own input residual. After the right branch ends, it is added element-wise to the output of the left branch, then added to the original input feature X using residuals. Multiple residual connections supplement the original information, ultimately outputting refined features X∽. This structure integrates the local feature extraction capability of convolution with the long-range context modeling capability of window self-attention, using residual connections to ensure the complete transmission and enhancement of feature information.

[0014] The specific operations for fusing residual blocks and head convolutional layers in the sub-network are as follows: (1) The head convolutional layer maps the number of channels of both infrared registration features and visible light features to 64, thus obtaining infrared branch features. and visible light branching characteristics ; (2) The two-branch features are input into the convolution - The hybrid block first extracts local features through a convolutional layer, and then... Window attention Capture global features to achieve local-global feature fusion; (3) The hybrid block output feature input multidimensional attention enhancement module: spatial attention reduces the number of feature channels to 2 through channel pooling, generates spatial attention weights through 5×5 convolution, and multiplies them with the input features to achieve spatial detail enhancement; channel attention compresses the features to 1×1 through adaptive average pooling, generates channel attention weights through convolutional layers, and multiplies them with the input features to achieve channel weight allocation; window attention divides the features into 8×8 windows, and through... Multi-head self-attention within the computation window captures global contextual features, and finally, spatial attention is fused through feature fusion. Channel attention Window attention The output features are concatenated along the channel dimension, and the number of channels is restored to 64 through a 1×1 convolution to obtain the multidimensional attention-enhanced calibrated features. and characteristics ; (4) Features The feature F_vis_att is added and fused to obtain the result. The input consists of two residual blocks for feature refinement, and finally, a tail convolutional layer maps the number of channels from 64 to 1, outputting the final infrared-visible fused image. .

[0015] Step 3 is implemented in the following steps: 3.1 Loss Function Design: A joint loss function is adopted, the formula of which is:

[0016] Registration loss The MSE loss of the registered infrared and visible light images, and the fusion loss. This is a combination of SSIM loss and MSE loss.

[0017] 3.2 Training Hyperparameters: Based on the PyTorch 2.1.0 framework, trained with an NVIDIA GeForce RTX 4090D graphics card, Adam optimizer, initial learning rate of 0.0001, cosine annealing learning rate decay, batch size=2, training for 800 epochs; 3.3 Training process: Input the training set images into the model, perform forward propagation to obtain the registered and fused images, calculate the joint loss value, perform backpropagation to optimize all parameters of the model, and save the model weights with the best performance on the validation set during the training process.

[0018] The beneficial effects of this invention are: 1. Significantly improved registration accuracy: By introducing a dual-domain deformation correction module into the registration sub-network, dual feature calibration of channel attention and spatial attention is achieved. Channel attention distinguishes the importance of different feature channels, while the spatial attention calibration layer's depthwise separable convolution mines the spatial correlation of features, solving the problem of insufficient feature calibration in traditional registration modules. This effectively improves the spatial registration accuracy of infrared and visible light images and reduces deformation errors; 2. Enhanced complementarity of fused features: The fusion sub-network is designed with a multi-dimensional attention enhancement module, organically combining channel attention, spatial attention, and window attention mechanisms. It precisely focuses on infrared and visible light features from three dimensions: channel, space, and window. This enhances both the target thermal radiation features of infrared images and the detailed texture features of visible light images, while also capturing local features. 1. **Global context information enables deep complementary fusion of image features from two modalities, solving the problems of single attention mechanism and insufficient feature fusion in traditional fusion methods:** 2. **Deep integration of registration and fusion modules:** A feature interaction module is set up to fuse the infrared registration features, visible light features, and deformation field features of the registration sub-network and input them into the fusion sub-network. This enables deep cross-module interaction between registration and fusion features, ensuring that the fusion sub-network performs feature fusion based on spatial registration, avoiding spatial misalignment in the fused image, and achieving a synergistic improvement in registration accuracy and fusion effect. 3. **Efficiency and robustness of the network structure:** The encoder-decoder structure of the registration sub-network, combined with residual blocks, effectively alleviates the gradient vanishing problem in deep networks and improves feature extraction capabilities. The dual-branch feature extraction module of the fusion sub-network combines the local feature extraction efficiency of convolution and the Transformer... It leverages the advantages of global feature capture, balancing the efficiency and effectiveness of feature extraction; the introduction of data augmentation and learning rate decay strategies enhances the model's generalization ability and robustness, enabling it to adapt to the requirements of infrared and visible light image registration and fusion under different complex environments; 5. Excellent fused image quality: the final output fused image retains the target thermal radiation identification of infrared images in complex environments, while also incorporating the rich spatial texture and detail information of visible light images. The target outline is clear and the background details are rich, making it highly practical in real-world application scenarios such as security monitoring, target detection, and remote sensing. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the implementation of the infrared-visible light image registration and fusion method based on multidimensional attention according to the present invention. Figure 2 This is a diagram illustrating the overall framework of the registration and fusion network model of this invention. Figure 3 This is a structural framework diagram of the dual-domain deformation correction module of the present invention; Figure 4 This is a structural framework diagram of the multidimensional attention enhancement module of the present invention; Figure 5 This is the dual-branch feature extraction module of the present invention; Figure 6(a) shows the unregistered infrared image from the experiment of this invention; Figure 6(b) shows the infrared image of the experimental registration of the present invention; Figure 6(c) shows the visible infrared image of the experiment of this invention; Figure 6(d) shows the image after registration and fusion in the experiment of this invention; Detailed Implementation The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0020] The present invention is a multi-dimensional attention-based infrared and visible light image registration and fusion method. The core of this method is to embed a dual-domain deformation correction module into the registration sub-network to achieve high-precision spatial registration, introduce a multi-dimensional attention enhancement module into the fusion sub-network to achieve deep feature fusion, and achieve organic connection between the registration and fusion modules through a feature interaction module. Finally, high-precision registration and high-quality fusion of infrared and visible light images are achieved. The flowchart is shown in Figure 1.

[0021] This invention relates to an infrared and visible light image registration and fusion method based on multidimensional attention, the flowchart of which is shown below. Figure 1 As shown, please follow these steps: Step 1: Construct and preprocess infrared and visible light image datasets to simulate the generation of image datasets required for registration and fusion. Then, divide the generated datasets into training and test sets. Step 1 is implemented in the following steps: Step 1.1: The infrared and visible light image dataset contains image pairs of various scenes, including nighttime, foggy days, indoors, and outdoors. The raw infrared and visible light image data is preprocessed, including image denoising, size normalization, and grayscale stretching, and all images are uniformly adjusted to a preset size. Data augmentation operations such as random cropping, rotation, and flipping are performed on the normalized images. All corresponding infrared and visible light images after processing are combined in pairs to form infrared-visible light image pairs. Step 1.2: Select infrared-visible image pairs from the preprocessed image data, set the preset size area at the center of the image pair as the test set, and the rest outside the center area as the training area; perform zero-filling processing on the test area, and randomly crop image blocks of preset size from the training area as the training set to ensure that the training set and the test set do not overlap.

[0022] Step 2: Construct a registration and fusion network model for infrared and visible light images based on multidimensional attention; the model includes a registration sub-network, a feature interaction module, and a fusion sub-network.

[0023] Combination Figure 2The specific structure of the registration and fusion network model in step 2 is as follows: The overall framework of the fusion registration network model is as follows: Figure 3 As shown, the system consists of a registration subnetwork and a fusion subnetwork. The output of the registration subnetwork serves as the input of the fusion subnetwork, achieving collaborative optimization of registration and fusion. The registration subnetwork is an improved MPDRNet network, including an infrared encoder, a decoder, a visible light encoder, a deformation site prediction module, a dual-domain deformation correction module, and a spatial transformation module. Both the encoder and decoder are four-level cascaded multi-scale downsampling feature extraction structures. The system starts with the input single-channel image and sequentially goes through four levels of feature extraction and downsampling processing. Each level uses a 3×3 convolutional layer with a stride of 2, combined with a batch normalization layer and a LeakyReLU activation layer as the basic downsampling unit, achieving feature transformation that halves the resolution and doubles the number of channels successively. The first level consists only of basic convolutional units. The second, third, and fourth levels all connect residual blocks after the basic convolutional units. Each residual block consists of two 3×3 convolutional layers, one LeakyReLU activation function, and residual skip connections. The final output resolution is the same as the original resolution. Figure 1 Four sets of multi-scale feature maps with channel numbers of 1 / 2, 1 / 4, 1 / 8, and 1 / 16, and channel numbers of 16, 32, 64, and 128 respectively, are used throughout the network. The network employs a pyramid-shaped downsampling structure to provide rich multi-scale features from shallow to deep and from local to global for subsequent image registration tasks. Residual blocks are used to mitigate gradient vanishing. The dual-domain deformation correction module structure is shown in Figure 3. This network is a feature refinement structure enhanced by multi-branch feature fusion and channel-space dual attention mechanisms. Starting with the three sets of input feature maps, a 1×1 convolutional layer is used to compress the channel dimension and perform preliminary feature transformation, resulting in three sets of dimensionality-reduced features. These are then fed into a 3×3 convolutional layer for spatial feature extraction and fusion. A Softmax activation function is then used to generate three sets of spatial weight maps. After fine-tuning the weights through a 1×1 convolutional layer, these weights are added element-wise with the original input feature maps to complete residual feature fusion, followed by ReLU. The activation function enhances the nonlinear expression, and finally multiplies it element-wise with the original input feature map to obtain the weighted fused features; The fused features are then sequentially fed into the channel attention module and the spatial attention module for dual enhancement. In the channel attention branch, the features are first subjected to max pooling and average pooling in parallel to extract global channel information. Then, the inter-channel dependencies are modeled through a shared multilayer perceptron (Shared MLP). After fusion through a 1×1 convolutional layer, the channel attention weights are generated by the sigmoid activation function and multiplied element-wise with the input features to complete the channel dimension weighting. In the spatial attention branch, the features are subjected to max pooling and average pooling in parallel to extract global spatial information. The spatial position dependencies are modeled through a convolutional layer. The spatial attention weights are generated by the sigmoid activation function and multiplied element-wise with the input features to complete the spatial dimension weighting. Finally, after two attention enhancements, refined features are output. Overall, adaptive weighted fusion of multi-source features and channel-space dual-dimensional attention enhancement are achieved to improve the discriminative and expressive power of the features.

[0024] The registration subnetwork in step 2 is implemented as follows: Step 2.1: The infrared image and the visible light image are input into the infrared encoder and the visible light encoder, respectively. The two encoders extract features at four scales and output the following features in sequence: an 8-channel 256×256 multi-scale feature map enc3, a 16-channel 128×128 multi-scale feature map enc2, a 32-channel 64×64 multi-scale feature map enc1, and a 64-channel 32×32 multi-scale feature map enc0. Step 2.2: The two enc0 feature maps of infrared and visible light are concatenated with 128 channels and input into the bottleneck layer of the decoder. After passing through two ConvBnLeakyRelu2d layers, the number of channels is reduced to 64 to obtain the initial feature dec0 of the decoder. Step 2.3: Perform three-level upsampling on dec0, with each level of upsampling introducing a dual-domain deformation correction module; First-level upsampling After transposing and upsampling to 64×64 with 32 channels, the upsampled decoded output is obtained. The deformation field prediction module predicts the deformation field based on the deformation field. For infrared The features are spatially deformed to obtain the deformation features. Upsampled decoding output Deformation characteristics Visible light Feature stitching, input to the dual-domain deformation correction module, and after dual calibration by channel attention and spatial attention, output the final deformation features. dec1 The second and third level upsampling processes are the same as the first level, sequentially processing multi-scale features. , The features are deformed and calibrated to output the final deformation features. dec2 , ; Step 2.4: The deformation field prediction module processes the initial features of the decoder. dec0、 dec1 dec2, dec3 Predict the deformation field separately flow0-flow3 The deformation field phi_4 is obtained by deformation fusion and vector integration through the dual-domain deformation correction module; the spatial transformation module performs spatial deformation on the original infrared image according to phi_4 to obtain the infrared registration image I_ir_reg that is registered with the visible light image.

[0025] The dual-domain deformation correction module is a feature refinement structure that integrates multi-branch feature adaptive weighting and channel-space dual attention mechanisms. The network starts with three sets of input feature maps. First, it compresses the channel dimension and performs preliminary feature transformation through a 1×1 convolutional layer, resulting in three sets of dimensionality-reduced features. These are then fed into a 3×3 convolutional layer for spatial feature extraction and fusion. A Softmax activation function generates three sets of spatial weight maps, which are then refined through a 1×1 convolutional layer. The adjusted weights are first added element-wise with the original input feature map to achieve residual feature fusion, then enhanced with a ReLU activation function for non-linear expression, and finally multiplied element-wise with the original input feature map to obtain the weighted fused features. These fused features are then enhanced by both the channel attention module and the spatial attention module. In the channel attention branch, global channel information is extracted in parallel through max pooling and average pooling, and then further enhanced by a shared multilayer perceptron (SMP). The MLP (Multi-Level Processing) model models inter-channel dependencies, fuses them through 1×1 convolutional layers, and generates channel attention weights using the Sigmoid activation function. These weights are then multiplied element-wise with the input features to achieve weighted enhancement of the channel dimension. In the spatial attention branch, features are extracted using max pooling and average pooling in parallel to extract global spatial information. Spatial location dependencies are modeled through convolutional layers, and spatial attention weights are generated using the Sigmoid activation function. These weights are then multiplied element-wise with the input features to achieve weighted enhancement of the spatial dimension. Finally, after two attention enhancements, the network outputs refined features. Overall, it achieves adaptive weighted fusion of multi-source features and dual-dimensional attention enhancement of channels and space, effectively improving the discriminative and expressive power of the features. The specific operation is as follows: Feature stitching: The decoder upsampled features, deformed infrared encoded features, and visible light encoded features are stitched together along the channel dimension; Channel attention calibration: Channel features are extracted through adaptive average pooling and max pooling, channel attention weights are generated through convolutional layers, and channel weights are assigned to the concatenated features; Spatial attention calibration: Spatial features are extracted through depthwise separable convolution, and then... Activate the generated spatial attention weights to perform spatial dimension calibration on the channel-calibrated features; Residual fusion: The spatially calibrated features are residually fused with the original spliced ​​features to improve the feature representation capability and output the fused features after dual calibration.

[0026] The feature interaction module consists of a registration sub-network, a feature stitching module, a 1×1 convolutional compression module, and a fusion sub-network. The registration sub-network takes infrared and visible light images as input and outputs infrared registration image features, visible light original image features, and deformation field features. The feature stitching module stitches the three types of features in the channel dimension to obtain high-dimensional stitched features. The 1×1 convolutional compression module compresses the number of stitched feature channels to 64, generating the registration fusion feature F_inter. The fusion sub-network takes F_inter as input and realizes cross-module interaction and deep fusion of registration features and fusion features, completing the core fusion task and realizing cross-module interaction between registration features and fusion features.

[0027] The structure of the fusion subnetwork is as follows: The fusion subnetwork is an improvement The network, with its two-branch structure, handles the data separately. The infrared registration features and visible light features in the network are used, with the core being a multi-dimensional attention enhancement module (structure shown in Figure 4). This network consists of three branches: the first branch directly feeds the input features into the channel attention module, models the channel dimension dependency, and generates channel attention weight features; the second branch directly feeds the input features into the spatial attention module, models the spatial dimension position dependency, and generates spatial attention weight features; the third branch first passes the input features through a 1×1 convolutional layer to complete channel dimension compression and feature transformation, then splits the transformed features into three groups of features: Q, K, and V. These are then processed sequentially through dot product, scaling, positional bias self-attention calculation, and the Softmax activation function to generate a window self-attention weight matrix Awin, which is multiplied with the V feature to complete self-attention weighting, and then integrated through a 1×1 convolutional layer to obtain the window self-attention enhanced features; the output features of the three branches are jointly fed into a multi-weight fusion module, which uses three sets of learnable weights W1, W2, and W3 to complete the weighted fusion of channel attention, spatial attention, and window self-attention features. The fused features are then finally processed by a 1×1 convolutional layer. The convolutional layer completes the adjustment of channel dimension and feature integration, and outputs refined feature f. Overall, it realizes the parallel enhancement and adaptive fusion of three attention mechanisms: channel, spatial and window self-attention, which effectively improves the global and local modeling capabilities of features. The fusion subnetwork also includes a head convolutional layer and a dual-branch feature extraction module, such as... Figure 5As shown, this is a residual feature enhancement structure with parallel fusion of two branches. Starting with the input feature X, it first undergoes a 1×1 convolutional layer for unified channel transformation, then is fed into parallel two branches for feature enhancement. Finally, it outputs refined features through multiple residual fusion: the left convolutional branch passes through two consecutive convolutional layers, using short inter-layer connections to add to the input feature residuals to extract local spatial features; the right branch first passes through a normalization layer, then connects to a window self-attention module to model global dependencies within a local window, followed by another normalization and multilayer perceptron for nonlinear feature transformation. The output of this branch is also added to its own input residual. After the right branch ends, it is added element-wise to the output of the left branch, then added to the original input feature X using residuals. Multiple residual connections supplement the original information, finally outputting refined features X∽. This structure integrates the local feature extraction capability of convolution with the long-range context modeling capability of window self-attention, using residual connections to ensure the complete transmission and enhancement of feature information.

[0028] The specific operations for fusing residual blocks and head convolutional layers in the sub-network are as follows: (1) The head convolutional layer maps the number of channels of both infrared registration features and visible light features to 64, thus obtaining infrared branch features. and visible light branching characteristics ; (2) The two-branch features are input into the convolution - The hybrid block first extracts local features through a convolutional layer, and then... Window attention Capture global features to achieve local-global feature fusion; (3) The hybrid block output feature input multidimensional attention enhancement module: Spatial attention reduces the number of feature channels to 2 through channel pooling (max pooling and average pooling), generates spatial attention weights through 5×5 convolution, and multiplies them with the input features to achieve spatial detail enhancement; Channel attention compresses the features to 1×1 through adaptive average pooling, generates channel attention weights through convolutional layers, and multiplies them with the input features to achieve channel weight allocation; Window attention divides the features into 8×8 windows, and through... Multi-head self-attention within the computation window captures global contextual features, and finally, spatial attention is fused through feature fusion. Channel attention Window attention The output features are concatenated along the channel dimension, and the number of channels is restored to 64 through a 1×1 convolution to obtain the multidimensional attention-enhanced calibrated features. and characteristics ; (4) Features The feature F_vis_att is added and fused to obtain the result. The input consists of two residual blocks for feature refinement, and finally, a tail convolutional layer maps the number of channels from 64 to 1, outputting the final infrared-visible fused image. .

[0029] Step 3: Train the constructed registration and fusion network model using supervised training. Optimize the model parameters jointly using multiple loss functions until the model converges. Step 3 is implemented in the following steps: 3.1 Loss Function Design: A joint loss function is adopted, the formula of which is:

[0030] Registration loss The MSE loss of the registered infrared and visible light images, and the fusion loss. This is a combination of SSIM loss and MSE loss.

[0031] 3.2 Training Hyperparameters: Based on the PyTorch 2.1.0 framework, trained with an NVIDIA GeForce RTX 4090D graphics card, Adam optimizer, initial learning rate of 0.0001, cosine annealing learning rate decay, batch size=2, training for 800 epochs; 3.3 Training process: Input the training set images into the model, perform forward propagation to obtain the registered and fused images, calculate the joint loss value, perform backpropagation to optimize all parameters of the model, and save the model weights with the best performance on the validation set during the training process.

[0032] Step 4: Use the trained registration and fusion network model to register and fuse the infrared and visible light image pairs in the test set, and output the final registered and fused image.

[0033] Example 1 This invention relates to an infrared and visible light image registration and fusion method based on multidimensional attention, the flowchart of which is shown below. Figure 1 As shown, please follow these steps: Step 1: Construct and preprocess infrared and visible light image datasets to simulate the generation of image datasets required for registration and fusion. Then, divide the generated datasets into training and test sets. Step 2: Construct a registration and fusion network model for infrared and visible light images based on multidimensional attention; the model includes a registration sub-network, a feature interaction module, and a fusion sub-network.

[0034] Step 3: Train the constructed registration and fusion network model using supervised training. Optimize the model parameters jointly using multiple loss functions until the model converges. Step 4: Use the trained registration and fusion network model to register and fuse the infrared and visible light image pairs in the test set, and output the final registered and fused image.

[0035] Example 2 This invention relates to an infrared and visible light image registration and fusion method based on multidimensional attention, the flowchart of which is shown below. Figure 1 As shown, please follow these steps: Step 1: Construct and preprocess infrared and visible light image datasets to simulate the generation of image datasets required for registration and fusion. Then, divide the generated datasets into training and test sets. Step 1 is implemented in the following steps: Step 1.1: The infrared and visible light image dataset contains image pairs of various scenes, including nighttime, foggy days, indoors, and outdoors. The raw infrared and visible light image data is preprocessed, including image denoising, size normalization, and grayscale stretching, and all images are uniformly adjusted to a preset size. Data augmentation operations such as random cropping, rotation, and flipping are performed on the normalized images. All corresponding infrared and visible light images after processing are combined in pairs to form infrared-visible light image pairs. Step 1.2: Select infrared-visible image pairs from the preprocessed image data, set the preset size area at the center of the image pair as the test set, and the rest outside the center area as the training area; perform zero-filling processing on the test area, and randomly crop image blocks of preset size from the training area as the training set to ensure that the training set and the test set do not overlap.

[0036] Step 2: Construct a registration and fusion network model for infrared and visible light images based on multidimensional attention; the model includes a registration sub-network, a feature interaction module, and a fusion sub-network.

[0037] The specific structure of the registration and fusion network model in step 2 is as follows: The overall framework of the fusion registration network model is as follows: Figure 3As shown, the system consists of a registration subnetwork and a fusion subnetwork. The output of the registration subnetwork serves as the input of the fusion subnetwork, achieving collaborative optimization of registration and fusion. The registration subnetwork is an improved MPDRNet network, including an infrared encoder, a decoder, a visible light encoder, a deformation site prediction module, a dual-domain deformation correction module, and a spatial transformation module. Both the encoder and decoder are four-level cascaded multi-scale downsampling feature extraction structures. The system starts with the input single-channel image and sequentially goes through four levels of feature extraction and downsampling processing. Each level uses a 3×3 convolutional layer with a stride of 2, combined with a batch normalization layer and a LeakyReLU activation layer as the basic downsampling unit, achieving feature transformation that halves the resolution and doubles the number of channels successively. The first level consists only of basic convolutional units. The second, third, and fourth levels all connect residual blocks after the basic convolutional units. Each residual block consists of two 3×3 convolutional layers, one LeakyReLU activation function, and residual skip connections. The final output resolution is the same as the original resolution. Figure 1 Four sets of multi-scale feature maps with channel numbers of 1 / 2, 1 / 4, 1 / 8, and 1 / 16, and channel numbers of 16, 32, 64, and 128 respectively, are used throughout the network. The network employs a pyramid-shaped downsampling structure to provide rich multi-scale features from shallow to deep and from local to global for subsequent image registration tasks. Residual blocks are used to mitigate gradient vanishing. The dual-domain deformation correction module structure is shown in Figure 3. This network is a feature refinement structure enhanced by multi-branch feature fusion and channel-space dual attention mechanisms. Starting with the three sets of input feature maps, a 1×1 convolutional layer is used to compress the channel dimension and perform preliminary feature transformation, resulting in three sets of dimensionality-reduced features. These are then fed into a 3×3 convolutional layer for spatial feature extraction and fusion. A Softmax activation function is then used to generate three sets of spatial weight maps. After fine-tuning the weights through a 1×1 convolutional layer, these weights are added element-wise with the original input feature maps to complete residual feature fusion, followed by ReLU. The activation function enhances the nonlinear expression, and finally multiplies it element-wise with the original input feature map to obtain the weighted fused features; Combination Figure 4The fused features are then sequentially fed into the channel attention module and the spatial attention module for dual enhancement. In the channel attention branch, the features are first extracted for global channel information through max pooling and average pooling in parallel. Then, the inter-channel dependencies are modeled through a shared multilayer perceptron (Shared MLP). After fusion through a 1×1 convolutional layer, the channel attention weights are generated by the sigmoid activation function and multiplied element-wise with the input features to complete the channel dimension weighting. In the spatial attention branch, the features are extracted for spatial global information through max pooling and average pooling in parallel. The spatial position dependencies are modeled through a convolutional layer. The spatial attention weights are generated by the sigmoid activation function and multiplied element-wise with the input features to complete the spatial dimension weighting. Finally, after two attention enhancements, refined features are output. Overall, adaptive weighted fusion of multi-source features and channel-space dual-dimensional attention enhancement are achieved to improve the discriminative and expressive power of the features.

[0038] Example 3 The registration subnetwork in step 2 is implemented as follows: Step 2.1: The infrared image and the visible light image are input into the infrared encoder and the visible light encoder, respectively. The two encoders extract features at four scales and output the following features in sequence: an 8-channel 256×256 multi-scale feature map enc3, a 16-channel 128×128 multi-scale feature map enc2, a 32-channel 64×64 multi-scale feature map enc1, and a 64-channel 32×32 multi-scale feature map enc0. Step 2.2: The two enc0 feature maps of infrared and visible light are concatenated with 128 channels and input into the bottleneck layer of the decoder. After passing through two ConvBnLeakyRelu2d layers, the number of channels is reduced to 64 to obtain the initial feature dec0 of the decoder. Step 2.3: Perform three-level upsampling on dec0, with each level of upsampling introducing a dual-domain deformation correction module; First-level upsampling After transposing and upsampling to 64×64 with 32 channels, the upsampled decoded output is obtained. The deformation field prediction module predicts the deformation field based on the deformation field. For infrared The features are spatially deformed to obtain the deformation features. Upsampled decoding output Deformation characteristics Visible light Feature stitching, input to the dual-domain deformation correction module, and after dual calibration by channel attention and spatial attention, output the final deformation features. dec1 The second and third level upsampling processes are the same as the first level, sequentially processing multi-scale features. , The features are deformed and calibrated to output the final deformation features. dec2 , ; Step 2.4: The deformation field prediction module processes the initial features of the decoder. dec0、 dec1 dec2, dec3 Predict the deformation field separately flow0-flow3 The deformation field phi_4 is obtained by deformation fusion and vector integration through the dual-domain deformation correction module; the spatial transformation module performs spatial deformation on the original infrared image according to phi_4 to obtain the infrared registration image I_ir_reg that is registered with the visible light image.

[0039] Step 3: Train the constructed registration and fusion network model using supervised training. Optimize the model parameters jointly using multiple loss functions until the model converges. Step 4: Use the trained registration and fusion network model to register and fuse the infrared and visible light image pairs in the test set, and output the final registered and fused image.

[0040] Example 4 This invention relates to an infrared and visible light image registration and fusion method based on multidimensional attention, the flowchart of which is shown below. Figure 1 As shown, please follow these steps: Step 1: Construct and preprocess infrared and visible light image datasets to simulate the generation of image datasets required for registration and fusion. Then, divide the generated datasets into training and test sets. Step 1 is implemented in the following steps: Step 1.1: The infrared and visible light image dataset contains image pairs of various scenes, including nighttime, foggy days, indoors, and outdoors. The raw infrared and visible light image data is preprocessed, including image denoising, size normalization, and grayscale stretching, and all images are uniformly adjusted to a preset size. Data augmentation operations such as random cropping, rotation, and flipping are performed on the normalized images. All corresponding infrared and visible light images after processing are combined in pairs to form infrared-visible light image pairs. Step 1.2: Select infrared-visible image pairs from the preprocessed image data, set the preset size area at the center of the image pair as the test set, and the rest outside the center area as the training area; perform zero-filling processing on the test area, and randomly crop image blocks of preset size from the training area as the training set to ensure that the training set and the test set do not overlap.

[0041] Step 2: Construct a registration and fusion network model for infrared and visible light images based on multidimensional attention; the model includes a registration sub-network, a feature interaction module, and a fusion sub-network.

[0042] The specific structure of the registration and fusion network model in step 2 is as follows: The overall framework of the fusion registration network model is as follows: Figure 3 As shown, the system consists of a registration subnetwork and a fusion subnetwork. The output of the registration subnetwork serves as the input of the fusion subnetwork, achieving collaborative optimization of registration and fusion. The registration subnetwork is an improved MPDRNet network, including an infrared encoder, a decoder, a visible light encoder, a deformation site prediction module, a dual-domain deformation correction module, and a spatial transformation module. Both the encoder and decoder are four-level cascaded multi-scale downsampling feature extraction structures. The system starts with the input single-channel image and sequentially goes through four levels of feature extraction and downsampling processing. Each level uses a 3×3 convolutional layer with a stride of 2, combined with a batch normalization layer and a LeakyReLU activation layer as the basic downsampling unit, achieving feature transformation that halves the resolution and doubles the number of channels successively. The first level consists only of basic convolutional units. The second, third, and fourth levels all connect residual blocks after the basic convolutional units. Each residual block consists of two 3×3 convolutional layers, one LeakyReLU activation function, and residual skip connections. The final output resolution is the same as the original resolution. Figure 1 Four sets of multi-scale feature maps with channel numbers of 1 / 2, 1 / 4, 1 / 8, and 1 / 16, and channel numbers of 16, 32, 64, and 128 respectively, are used throughout the network. The network employs a pyramid-shaped downsampling structure to provide rich multi-scale features from shallow to deep and from local to global for subsequent image registration tasks. Residual blocks are used to mitigate gradient vanishing. The dual-domain deformation correction module structure is shown in Figure 3. This network is a feature refinement structure enhanced by multi-branch feature fusion and channel-space dual attention mechanisms. Starting with the three sets of input feature maps, a 1×1 convolutional layer is used to compress the channel dimension and perform preliminary feature transformation, resulting in three sets of dimensionality-reduced features. These are then fed into a 3×3 convolutional layer for spatial feature extraction and fusion. A Softmax activation function is then used to generate three sets of spatial weight maps. After fine-tuning the weights through a 1×1 convolutional layer, these weights are added element-wise with the original input feature maps to complete residual feature fusion, followed by ReLU. The activation function enhances the nonlinear expression, and finally multiplies it element-wise with the original input feature map to obtain the weighted fused features; The fused features are then sequentially fed into the channel attention module and the spatial attention module for dual enhancement. In the channel attention branch, the features are first subjected to max pooling and average pooling in parallel to extract global channel information. Then, the inter-channel dependencies are modeled through a shared multilayer perceptron (Shared MLP). After fusion through a 1×1 convolutional layer, the channel attention weights are generated by the sigmoid activation function and multiplied element-wise with the input features to complete the channel dimension weighting. In the spatial attention branch, the features are subjected to max pooling and average pooling in parallel to extract global spatial information. The spatial position dependencies are modeled through a convolutional layer. The spatial attention weights are generated by the sigmoid activation function and multiplied element-wise with the input features to complete the spatial dimension weighting. Finally, after two attention enhancements, refined features are output. Overall, adaptive weighted fusion of multi-source features and channel-space dual-dimensional attention enhancement are achieved to improve the discriminative and expressive power of the features.

[0043] The registration subnetwork in step 2 is implemented as follows: Step 2.1: The infrared image and the visible light image are input into the infrared encoder and the visible light encoder, respectively. The two encoders extract features at four scales and output the following features in sequence: an 8-channel 256×256 multi-scale feature map enc3, a 16-channel 128×128 multi-scale feature map enc2, a 32-channel 64×64 multi-scale feature map enc1, and a 64-channel 32×32 multi-scale feature map enc0. Step 2.2: The two enc0 feature maps of infrared and visible light are concatenated with 128 channels and input into the bottleneck layer of the decoder. After passing through two ConvBnLeakyRelu2d layers, the number of channels is reduced to 64 to obtain the initial feature dec0 of the decoder. Step 2.3: Perform three-level upsampling on dec0, with each level of upsampling introducing a dual-domain deformation correction module; First-level upsampling After transposing and upsampling to 64×64 with 32 channels, the upsampled decoded output is obtained. The deformation field prediction module predicts the deformation field based on the deformation field. For infrared The features are spatially deformed to obtain the deformation features. Upsampled decoding output Deformation characteristics Visible light Feature stitching, input to the dual-domain deformation correction module, and after dual calibration by channel attention and spatial attention, output the final deformation features. dec1 The second and third level upsampling processes are the same as the first level, sequentially processing multi-scale features. , The features are deformed and calibrated to output the final deformation features. dec2 , ; Step 2.4: The deformation field prediction module processes the initial features of the decoder. dec0、 dec1 dec2, dec3 Predict the deformation field separately flow0-flow3 The deformation field phi_4 is obtained by deformation fusion and vector integration through the dual-domain deformation correction module; the spatial transformation module performs spatial deformation on the original infrared image according to phi_4 to obtain the infrared registration image I_ir_reg that is registered with the visible light image.

[0044] The dual-domain deformation correction module is a feature refinement structure that integrates multi-branch feature adaptive weighting and channel-space dual attention mechanisms. The network starts with three sets of input feature maps. First, it compresses the channel dimension and performs preliminary feature transformation through a 1×1 convolutional layer, resulting in three sets of dimensionality-reduced features. These are then fed into a 3×3 convolutional layer for spatial feature extraction and fusion. A Softmax activation function generates three sets of spatial weight maps, which are then refined through a 1×1 convolutional layer. The adjusted weights are first added element-wise with the original input feature map to achieve residual feature fusion, then enhanced with a ReLU activation function for non-linear expression, and finally multiplied element-wise with the original input feature map to obtain the weighted fused features. These fused features are then enhanced by both the channel attention module and the spatial attention module. In the channel attention branch, global channel information is extracted in parallel through max pooling and average pooling, and then further enhanced by a shared multilayer perceptron (SMP). The MLP (Multi-Level Processing) model models inter-channel dependencies, fuses them through 1×1 convolutional layers, and generates channel attention weights using the Sigmoid activation function. These weights are then multiplied element-wise with the input features to achieve weighted enhancement of the channel dimension. In the spatial attention branch, features are extracted using max pooling and average pooling in parallel to extract global spatial information. Spatial location dependencies are modeled through convolutional layers, and spatial attention weights are generated using the Sigmoid activation function. These weights are then multiplied element-wise with the input features to achieve weighted enhancement of the spatial dimension. Finally, after two attention enhancements, the network outputs refined features. Overall, it achieves adaptive weighted fusion of multi-source features and dual-dimensional attention enhancement of channels and space, effectively improving the discriminative and expressive power of the features. The specific operation is as follows: Feature stitching: The decoder upsampled features, deformed infrared encoded features, and visible light encoded features are stitched together along the channel dimension; Channel attention calibration: Channel features are extracted through adaptive average pooling and max pooling, channel attention weights are generated through convolutional layers, and channel weights are assigned to the concatenated features; Spatial attention calibration: Spatial features are extracted through depthwise separable convolution, and then... Activate the generated spatial attention weights to perform spatial dimension calibration on the channel-calibrated features; Residual fusion: The spatially calibrated features are residually fused with the original spliced ​​features to improve the feature representation capability and output the fused features after dual calibration.

[0045] The feature interaction module consists of a registration sub-network, a feature stitching module, a 1×1 convolutional compression module, and a fusion sub-network. The registration sub-network takes infrared and visible light images as input and outputs infrared registration image features, visible light original image features, and deformation field features. The feature stitching module stitches the three types of features in the channel dimension to obtain high-dimensional stitched features. The 1×1 convolutional compression module compresses the number of stitched feature channels to 64, generating the registration fusion feature F_inter. The fusion sub-network takes F_inter as input and realizes cross-module interaction and deep fusion of registration features and fusion features, completing the core fusion task and realizing cross-module interaction between registration features and fusion features.

[0046] The structure of the fusion subnetwork is as follows: The fusion subnetwork is an improvement The network, with its two-branch structure, handles the data separately. The infrared registration features and visible light features in the network are used, with the core being a multi-dimensional attention enhancement module (structure shown in Figure 4). This network consists of three branches: the first branch directly feeds the input features into the channel attention module, models the channel dimension dependency, and generates channel attention weight features; the second branch directly feeds the input features into the spatial attention module, models the spatial dimension position dependency, and generates spatial attention weight features; the third branch first passes the input features through a 1×1 convolutional layer to complete channel dimension compression and feature transformation, then splits the transformed features into three groups of features: Q, K, and V. These are then processed sequentially through dot product, scaling, positional bias self-attention calculation, and the Softmax activation function to generate a window self-attention weight matrix Awin, which is multiplied with the V feature to complete self-attention weighting, and then integrated through a 1×1 convolutional layer to obtain the window self-attention enhanced features; the output features of the three branches are jointly fed into a multi-weight fusion module, which uses three sets of learnable weights W1, W2, and W3 to complete the weighted fusion of channel attention, spatial attention, and window self-attention features. The fused features are then finally processed by a 1×1 convolutional layer. The convolutional layer completes the adjustment of channel dimension and feature integration, and outputs refined feature f. Overall, it realizes the parallel enhancement and adaptive fusion of three attention mechanisms: channel, spatial and window self-attention, which effectively improves the global and local modeling capabilities of features. The fusion subnetwork also includes a head convolutional layer and a dual-branch feature extraction module, such as... Figure 5As shown, this is a residual feature enhancement structure with parallel fusion of two branches. Starting with the input feature X, it first undergoes a 1×1 convolutional layer for unified channel transformation, then is fed into parallel two branches for feature enhancement. Finally, it outputs refined features through multiple residual fusion: the left convolutional branch passes through two consecutive convolutional layers, using short inter-layer connections to add to the input feature residuals to extract local spatial features; the right branch first passes through a normalization layer, then connects to a window self-attention module to model global dependencies within a local window, followed by another normalization and multilayer perceptron for nonlinear feature transformation. The output of this branch is also added to its own input residual. After the right branch ends, it is added element-wise to the output of the left branch, then added to the original input feature X using residuals. Multiple residual connections supplement the original information, finally outputting refined features X∽. This structure integrates the local feature extraction capability of convolution with the long-range context modeling capability of window self-attention, using residual connections to ensure the complete transmission and enhancement of feature information.

[0047] The specific operations for fusing residual blocks and head convolutional layers in the sub-network are as follows: (1) The head convolutional layer maps the number of channels of both infrared registration features and visible light features to 64, thus obtaining infrared branch features. and visible light branching characteristics ; (2) The two-branch features are input into the convolution - The hybrid block first extracts local features through a convolutional layer, and then... Window attention Capture global features to achieve local-global feature fusion; (3) The hybrid block output feature input multidimensional attention enhancement module: Spatial attention reduces the number of feature channels to 2 through channel pooling (max pooling and average pooling), generates spatial attention weights through 5×5 convolution, and multiplies them with the input features to achieve spatial detail enhancement; Channel attention compresses the features to 1×1 through adaptive average pooling, generates channel attention weights through convolutional layers, and multiplies them with the input features to achieve channel weight allocation; Window attention divides the features into 8×8 windows, and through... Multi-head self-attention within the computation window captures global contextual features, and finally, spatial attention is fused through feature fusion. Channel attention Window attention The output features are concatenated along the channel dimension, and the number of channels is restored to 64 through a 1×1 convolution to obtain the multidimensional attention-enhanced calibrated features. and characteristics ; (4) Features The feature F_vis_att is added and fused to obtain the result. The input consists of two residual blocks for feature refinement, and finally, a tail convolutional layer maps the number of channels from 64 to 1, outputting the final infrared-visible fused image. .

[0048] Example 5 This invention relates to an infrared and visible light image registration and fusion method based on multidimensional attention, the flowchart of which is shown below. Figure 1 As shown, please follow these steps: Step 1: Construct and preprocess infrared and visible light image datasets to simulate the generation of image datasets required for registration and fusion. Then, divide the generated datasets into training and test sets. Step 1 is implemented in the following steps: Step 1.1: The infrared and visible light image dataset contains image pairs of various scenes, including nighttime, foggy days, indoors, and outdoors. The raw infrared and visible light image data is preprocessed, including image denoising, size normalization, and grayscale stretching, and all images are uniformly adjusted to a preset size. Data augmentation operations such as random cropping, rotation, and flipping are performed on the normalized images. All corresponding infrared and visible light images after processing are combined in pairs to form infrared-visible light image pairs. Step 1.2: Select infrared-visible image pairs from the preprocessed image data, set the preset size area at the center of the image pair as the test set, and the rest outside the center area as the training area; perform zero-filling processing on the test area, and randomly crop image blocks of preset size from the training area as the training set to ensure that the training set and the test set do not overlap.

[0049] Step 2: Construct a registration and fusion network model for infrared and visible light images based on multidimensional attention; the model includes a registration sub-network, a feature interaction module, and a fusion sub-network.

[0050] The specific structure of the registration and fusion network model in step 2 is as follows: The overall framework of the fusion registration network model is as follows: Figure 3As shown, the system consists of a registration subnetwork and a fusion subnetwork. The output of the registration subnetwork serves as the input of the fusion subnetwork, achieving collaborative optimization of registration and fusion. The registration subnetwork is an improved MPDRNet network, including an infrared encoder, a decoder, a visible light encoder, a deformation site prediction module, a dual-domain deformation correction module, and a spatial transformation module. Both the encoder and decoder are four-level cascaded multi-scale downsampling feature extraction structures. The system starts with the input single-channel image and sequentially goes through four levels of feature extraction and downsampling processing. Each level uses a 3×3 convolutional layer with a stride of 2, combined with a batch normalization layer and a LeakyReLU activation layer as the basic downsampling unit, achieving feature transformation that halves the resolution and doubles the number of channels successively. The first level consists only of basic convolutional units. The second, third, and fourth levels all connect residual blocks after the basic convolutional units. Each residual block consists of two 3×3 convolutional layers, one LeakyReLU activation function, and residual skip connections. The final output resolution is the same as the original resolution. Figure 1 Four sets of multi-scale feature maps with channel numbers of 1 / 2, 1 / 4, 1 / 8, and 1 / 16, and channel numbers of 16, 32, 64, and 128 respectively, are used throughout the network. The network employs a pyramid-shaped downsampling structure to provide rich multi-scale features from shallow to deep and from local to global for subsequent image registration tasks. Residual blocks are used to mitigate gradient vanishing. The dual-domain deformation correction module structure is shown in Figure 3. This network is a feature refinement structure enhanced by multi-branch feature fusion and channel-space dual attention mechanisms. Starting with the three sets of input feature maps, a 1×1 convolutional layer is used to compress the channel dimension and perform preliminary feature transformation, resulting in three sets of dimensionality-reduced features. These are then fed into a 3×3 convolutional layer for spatial feature extraction and fusion. A Softmax activation function is then used to generate three sets of spatial weight maps. After fine-tuning the weights through a 1×1 convolutional layer, these weights are added element-wise with the original input feature maps to complete residual feature fusion, followed by ReLU. The activation function enhances the nonlinear expression, and finally multiplies it element-wise with the original input feature map to obtain the weighted fused features; The fused features are then sequentially fed into the channel attention module and the spatial attention module for dual enhancement. In the channel attention branch, the features are first subjected to max pooling and average pooling in parallel to extract global channel information. Then, the inter-channel dependencies are modeled through a shared multilayer perceptron (Shared MLP). After fusion through a 1×1 convolutional layer, the channel attention weights are generated by the sigmoid activation function and multiplied element-wise with the input features to complete the channel dimension weighting. In the spatial attention branch, the features are subjected to max pooling and average pooling in parallel to extract global spatial information. The spatial position dependencies are modeled through a convolutional layer. The spatial attention weights are generated by the sigmoid activation function and multiplied element-wise with the input features to complete the spatial dimension weighting. Finally, after two attention enhancements, refined features are output. Overall, adaptive weighted fusion of multi-source features and channel-space dual-dimensional attention enhancement are achieved to improve the discriminative and expressive power of the features.

[0051] The registration subnetwork in step 2 is implemented as follows: Step 2.1: The infrared image and the visible light image are input into the infrared encoder and the visible light encoder, respectively. The two encoders extract features at four scales and output the following features in sequence: an 8-channel 256×256 multi-scale feature map enc3, a 16-channel 128×128 multi-scale feature map enc2, a 32-channel 64×64 multi-scale feature map enc1, and a 64-channel 32×32 multi-scale feature map enc0. Step 2.2: The two enc0 feature maps of infrared and visible light are concatenated with 128 channels and input into the bottleneck layer of the decoder. After passing through two ConvBnLeakyRelu2d layers, the number of channels is reduced to 64 to obtain the initial feature dec0 of the decoder. Step 2.3: Perform three-level upsampling on dec0, with each level of upsampling introducing a dual-domain deformation correction module; First-level upsampling After transposing and upsampling to 64×64 with 32 channels, the upsampled decoded output is obtained. The deformation field prediction module predicts the deformation field based on the deformation field. For infrared The features are spatially deformed to obtain the deformation features. Upsampled decoding output Deformation characteristics Visible light Feature stitching, input to the dual-domain deformation correction module, and after dual calibration by channel attention and spatial attention, output the final deformation features. dec1 The second and third level upsampling processes are the same as the first level, sequentially processing multi-scale features. , The features are deformed and calibrated to output the final deformation features. dec2 , ; Step 2.4: The deformation field prediction module processes the initial features of the decoder. dec0、 dec1 dec2, dec3 Predict the deformation field separately flow0-flow3 The deformation field phi_4 is obtained by deformation fusion and vector integration through the dual-domain deformation correction module; the spatial transformation module performs spatial deformation on the original infrared image according to phi_4 to obtain the infrared registration image I_ir_reg that is registered with the visible light image.

[0052] The dual-domain deformation correction module is a feature refinement structure that integrates multi-branch feature adaptive weighting and channel-space dual attention mechanisms. The network starts with three sets of input feature maps. First, it compresses the channel dimension and performs preliminary feature transformation through a 1×1 convolutional layer, resulting in three sets of dimensionality-reduced features. These are then fed into a 3×3 convolutional layer for spatial feature extraction and fusion. A Softmax activation function generates three sets of spatial weight maps, which are then refined through a 1×1 convolutional layer. The adjusted weights are first added element-wise with the original input feature map to achieve residual feature fusion, then enhanced with a ReLU activation function for non-linear expression, and finally multiplied element-wise with the original input feature map to obtain the weighted fused features. These fused features are then enhanced by both the channel attention module and the spatial attention module. In the channel attention branch, global channel information is extracted in parallel through max pooling and average pooling, and then further enhanced by a shared multilayer perceptron (SMP). The MLP (Multi-Level Processing) model models inter-channel dependencies, fuses them through 1×1 convolutional layers, and generates channel attention weights using the Sigmoid activation function. These weights are then multiplied element-wise with the input features to achieve weighted enhancement of the channel dimension. In the spatial attention branch, features are extracted using max pooling and average pooling in parallel to extract global spatial information. Spatial location dependencies are modeled through convolutional layers, and spatial attention weights are generated using the Sigmoid activation function. These weights are then multiplied element-wise with the input features to achieve weighted enhancement of the spatial dimension. Finally, after two attention enhancements, the network outputs refined features. Overall, it achieves adaptive weighted fusion of multi-source features and dual-dimensional attention enhancement of channels and space, effectively improving the discriminative and expressive power of the features. The specific operation is as follows: Feature stitching: The decoder upsampled features, deformed infrared encoded features, and visible light encoded features are stitched together along the channel dimension; Channel attention calibration: Channel features are extracted through adaptive average pooling and max pooling, channel attention weights are generated through convolutional layers, and channel weights are assigned to the concatenated features; Spatial attention calibration: Spatial features are extracted through depthwise separable convolution, and then... Activate the generated spatial attention weights to perform spatial dimension calibration on the channel-calibrated features; Residual fusion: The spatially calibrated features are residually fused with the original spliced ​​features to improve the feature representation capability and output the fused features after dual calibration.

[0053] The feature interaction module consists of a registration sub-network, a feature stitching module, a 1×1 convolutional compression module, and a fusion sub-network. The registration sub-network takes infrared and visible light images as input and outputs infrared registration image features, visible light original image features, and deformation field features. The feature stitching module stitches the three types of features in the channel dimension to obtain high-dimensional stitched features. The 1×1 convolutional compression module compresses the number of stitched feature channels to 64, generating the registration fusion feature F_inter. The fusion sub-network takes F_inter as input and realizes cross-module interaction and deep fusion of registration features and fusion features, completing the core fusion task and realizing cross-module interaction between registration features and fusion features.

[0054] The structure of the fusion subnetwork is as follows: The fusion subnetwork is an improvement The network, with its two-branch structure, handles the data separately. The infrared registration features and visible light features in the network are used, with the core being a multi-dimensional attention enhancement module (structure shown in Figure 4). This network consists of three branches: the first branch directly feeds the input features into the channel attention module, models the channel dimension dependency, and generates channel attention weight features; the second branch directly feeds the input features into the spatial attention module, models the spatial dimension position dependency, and generates spatial attention weight features; the third branch first passes the input features through a 1×1 convolutional layer to complete channel dimension compression and feature transformation, then splits the transformed features into three groups of features: Q, K, and V. These are then processed sequentially through dot product, scaling, positional bias self-attention calculation, and the Softmax activation function to generate a window self-attention weight matrix Awin, which is multiplied with the V feature to complete self-attention weighting, and then integrated through a 1×1 convolutional layer to obtain the window self-attention enhanced features; the output features of the three branches are jointly fed into a multi-weight fusion module, which uses three sets of learnable weights W1, W2, and W3 to complete the weighted fusion of channel attention, spatial attention, and window self-attention features. The fused features are then finally processed by a 1×1 convolutional layer. The convolutional layer completes the adjustment of channel dimension and feature integration, and outputs refined feature f. Overall, it realizes the parallel enhancement and adaptive fusion of three attention mechanisms: channel, spatial and window self-attention, which effectively improves the global and local modeling capabilities of features. The fusion subnetwork also includes a head convolutional layer and a dual-branch feature extraction module, such as... Figure 5As shown, this is a residual feature enhancement structure with parallel fusion of two branches. Starting with the input feature X, it first undergoes a 1×1 convolutional layer for unified channel transformation, then is fed into parallel two branches for feature enhancement. Finally, it outputs refined features through multiple residual fusion: the left convolutional branch passes through two consecutive convolutional layers, using short inter-layer connections to add to the input feature residuals to extract local spatial features; the right branch first passes through a normalization layer, then connects to a window self-attention module to model global dependencies within a local window, followed by another normalization and multilayer perceptron for nonlinear feature transformation. The output of this branch is also added to its own input residual. After the right branch ends, it is added element-wise to the output of the left branch, then added to the original input feature X using residuals. Multiple residual connections supplement the original information, finally outputting refined features X∽. This structure integrates the local feature extraction capability of convolution with the long-range context modeling capability of window self-attention, using residual connections to ensure the complete transmission and enhancement of feature information.

[0055] The specific operations for fusing residual blocks and head convolutional layers in the sub-network are as follows: (1) The head convolutional layer maps the number of channels of both infrared registration features and visible light features to 64, thus obtaining infrared branch features. and visible light branching characteristics ; (2) The two-branch features are input into the convolution - The hybrid block first extracts local features through a convolutional layer, and then... Window attention Capture global features to achieve local-global feature fusion; (3) The hybrid block output feature input multidimensional attention enhancement module: Spatial attention reduces the number of feature channels to 2 through channel pooling (max pooling and average pooling), generates spatial attention weights through 5×5 convolution, and multiplies them with the input features to achieve spatial detail enhancement; Channel attention compresses the features to 1×1 through adaptive average pooling, generates channel attention weights through convolutional layers, and multiplies them with the input features to achieve channel weight allocation; Window attention divides the features into 8×8 windows, and through... Multi-head self-attention within the computation window captures global contextual features, and finally, spatial attention is fused through feature fusion. Channel attention Window attention The output features are concatenated along the channel dimension, and the number of channels is restored to 64 through a 1×1 convolution to obtain the multidimensional attention-enhanced calibrated features. and characteristics ; (4) Features The feature F_vis_att is added and fused to obtain the result. The input consists of two residual blocks for feature refinement, and finally, a tail convolutional layer maps the number of channels from 64 to 1, outputting the final infrared-visible fused image. .

[0056] Step 3: Train the constructed registration and fusion network model using supervised training. Optimize the model parameters jointly using multiple loss functions until the model converges. Step 4: Use the trained registration and fusion network model to register and fuse the infrared and visible light image pairs in the test set, and output the final registered and fused image.

[0057] Example 6 In step 3, the fusion loss formula is:

[0058]

[0059] in The values ​​were 0.3, 0.3, 0.2, and 0.2 respectively. To merge images, The registered infrared image, This is the original visible light image.

[0060] The model was trained using the Adam optimizer with an initial learning rate of 0.0001 and a cosine annealing learning rate decay strategy. The batch size was 2, and the training iterations were 800 epochs. End-to-end training was implemented using the PyTorch deep learning framework.

[0061] Example 7 Step 4 is as follows: The infrared and visible light image pairs from the test set are preprocessed and then input into the trained model. The model passes through the registration sub-network and the fusion sub-network in sequence, and outputs the registered image pairs and the fused image.

[0062] Registration performance was evaluated using registration error and PSNR, while fusion performance was evaluated using SSIM, VIF, and Qabf. Referring to Figures 6(a) to 6(d), experimental results show that the registration error of the method described in this invention is reduced by more than 5% compared to traditional methods, and the PSNR of the registered image is improved by more than 8 dB. The fused image achieves an SSIM of 0.4957, a VIF of 0.8536, and a Qabf of 0.433, all superior to existing comparative methods. The fused image simultaneously preserves the thermal radiation target of the infrared image and the spatial details of the visible light image. The registration and fusion effect is shown in Figure 6(d).

Claims

1. An infrared and visible image registration and fusion method based on multi-dimensional attention, characterized in that, The specific steps are as follows: Step 1: Construct and preprocess infrared and visible light image datasets to simulate the generation of image datasets required for registration and fusion. Then, divide the generated datasets into training and test sets. Step 2: Construct a registration and fusion network model for infrared and visible light images based on multidimensional attention; Step 3: Train the constructed registration and fusion network model using supervised training, and jointly optimize the model parameters through multiple loss functions until the model converges; Step 4: Use the trained registration and fusion network model to register and fuse the infrared and visible light image pairs in the test set, and output the final registered and fused image.

2. The infrared and visible light image registration and fusion method based on multidimensional attention according to claim 1, characterized in that, Step 1 is implemented in the following steps: Step 1.1: The infrared and visible light image dataset contains image pairs of various scenes, including nighttime, foggy days, indoors, and outdoors. The raw infrared and visible light image data is preprocessed, including image denoising, size normalization, and grayscale stretching, and all images are uniformly adjusted to the preset size. Data augmentation operations such as random cropping, rotation, and flipping are performed on the normalized images; all corresponding infrared and visible light images after processing are paired to form infrared-visible light image pairs. Step 1.2: Select infrared-visible image pairs from the preprocessed image data, set the preset size area at the center of the image pair as the test set, and the rest outside the center area as the training area. The test area is zero-filled, and image blocks of a preset size are randomly cropped from the training area to serve as the training set, ensuring that the training set and the test set do not overlap.

3. The infrared and visible light image registration and fusion method based on multidimensional attention according to claim 2, characterized in that, The specific structure of the registration and fusion network model in step 2 is as follows: The fusion registration network model consists of a registration sub-network and a fusion sub-network. The output of the registration sub-network serves as the input of the fusion sub-network, achieving collaborative optimization of registration and fusion. The registration sub-network is an improved MPDRNet network, which includes an infrared encoder, a decoder, a visible light encoder, a deformation site prediction module, a dual-domain deformation correction module, and a spatial transformation module. Both the encoder and decoder are four-level cascaded multi-scale downsampling feature extraction structures. The entire system starts with the input single-channel image and sequentially goes through four levels of feature extraction and downsampling processing. Each level uses a 3×3 convolutional layer with a stride of 2, combined with a batch normalization layer and a LeakyReLU activation layer as the basic downsampling unit, to achieve feature transformation that halves the resolution and doubles the number of channels successively. The first layer consists only of basic convolutional units. The second, third, and fourth layers all connect residual blocks after the basic convolutional units. Each residual block consists of two 3×3 convolutional layers, one LeakyReLU activation function, and residual skip connections. The final output consists of four sets of multi-scale feature maps with resolutions of 1 / 2, 1 / 4, 1 / 8, and 1 / 16 of the original image, and with 16, 32, 64, and 128 channels, respectively. LeakyReLU is used as the activation function throughout the process. The overall structure is a pyramid-shaped downsampling structure, which is used to provide rich multi-scale features from shallow to deep and from local to global for subsequent image registration tasks. The residual blocks are used to alleviate gradient vanishing. The dual-domain deformation correction module is a feature refinement structure enhanced by multi-branch feature fusion and channel-space dual attention mechanism. The whole system starts with the three sets of input feature maps. First, a 1×1 convolutional layer is used to complete the compression of channel dimensions and the initial feature transformation, resulting in three sets of dimensionality-reduced features. The feature is then fed into a 3×3 convolutional layer for spatial feature extraction and fusion. After passing through a Softmax activation function, three sets of spatial weight maps are generated. The weights are then finely adjusted through a 1×1 convolutional layer and added element-wise with the original input feature map to complete residual feature fusion. The ReLU activation function is then used to enhance the nonlinear expression. Finally, the feature is multiplied element-wise with the original input feature map to obtain the weighted fused feature. The fused features are then sequentially fed into the channel attention module and the spatial attention module for dual enhancement. In the channel attention branch, the features are first extracted for global channel information through parallel max pooling and average pooling, and then the inter-channel dependencies are modeled through a shared multilayer perceptron. After fusion through a 1×1 convolutional layer, the channel attention weights are generated by the Sigmoid activation function and multiplied element-wise with the input features to complete channel dimension weighting. In the spatial attention branch, the features are extracted for spatial global information through parallel max pooling and average pooling, and the spatial position dependencies are modeled through a convolutional layer. The spatial attention weights are generated by the Sigmoid activation function and multiplied element-wise with the input features to complete spatial dimension weighting. Finally, after two attention enhancements, refined features are output. Overall, adaptive weighted fusion of multi-source features and channel-space dual-dimensional attention enhancement are achieved to improve the discriminative and expressive power of the features.

4. The infrared and visible light image registration and fusion method based on multidimensional attention according to claim 3, characterized in that, The registration sub-network in step 2 is implemented as follows: Step 2.1: The infrared image and the visible light image are input into the infrared encoder and the visible light encoder, respectively. The two encoders extract features at four scales and output the following features in sequence: an 8-channel 256×256 multi-scale feature map enc3, a 16-channel 128×128 multi-scale feature map enc2, a 32-channel 64×64 multi-scale feature map enc1, and a 64-channel 32×32 multi-scale feature map enc0. Step 2.2: The two enc0 feature maps of infrared and visible light are concatenated with 128 channels and input into the bottleneck layer of the decoder. After passing through two ConvBnLeakyRelu2d layers, the number of channels is reduced to 64 to obtain the initial feature dec0 of the decoder. Step 2.3: Perform three-level upsampling on dec0, with each level of upsampling introducing a dual-domain deformation correction module; First-level upsampling After transposing and upsampling to 64×64 with 32 channels, the upsampled decoded output is obtained. ; The deformation field prediction module predicts the deformation field based on the deformation field. For infrared The features are spatially deformed to obtain the deformation features. Upsampled decoding output Deformation characteristics Visible light Feature stitching, input to the dual-domain deformation correction module, and after dual calibration by channel attention and spatial attention, output the final deformation features. dec1 The second and third level upsampling processes are the same as the first level, sequentially processing multi-scale features. , The features are deformed and calibrated to output the final deformation features. dec2 , ; Step 2.4: The deformation field prediction module processes the initial features of the decoder. dec0、 dec1 dec2, dec3 Predict the deformation field separately flow0-flow3 After deformation fusion and vector integration by the dual-domain deformation correction module, the final deformation field phi_4 is obtained; space The transformation module performs spatial deformation on the original infrared image based on phi_4 to obtain an infrared registration image I_ir_reg that is registered with the visible light image.

5. The infrared and visible light image registration and fusion method based on multidimensional attention according to claim 4, characterized in that, The dual-domain deformation correction module is a feature refinement structure that integrates multi-branch feature adaptive weighting and channel-space dual attention mechanism. The network starts with three sets of input feature maps, first compresses the channel dimension through a 1×1 convolutional layer and completes the initial feature transformation to obtain three sets of dimensionality-reduced features. The feature is then fed into a 3×3 convolutional layer for spatial feature extraction and fusion. Three sets of spatial weight maps are generated by the Softmax activation function, and the weights are then finely adjusted by a 1×1 convolutional layer. The adjusted weights are first added element-wise to the original input feature map to complete residual feature fusion, and then the nonlinear expression is enhanced by the ReLU activation function. Finally, the weights are multiplied element-wise with the original input feature map to obtain the weighted fused features. The fused features are then enhanced by the channel attention module and the spatial attention module. In the channel attention branch, the features are extracted by max pooling and average pooling in parallel to extract global channel information. The channel dependencies are then modeled by a shared multilayer perceptron. After fusion by a 1×1 convolutional layer, the channel attention weights are generated by the Sigmoid activation function and multiplied element-wise with the input features to complete the weighted enhancement of the channel dimension. In the spatial attention branch, features are extracted using max pooling and average pooling in parallel to obtain global spatial information. Spatial location dependencies are modeled through convolutional layers, and spatial attention weights are generated using the sigmoid activation function. These weights are then multiplied element-wise with the input features to achieve weighted enhancement of the spatial dimension. Finally, after two attention enhancements, the network outputs refined features. Overall, this achieves adaptive weighted fusion of multi-source features and dual-dimensional attention enhancement of channels and space, effectively improving the discriminative and expressive power of the features. The specific operation is as follows: Feature stitching: The decoder upsampled features, deformed infrared encoded features, and visible light encoded features are stitched together along the channel dimension; Channel attention calibration: Channel features are extracted through adaptive average pooling and max pooling, channel attention weights are generated through convolutional layers, and channel weights are assigned to the concatenated features; Spatial attention calibration: Spatial features are extracted through depthwise separable convolution, and then... Activate the generated spatial attention weights to perform spatial dimension calibration on the channel-calibrated features; Residual fusion: The spatially calibrated features are residually fused with the original spliced ​​features to improve the feature representation capability and output the fused features after dual calibration.

6. The infrared and visible light image registration and fusion method based on multidimensional attention according to claim 5, characterized in that, The feature interaction module consists of a registration sub-network, a feature stitching module, a 1×1 convolutional compression module, and a fusion sub-network. The registration sub-network takes infrared and visible light images as input and outputs infrared registration image features, visible light original image features, and deformation field features. The feature stitching module stitches the three types of features in the channel dimension to obtain high-dimensional stitched features. The 1×1 convolutional compression module compresses the number of stitched feature channels to 64 to generate the registration fusion feature F_inter. The fusion sub-network takes F_inter as input to realize cross-module interaction and deep fusion of registration features and fusion features, completes the core fusion task, and realizes cross-module interaction of registration features and fusion features.

7. The infrared and visible light image registration and fusion method based on multidimensional attention according to claim 6, characterized in that, The structure of the fusion subnetwork is as follows: The fusion subnetwork is an improvement The network, with its two-branch structure, handles the data separately. The infrared registration features and visible light features in the core are multi-dimensional attention enhancement modules consisting of three branches. The first branch: the input features are directly fed into the channel attention module to model the channel dimension dependency and generate channel attention weight features. The second branch: Input features are directly fed into the spatial attention module to model the spatial dimension's positional dependencies and generate spatial attention weight features. The third branch: Input features are first compressed and transformed through a 1×1 convolutional layer, then the transformed features are split into three groups of features: Q, K, and V. These are then processed sequentially through dot product, scaling, positional bias self-attention calculation, and the Softmax activation function to generate a window self-attention weight matrix Awin. This matrix is ​​multiplied by the V feature to complete self-attention weighting, and then integrated through a 1×1 convolutional layer to obtain window self-attention enhanced features. The output features from all three branches are fed into the multi-weight fusion module, which uses three sets of learnable weights (W1, W2, W3) to perform weighted fusion of channel attention, spatial attention, and window self-attention features. The fused features are finally adjusted and integrated through a 1×1 convolutional layer, outputting refined features f. Overall, this achieves parallel enhancement and adaptive fusion of the three attention mechanisms: channel, spatial, and window self-attention, effectively improving the global and local modeling capabilities of features. The fusion subnetwork also includes a head convolutional layer and a dual-branch feature extraction module. The whole starts with the input feature X, first goes through a 1×1 convolutional layer to complete the channel uniform transformation, and then is fed into the parallel dual branches for feature enhancement. Finally, it outputs refined features through multiple residual fusion: the left convolutional branch goes through two convolutional layers in succession, and uses short inter-layer connections to add the input feature residuals to extract local spatial features. The right branch first passes through a normalization layer, then connects to a window self-attention module to model the global dependencies within the local window. Subsequently, it undergoes another normalization and a multilayer perceptron to complete the nonlinear feature transformation. The output of this branch is also added to its own input residual. After the right branch ends, it is added element-wise to the output of the left branch, and then added to the original input feature X. The original information is supplemented through multiple residual connections, and finally, the refined feature X∽ is output. This structure integrates the local feature extraction capability of convolution with the long-range context modeling capability of window self-attention, and uses residual connections to ensure the complete transmission and enhancement of feature information.

8. The infrared and visible light image registration and fusion method based on multidimensional attention according to claim 7, characterized in that, The specific operations of the residual blocks and head convolutional layers in the fusion subnetwork are as follows: (1) The head convolutional layer maps the number of channels of both infrared registration features and visible light features to 64, thus obtaining infrared branch features. and visible light branching characteristics ; (2) The two-branch features are input into the convolution - The hybrid block first extracts local features through a convolutional layer, and then... Window attention Capture global features to achieve local-global feature fusion; (3) The hybrid block output feature input multidimensional attention enhancement module: spatial attention reduces the number of feature channels to 2 through channel pooling, generates spatial attention weights through 5×5 convolution, and multiplies them with the input features to achieve spatial detail enhancement; channel attention compresses the features to 1×1 through adaptive average pooling, generates channel attention weights through convolutional layers, and multiplies them with the input features to achieve channel weight allocation; window attention divides the features into 8×8 windows, and through... Multi-head self-attention within the computation window captures global contextual features, and finally, spatial attention is fused through feature fusion. Channel attention Window attention The output features are concatenated along the channel dimension, and the number of channels is restored to 64 through a 1×1 convolution to obtain the multidimensional attention-enhanced calibrated features. and characteristics ; (4) Features The feature F_vis_att is added and fused to obtain the result. The input consists of two residual blocks for feature refinement, and finally, a tail convolutional layer maps the number of channels from 64 to 1, outputting the final infrared-visible fused image. .

9. The infrared and visible light image registration and fusion method based on multidimensional attention according to claim 8, characterized in that, Step 3 is implemented in the following steps: 3.1 Loss Function Design: A joint loss function is adopted, the formula of which is: Registration loss The MSE loss of the registered infrared and visible light images, and the fusion loss. This is a combination of SSIM loss and MSE loss. 3.2 Training Hyperparameters: Based on the PyTorch 2.1.0 framework, trained with an NVIDIA GeForce RTX 4090D graphics card, Adam optimizer, initial learning rate of 0.0001, cosine annealing learning rate decay, batch size=2, training for 800 epochs; 3.3 Training process: Input the training set images into the model, perform forward propagation to obtain the registered and fused images, calculate the joint loss value, perform backpropagation to optimize all parameters of the model, and save the model weights with the best performance on the validation set during the training process.

Images