An infrared and visible image fusion method based on state space model

Abstract

This invention discloses an infrared and visible light image fusion method based on a state-space model, comprising the following steps: collecting data to obtain paired infrared and visible light image data; dividing the dataset into training and testing sets; data normalization by scaling the size of each image to the same size and then standardizing the pixel values ​​of the images to between 0 and 1; training an image fusion model based on the state-space model; and testing the model's performance. Compared with other methods, this invention can reasonably fuse complementary information from different modalities to obtain information-rich fused images, thereby assisting professionals in fields such as transportation and military, and has good market application prospects. It effectively utilizes the advantages of the state-space model and compensates for its shortcomings in the spatial and frequency domains, forming an end-to-end learning framework that can provide the necessary foundation and support for subsequent image fusion tasks.

Description

Technical Field

[0001] This invention belongs to the fields of deep learning, computer vision, and image fusion, specifically a method for fusing infrared and visible light images based on a state-space model. Background Technology

[0002] Infrared and Visible Image Fusion (IVIF) aims to integrate complementary information acquired from different sensors in the same scene to obtain a fused image with richer visual information. As one of the fundamental tasks in computer vision, IVIF serves as the basis for many computer vision-related problems, such as depth prediction, object detection, and semantic segmentation. Therefore, IVIF has extremely broad application prospects in many fields, including defense, agriculture, industry, and transportation.

[0003] With the development of deep learning technology, the IVIF algorithm has gradually transitioned from traditional methods to methods based on deep networks, including convolutional neural networks, Transformers, and state-space models. These methods, through end-to-end learning, have effectively improved the quality of fused images and promoted the further development of computer vision tasks.

[0004] In IVIF, convolutional neural networks primarily extract features from the input image through stacked convolution operations and utilize feature fusion strategies to generate the final fused image. Compared to traditional manual feature extraction methods, convolutional neural network-based methods can automatically learn the mapping relationships between different modalities, improving fusion quality. However, since the receptive field of convolutional neural networks mainly depends on the size of the convolution kernel, the limitations of their local feature extraction make it difficult to capture global dependencies between distant pixels.

[0005] The Transformer initially achieved breakthroughs in natural language processing tasks and has gradually expanded into the field of computer vision. Its core lies in its self-attention mechanism, which can globally model the feature correlations of input data. Therefore, in IVIF, Transformer-based methods can effectively model global features and perform deep interactions between different modalities to generate more complete fused images. However, the standard Transformer requires calculating the global correlations between input data, resulting in quadratic computational complexity and high computational cost.

[0006] In recent years, state-space models have gradually gained attention as an emerging architecture in deep learning. State-space models model sequential information through a hidden state update mechanism, achieving linear computational complexity and improving computational efficiency while preserving the global receptive field. In IVIF (Inductively Interpretive Imagery), state-space models can effectively capture modal information and achieve information fusion at a relatively low computational cost. However, state-space models lack complete spatial and frequency domain awareness capabilities, both of which are highly correlated with the preservation of modal information and are crucial to the performance of IVIF.

[0007] In summary, IVIF is an important research direction in computer vision, and how to make full use of the advantages of state-space models and make up for their shortcomings is one of the challenges that IVIF urgently needs to solve. Summary of the Invention

[0008] To address the shortcomings of existing methods, this invention proposes an IVIF method based on a state-space model, which enhances the state-space model in both the spatial and frequency domains, thereby improving the performance of the IVIF algorithm.

[0009] The technical solution of the present invention:

[0010] An IVIF method based on a state-space model includes the following steps:

[0011] Step 1: Collect data and acquire paired infrared and visible light image data;

[0012] Step 2: Divide the dataset into training and test sets by dividing the acquired infrared and visible light image datasets.

[0013] Step 3: Data normalization. Use scaling to scale the size of each image to the same size, and then normalize the range of pixel values ​​of the image to between 0 and 1.

[0014] Step 4: Train the network model. Use the training set to train the IVIF model based on the state space model, observe the changes in the loss function value during training, and terminate training when the loss value converges to obtain the currently trained fusion model.

[0015] The IVIF model based on the state-space model includes two image reconstruction branches and one image fusion branch. The two image reconstruction branches have the same structure, including a primary convolutional layer and a feature extraction module: the Spatial-Frequency enhanced Mamba Block (SFMB). The image fusion branch includes a primary convolutional layer, the SFMB feature extraction module, and the Dynamic Fusion Mamba Block (DFMB) feature fusion module.

[0016] The Space-Frequency Enhanced Mamba Module (SFMB) includes a Mixed-scale Mamba Block (MMB), a Channel Enhanced Block (CEB), and a Frequency Enhanced Block (FEB), given input features. It is represented as:

[0017]

[0018]

[0019]

[0020]

[0021]

[0022] in, Layer normalization, which is independent of batch size during computation, normalizes each sample independently, which helps improve the stability of network training and accelerates convergence during training.

[0023] The hybrid-scale state-space model module is used to enhance state-space models in the spatial dimension of the spatial domain, including linear layers. A convolutional layer with a kernel size of 3x3 State-space model layer Upsampling layer downsampling layer SiLU activation function And layer normalization, given input features It is represented as:

[0024]

[0025]

[0026]

[0027]

[0028]

[0029]

[0030]

[0031]

[0032] By extracting features at mixed scales, the mixed-scale state space model module can better capture fine-grained details and high-level semantics, thereby enhancing the performance of the state space model in the spatial dimension of the spatial domain.

[0033] The channel enhancement module is used to enhance the state-space model along the channel dimension in the spatial domain, including global max pooling. Global average pooling A convolutional layer with a kernel size of 1x1 ReLU activation function and the Sigmoid activation function Given input features After passing through convolutional layers and channels, they are separated into and Then, the input is routed to two branches, which are represented as follows:

[0034]

[0035]

[0036] Finally, let's... and Perform channel mixing, then add the feature of skip connections. Get output By simultaneously considering both the average and maximum responses across channels, the channel enhancement module can better model channel-level information, thereby enhancing the performance of the state-space model in the channel dimension of the spatial domain.

[0037] The frequency domain enhancement module is used to enhance the state-space model in the amplitude and phase spectra in the frequency domain. It includes Fast Fourier Transform (FFT), convolutional layers, ReLU activation function, and Inverse Fast Fourier Transform (IFFT), given input features. After fast Fourier transform, the following results were obtained: and The inputs are then routed to two branches, which are represented as follows:

[0038]

[0039]

[0040] Finally, let's... and After being transformed to the spatial domain by inverse fast Fourier transform, and compared with the input... Add to get the output By processing the amplitude spectrum and phase spectrum separately, the frequency domain enhancement module can better handle different types of information, thereby enhancing the performance of the state-space model in the frequency domain.

[0041] The feature fusion module integrates features from the three task branches. While simple fusion strategies like addition or concatenation can be used, these strategies fail to fully utilize the features from different branches because they are fixed and cannot adapt to varying features across different modalities. The Dynamic Fusion Mamba module (DFMB) integrates features from the three task branches, including linear layers, convolutional layers, SiLU activation functions, Sigmoid activation functions, and a state-space model layer, given input features. , and After layer normalization, we get , and The subsequent process is represented as follows:

[0042]

[0043]

[0044]

[0045]

[0046]

[0047] in, and This represents a learnable scaling factor; by dynamically fusing features from different branches, the Dynamic Fusion Mamba module can better preserve the complete content of the source image.

[0048] The fusion model uses a loss function to supervise the fusion result; the total loss consists of intensity error loss and gradient error loss; the input infrared and visible light images are denoted as follows: and The fused image is denoted as Strength error loss Represented as:

[0049]

[0050] in, and The length and width of the image are respectively. This indicates the selection of the maximum value among all elements. express Norm; Gradient error loss Represented as:

[0051]

[0052] in, Represents the Sobel gradient operator. Represents absolute value operation;

[0053] Step 5: Test the model performance. Input the infrared and visible light images to be fused from the test set into the trained fusion model to obtain the fused image and measure the indicators.

[0054] Engineering technical indicators include: whether the model can preserve the texture details in the image; whether the model can highlight targets in the image, such as people and vehicles; and whether the model fully integrates the complementary information of the two modalities.

[0055] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0056] This invention realizes an image fusion method based on a state-space model, which can reasonably fuse complementary information from different modalities to obtain information-rich fused images. This can assist professionals in fields such as transportation and military, and has good market application prospects.

[0057] The image fusion method based on the state space model provided by this invention effectively utilizes the advantages of the state space model and makes up for its shortcomings in the spatial and frequency domains, forming an end-to-end learning framework that can provide the necessary foundation and support for subsequent image fusion tasks. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the overall structure of the present invention.

[0059] Figure 2 This is a schematic diagram of the hybrid scale state space module structure of the present invention.

[0060] Figure 3 This is a schematic diagram of the channel enhancement module structure of the present invention.

[0061] Figure 4 This is a schematic diagram of the frequency domain enhancement module structure of the present invention.

[0062] Figure 5 This is a schematic diagram of the feature fusion module structure of the present invention. Detailed Implementation

[0063] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0064] The specific implementation of this invention is as follows:

[0065] Step 1: Data Collection. In this invention, the first step is to collect paired infrared and visible light image data. To ensure the diversity and breadth of the data, this invention primarily uses a publicly available dataset: the MSRS dataset.

[0066] Step Two: Dataset Splitting. The collected data needs to be divided into training and testing sets for effective model training and evaluation. Specifically, this invention only trains and tests on the MSRS dataset. The main purpose of splitting the dataset is to ensure that the model can generalize in various data environments and to test its fusion performance under different data distributions.

[0067] Step 3: Data Regularization. To enable the network to effectively process images of different sizes and resolutions, the images are first preprocessed uniformly. The specific steps are as follows: 1. Image Scaling: Scaling all images to a uniform size of 128×128 ensures consistent input image dimensions, facilitating neural network processing. 2. Pixel Value Standardization: Dividing all image pixel values ​​by 255 ensures a consistent range of input data values. This standardizes pixel values ​​to between 0 and 1, reducing numerical fluctuations during training and accelerating model training. 3. Removing Redundant Color Information: For visible light images containing multi-channel color information, color space conversion is performed, retaining the luminance channel to simplify calculations.

[0068] Step 4: Train the Network Model. After data normalization, train the IVIF model based on the state-space model using the training set. During training, the network model continuously optimizes to minimize the loss function, thereby improving the quality of the fused images. During training, the aforementioned loss function is used to evaluate the difference between the network output and the true labels. As training progresses, the value of the loss function should gradually decrease, indicating that the model is learning and optimizing. Monitor the changes in the loss function in real time during training. By observing the convergence of the loss value, determine whether the training has achieved the expected results. When the loss value tends to stabilize after several iterations, it indicates that the model has converged, and the training process can be terminated. After training, save the current model weights for inference and testing in subsequent steps.

[0069] Step 5: Test Model Performance. After model training is complete, infrared and visible light image data from the test set are used for evaluation to obtain the final fused image. The specific steps are as follows: 1. Load Saved Weights: First, load the model weights saved during training to ensure that the trained model is used for testing. 2. Input Test Data: Input the images to be fused from the test set into the trained fusion model. The model will generate the fused image based on the input infrared and visible light images. 3. Evaluate Fusion Performance: Compare the fused image obtained in the previous step with the fused images obtained by other methods to determine the differences in metrics. The comparison results are shown in the table below. Of the eight metrics compared, the method of this invention achieved optimal performance in seven metrics and near-optimal performance in one metric, demonstrating the superiority of this invention.

[0070]

Claims

1. An infrared and visible image fusion method based on a state space model, characterized in that, The steps include the following: Step one: collect data, obtain pairs of infrared and visible light image data; Step two: divide the data set, divide the obtained infrared and visible light image data set into a training set and a test set; Step three: data regularization, scale the size of each image to the same size using scaling, and then standardize the pixel value range of the image to between 0 and 1; Step four: train the network model, train the IVIF model based on the state space model using the training set, observe the change of the loss function value during training, and terminate the training when the loss value converges, to obtain the current trained fusion model; The IVIF model based on the state space model includes two image reconstruction branches and an image fusion branch; The two image reconstruction branches are the same in structure, including a primary convolutional layer and a feature extraction module: a spatial-frequency enhancement Mamba module SFMB; the image fusion branch includes a primary convolutional layer, a feature extraction module SFMB, and a feature fusion module: a dynamic fusion Mamba module DFMB; The spatial-frequency enhancement Mamba module SFMB includes a mixed scale state space model module MMB, a channel enhancement module CEB, and a frequency domain enhancement module FEB, given an input feature which is expressed as: ； ； ； ； ； wherein is the layer normalization; a mixed-scale state space model module for augmenting the state space model in the spatial dimension of the spatial domain, comprising a linear layer a convolutional layer with a convolution kernel size of 3x3 a state space model layer an up-sampling layer a down-sampling layer a SiLU activation function and layer normalization, given input features which is expressed as: ； ； ； ； a channel enhancement module for enhancing the state space model in the channel dimension of the spatial domain, including global max pooling , global average pooling , a convolution layer with a kernel size of 1x1 , a ReLU activation function , and a Sigmoid activation function , given an input feature , after passing through the convolution layer and being separated into and , and then inputting into two branches respectively, which are represented as: Finally, add the output of the last convolutional layer and perform channel mixing, then add the features of the skip connection get the output ; a frequency domain enhancement module for enhancing the state space model on the amplitude spectrum and the phase spectrum in the frequency domain, including a fast Fourier transform, a convolution layer, a ReLU activation function, and an inverse fast Fourier transform, given an input feature , after fast Fourier transform, respectively and , then input to two branches respectively, which are represented as: ； Finally, the output is obtained by adding the outputs of the two filters and transformed to the spatial domain by inverse fast Fourier transform, and the input is added to obtain the output ; The dynamic fusion Mamba module DFMB is used to integrate features of three task branches, including linear layers, convolution layers, SiLU activation functions, Sigmoid activation functions, and state space model layers, given input features , and , after layer normalization, obtaining , and , the subsequent process is represented as: ； ； wherein, and denotes a learnable scaling factor; The fusion model is supervised by using a loss function on the fusion result; a total loss is composed of an intensity error loss and a gradient error loss; input infrared light and visible light images are respectively denoted as and , a fusion image is denoted as , and the intensity error loss is expressed as: ； wherein, and are the length and width of the image, respectively, denotes an element-wise maximum selection, denotes norm; gradient error loss is denoted as: ； wherein represents a Sobel gradient operator, represents an absolute value operation; Step five: test the model effect, input the infrared and visible light images to be fused in the test set into the trained fusion model, obtain the fused image, and measure the indicators.

Images