An infrared and visible light image fusion system based on multi-modal feature difference

Abstract

The application provides an infrared and visible light image fusion system based on multi-modal feature difference. The system realizes improving the quality and definition of infrared and visible light image fusion, first, multi-level and multi-scale features are extracted through a feature extraction module; then, the features are enhanced through a TAFEM module, in the feature fusion module, the same components and difference components between the two modal images are fused through the attention fusion mode of DIIM and CMIFM, finally, the image reconstruction module refines and up-samples the fused features to generate a high-quality fused image. The application improves the feature extraction quality of the model by innovatively adopting a parallel extraction mode of texture information and multi-scale features, also proposes a TAFEM module to reduce information loss in the feature extraction stage, and improves a CMIFM module on the basis of a traditional DIIM module and an ACIIM module to make up for the deficiency of the traditional cross-attention mechanism in extracting common information.

Description

Technical Field

[0001] This invention relates to the field of infrared and visible light image fusion, specifically an infrared and visible light image fusion system based on multimodal feature differences. Background Technology

[0002] Infrared and visible light image fusion technology aims to effectively combine information from infrared and visible light images to generate a new image that incorporates the advantages of both. Infrared images can capture heat source information and are suitable for low-light or nighttime environments, while visible light images provide rich environmental details and are suitable for daytime or well-lit scenarios. By fusing these two types of images, it is possible to detect and navigate concealed targets, thereby improving the system's security and combat capabilities.

[0003] In recent years, with the rapid development of deep learning technology, infrared and visible light image fusion systems based on multimodal feature differences have gradually become a research hotspot. These methods utilize the powerful feature extraction and representation capabilities of deep neural networks to automatically learn complex features in images and achieve high-quality image fusion.

[0004] Furthermore, infrared and visible light image fusion technology has broad application prospects in many fields. In intelligent monitoring, fusion technology can provide clearer images in complex environments and improve the reliability of monitoring systems. In medical image analysis, fusion technology helps improve the accuracy of diagnosis. In autonomous driving, fusion technology can enhance the vehicle's perception of the surrounding environment and improve driving safety. However, existing fusion methods still face some challenges, such as how to retain information in the source image and reduce information loss, and how to better focus on and process the differences and common components between different modal images. Summary of the Invention

[0005] To address the above issues, this invention starts with infrared and visible light images, extracts multi-level and multi-scale features through a feature extraction module, then enhances the features using a TAFEM module to reduce information loss. In the feature fusion module, attention fusion methods DIIM and CMIFM are used to fuse the similar and different components between the two modalities. Finally, the image reconstruction module refines and upsamples the fused features to generate a high-quality fused image. Based on this, this invention proposes an infrared and visible light image fusion system based on multi-modal feature differences.

[0006] Therefore, the technical solution adopted by the present invention is as follows:

[0007] M1, Feature Extraction Module: The original infrared image and the visible light image are respectively input into the edge enhancement feature extraction unit and the multi-scale convolutional feature fusion unit, and feature extraction is performed on the two images. The extracted features of the original infrared and visible light images are concatenated in the channel dimension. The output of the feature extraction module is the concatenated infrared image features and visible light image features.

[0008] M2, Pre-reconstruction Module: The input of the pre-reconstruction module is the infrared and visible light image features extracted by the feature extraction module. The extracted infrared and visible light image features are reverse-engineered to restore the original input infrared and visible light image. Pixel-by-pixel loss calculation is performed between the pre-reconstruction module and the original input image, and optimization is performed through the loss function. The infrared and visible light features are respectively processed through grouped convolution, 3×3 convolution, layer normalization, and Hardwish activation function to obtain the embedded features.

[0009] M3, Feature Fusion Module: The input of the feature fusion module is the embedded features obtained by the pre-reconstruction through grouped convolution, 3×3 convolution, layer normalization and Hardwish activation function. Through the feature fusion module, the same and different components between infrared and visible light images are fused to complete the feature fusion.

[0010] M4, Reconstruction Module: Upsamples the fused features output by the feature fusion module through pixel rearrangement operation to restore the high resolution of the image. The reconstruction module generates a high-quality fused image through multi-level feature processing, optimization and reconstruction.

[0011] Furthermore, the feature extraction of M1 specifically comprises two core parts: an edge enhancement feature extraction unit (ER-FEM) and a multi-scale convolutional feature fusion unit (MS-CFFM).

[0012] The edge enhancement feature extraction unit extracts edge information from the image using the Laplacian operator and the Sobel operator.

[0013] Furthermore, the multi-scale convolutional feature fusion unit employs convolutional kernels of different scales to extract features at different scales of the image.

[0014] The infrared and visible light image features extracted by the edge enhancement feature extraction unit and the multi-scale convolutional feature fusion unit are concatenated along the channel dimension and used as the output of the feature extraction module.

[0015] The ER-FEM unit extracts edge features and outputs feature map F1; the MS-CFFM unit extracts multi-scale features and outputs feature map F2.

[0016] F1=H×W×C1 F2=H×W×C2

[0017] Where H and W are the spatial dimensions of the image, and C1 and C2 are the number of channels.

[0018] F1 and F2 are concatenated along the channel dimension to obtain a new feature map F. concat The formula is:

[0019] F concat =concat(F1,F2)

[0020] The feature map F concat The dimensions are H×W×(C1+C2).

[0021] Furthermore, the pre-reconstruction module is used for feature reconstruction during the training phase. The pre-reconstruction module includes one convolutional layer and two residual blocks. During network training, the infrared and visible light image features extracted by the feature extraction module are fed into the pre-reconstruction module. The pre-reconstruction module inversely reconstructs the original input infrared and visible light images from the infrared and visible light image features.

[0022] The steps of the training phase are as follows:

[0023] First, the infrared and visible light image features are processed through a convolutional layer to extract local information for subsequent processing. Then, the features are further passed to two residual blocks, and deep modeling is used to further enhance the expressive power of the features.

[0024] Finally, the pre-reconstructed image generated by the pre-reconstruction module is compared with the original input image using pixel-by-pixel loss calculation. By optimizing the loss function, information loss is reduced. The formula is as follows:

[0025]

[0026] in, I is a visible light image reconstructed by the model based on the visible light features extracted by the feature extraction module. vi The original visible light image, The L1 loss between the pre-reconstructed visible light image and the original visible light image, I is an infrared image reconstructed by the model based on the infrared features extracted by the feature extractor. ir The original infrared image, To reconstruct the L1 loss between the infrared image and the input infrared image, L reconstruct For pre-reconstruction module loss;

[0027] The infrared and visible light features are respectively processed through grouped convolution, 3×3 convolution, layer normalization, and Hardwish activation function to obtain embedded features.

[0028] The pre-reconstruction module is optimized based on the loss function. The pre-reconstruction module only participates in the calculation during the training phase. During the inference phase, the module will be removed and will no longer participate in the calculation.

[0029] Furthermore, the specific implementation steps of the feature fusion module are as follows:

[0030] Based on the Differential Information Injection Unit (DIIM) and Alternating Common Information Injection Unit (ACIIM) in the ATFuse network, a Common Mode Injection Fusion Unit (CMIFM) is constructed. This CMIFM module replaces the Alternating Common Information Injection Unit in the original ATFuse network for feature fusion.

[0031] The feature fusion is as follows: The feature fusion module consists of a difference information injection unit (DIIM) and a common mode injection fusion unit (CMIFM). The difference information injection unit and the common mode injection fusion unit fuse the same and different components between infrared and visible light images, and apply them to the common mode injection fusion unit to complete the feature fusion.

[0032] Furthermore, the upsampling of M4, through upsampling technology, performs pixel rearrangement operations to restore the high resolution of the image.

[0033] The image reconstruction module generates a high-quality fused image through multi-level feature processing, optimization, and reconstruction.

[0034] The image reconstruction module introduces the loss function trained on the ATFuse architecture, and adds the SSIM loss and TAFEM loss as the total loss function:

[0035] L total =L ATFuse +αL ssim (I f ,I vi )+(1-α)L ssim (I f ,I ir )+βL reconstruct

[0036] Among them, L ATFuse The loss function used for training the ATFuse neural network, I f For the fused image output by the model, I vi For the original visible light image, I ir The image is the original infrared image. α is a hyperparameter, ranging from 0 to 1, used to balance the weights of the two structural similarity loss operations. L reconstruct The reconstruction loss is represented by β, a hyperparameter that describes the contribution of the reconstruction loss to the overall system and must be a positive number.

[0037] L ssim(I f ,I vi ) = 1 - SSIM(I f ,I vi )L ssim (I f ,I ir ) = 1 - SSIM(I f ,I ir )

[0038] Among them, SSIM(I f ,I vi SSIM(I) represents the structural similarity between the fused image and the visible light image. f ,I ir () represents the structural similarity between the fused image and the infrared image.

[0039] Compared with the prior art, the advantages of the present invention are as follows:

[0040] 1. This invention adopts a parallel extraction method of texture information and multi-scale features, using two parallel branches to extract and integrate features at different levels of the image, thereby improving the feature extraction quality of the model without affecting the model's inference efficiency.

[0041] 2. The pre-reconstruction module designed in this invention recovers the source image during the feature extraction stage, performs supervised training on the feature extraction module, enhances the network's ability to retain feature information of the source image, minimizes information loss during the feature extraction stage, and retains as much information from the source image as possible in the fused image.

[0042] 3. This invention draws on the concepts of common mode and differential mode in signal analysis, and applies the method of extracting the common mode component of two signals to the traditional cross-attention mechanism, which significantly enhances the cross-attention module's ability to extract common features between the two modes. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart of the infrared and visible light image fusion process of the present invention;

[0045] Figure 2 This is a schematic diagram of a visible light image reconstructed by the inference stage model of the present invention based on the extracted features;

[0046] Figure 3This is a schematic diagram of an infrared image reconstructed by the inference stage model of the present invention based on the extracted features;

[0047] Figure 4 This is a schematic diagram of the final fusion image during the reasoning stage of the present invention. Detailed Implementation

[0048] To achieve the above objectives, the present invention provides an infrared and visible light image fusion system based on multimodal feature differences, the method comprising:

[0049] M1. The original infrared image and the visible light image are respectively input into the edge enhancement feature extraction unit and the multi-scale convolutional feature fusion unit, and feature extraction is performed on the two images. The extracted features of the original infrared and visible light images are concatenated along the channel dimension. The output of the feature extraction module is the concatenated infrared image features and visible light image features.

[0050] In the feature extraction module, an innovative multi-module collaborative working method is adopted, combining different feature extraction strategies to achieve the interaction and fusion of features of infrared and visible light images at different scales and levels.

[0051] The feature extraction module mainly consists of two core parts: the Edge Enhancement Feature Extraction Unit (ER-FEM) and the Multi-Scale Convolutional Feature Fusion Unit (MS-CFFM).

[0052] First, the edge enhancement feature extraction unit extracts edge information from the image using the Laplacian and Sobel operators. This unit pays particular attention to capturing details and edge features in the image, especially in infrared and low-contrast visible light images, where edge information is often ignored or difficult to distinguish. Therefore, by enhancing the edge information of the image, the ER-FEM module improves the image's recognizability.

[0053] The Laplacian operator, as a second-order differential operator, can efficiently capture edge and detail changes. It is used to initialize the second derivative of an image and can detect abrupt changes in grayscale values, thus highlighting edge information. By adjusting the corners of the image, an image reflecting the second-order gradient information can be obtained, thereby highlighting changes in edges and details within the image.

[0054] In infrared images, applying the Laplacian operator makes the edges of heat sources more clearly detectable, which helps in target localization and segmentation.

[0055] In visible light images, the Laplacian operator can be used to extract the image's simulation and significant changes, enhancing the image's structural features.

[0056] The Sobel operator highlights edge information in an image by calculating the first derivatives of the image in the horizontal and vertical directions.

[0057] The horizontal Sobel operator is used to detect edges in the horizontal direction.

[0058]

[0059] The vertical Sobel operator is used to detect edges in the vertical direction.

[0060]

[0061] These two operators can be used to calculate the gradient of the image in the horizontal and vertical directions, thereby obtaining the edge information of the image.

[0062] Horizontal direction G x and vertical direction G y The gradients are combined to obtain the edge strength, which is usually reflected by calculating the magnitude of the gradient:

[0063]

[0064] The Sobel operator can be used to extract important edge features from infrared and visible light images, which helps to complete subsequent image processing tasks.

[0065] Secondly, the multi-scale convolutional feature fusion unit uses convolutional kernels of different scales, such as 3×3 and 5×5, to extract features of different scales in the image. In the image fusion task, features of different scales can help the model to fully understand the overall structure and local details of the image. Multi-scale feature fusion enables the model to pay attention to both large-scale structural features and subtle local changes in the image at the same time, thereby enhancing the global consistency and local detail representation of the image.

[0066] Small-scale convolution kernels, such as 3×3, can capture local details and small-scale texture information in an image. For details such as edges, textures, and small objects in an image, 3×3 convolution kernels can provide better extraction results.

[0067] Large-scale convolutional kernels, such as 5×5, can capture a wider range of contextual information and global structural features, helping to extract macroscopic information such as large objects and backgrounds in images.

[0068] Through this multi-scale feature extraction, the model can simultaneously acquire local details and global structural information of an image.

[0069] Finally, the features extracted from the two units are concatenated along the channel dimension to serve as the output of the feature extraction module.

[0070] The ER-FEM module extracts edge features and outputs feature map F1; the MS-CFFM module extracts multi-scale features and outputs feature map F2.

[0071] F1=H×W×C1 F2=H×W×C2

[0072] Where H and W are the spatial dimensions of the image, and C1 and C2 are the number of channels.

[0073] F1 and F2 are concatenated along the channel dimension to obtain a new feature map F. concat The formula is:

[0074] F concat =concat(F1,F2)

[0075] The spliced ​​feature map F concat The dimensions are H×W×(C1+C2).

[0076] M2. The infrared and visible light image features extracted by the feature extraction module are sent to the pre-reconstruction module. The pre-reconstruction module reversely restores the extracted infrared and visible light image features to the original input infrared and visible light image, performs pixel-by-pixel loss calculation with the original input image, and optimizes it through the loss function. The infrared and visible light features are respectively processed by group convolution, 3×3 convolution, layer normalization and Hardwish activation function to obtain the embedded features.

[0077] To optimize the performance of the feature extraction module and minimize information loss, a pre-reconstruction module, abbreviated as TAFEM module, is designed for feature reconstruction during the training phase. This enhances the perceptual and expressive capabilities of the feature extraction module through a supervised mechanism. The module contains one convolutional layer and two residual blocks. During the network training phase, the infrared and visible light features extracted by the feature extraction module are fed into the pre-reconstruction module, which inversely restores these features to the original input infrared and visible light images.

[0078] First, the infrared and visible light features undergo a convolutional operation to adjust their dimensions and scale, and extract local information for subsequent processing. Then, the infrared and visible light features are further passed to two residual blocks. Deep modeling further enhances the expressive power of the features. The design of the residual blocks effectively alleviates the gradient vanishing problem in deep networks and can capture higher-order semantic information, providing stronger expressiveness and robustness for the infrared and visible light features.

[0079] Finally, the pre-reconstructed image generated by the pre-reconstruction module is compared with the original input image using pixel-by-pixel loss calculation. By optimizing the loss function, the feature extractor is guided to extract more comprehensive and information-rich features, reducing information loss.

[0080]

[0081] in, I is the visible light image reconstructed by the model based on the features extracted by the feature extraction module. vi The original visible light image, The L1 loss between the pre-reconstructed visible light image and the original visible light image, I is the infrared image reconstructed by the model based on the features extracted by the feature extractor. ir The original infrared image, The L1 loss between the pre-reconstructed infrared image and the input infrared image,

[0082] The two input features, infrared and visible light features extracted by the feature extraction module, undergo a feature embedding operation. First, they are grouped into 3×3 convolutions with 64 groups. Then, they are processed by 3×3 convolutions, layer normalization, and the Hardwish activation function to obtain the embedded features. This operation transforms the input features into tokens, preparing them for subsequent feature fusion.

[0083] The pre-reconstruction module is optimized based on the loss function. The pre-reconstruction module only participates in the computation during the training phase. The performance of the feature extractor is optimized through supervision. During the inference phase, this module will be removed and will no longer participate in the computation, thereby improving the network's running speed. Through this design, we not only enhance the network's learning ability, but also ensure the efficiency of the inference process.

[0084] S3. Input the embedded features into the feature fusion module. Through the feature fusion module, extract the common mode information and differential mode information between the infrared and visible light images to complete the feature fusion.

[0085] Based on the Differential Information Injection Unit (DIIM) and Alternating Common Information Injection Unit (ACIIM) in the ATFuse network, a Common Mode Injection Fusion Unit (CMIFM) is constructed. This CMIFM replaces the Alternating Common Information Injection Unit in the original ATFuse network for feature fusion. The specific structure of the CMIFM is shown in [link to CMIFM]. Figure 1 ,

[0086] Based on the common-mode and differential-mode signal decomposition method in analog circuit signal analysis, in traditional signal analysis, a signal can be divided into common-mode signal and differential-mode signal, representing the similar and different parts of two signals, respectively. In image fusion tasks, this invention adopts a similar decomposition method to fuse the same and different components between infrared and visible light images, thereby optimizing the information fusion process.

[0087] This innovative approach is applied to the feature fusion module, which consists of a difference information injection unit (DIIM) and a common mode injection fusion unit (CMIFM). The difference information injection unit and the common mode injection fusion unit extract the common mode information and difference mode information between infrared and visible light images, and apply the extracted common mode information and difference mode information to the common mode injection fusion unit to complete feature fusion.

[0088] M4 performs upsampling through pixel rearrangement to restore the high resolution of the image. The reconstruction module generates a high-quality fused image through multi-level feature processing, optimization, and reconstruction.

[0089] In this step, pixel rearrangement is performed using upsampling technology to restore the high resolution of the image. This operation effectively reduces information loss, and the residual blocks further enhance the expressive power of image features through depth modeling, thereby optimizing the image reconstruction effect.

[0090] Ultimately, the entire image reconstruction module generates a high-quality fused image through multi-level feature processing, optimization, and reconstruction. This module not only effectively integrates information from infrared and visible light images, but also ensures that the generated image quality and detail reach the highest level through meticulous feature refinement and upsampling operations.

[0091] We introduce the loss function used in training with the ATFuse architecture, and add the SSIM loss and TAFEM loss to form the total loss function:

[0092] L total =l ATFuse +αL ssim (I f ,I vi )+(1-α)L ssim (I f ,I ir )+βL reconstruct

[0093] Among them, l ATFuse The loss function used for training the ATFuse neural network, I f For the fused image output by the model, I vi For the original visible light image, I ir The image is the original infrared image. α is a hyperparameter, ranging from 0 to 1, used to balance the weights of the two structural similarity loss operations. L reconstruct The reconstruction loss is represented by β, a hyperparameter that describes the contribution of the reconstruction loss to the overall system and must be a positive number.

[0094] L ssim (I f ,I vi ) = 1 - SSIM(I f ,Ivi )L ssim (I f ,I ir ) = 1 - SSIM(I f ,I ir )

[0095] Among them, SSIM(I f ,I vi SSIM(I) represents the structural similarity between the fused image and the visible light image. f ,I ir () represents the structural similarity between the fused image and the infrared image.

[0096] Comparative experiment:

[0097] Five classic infrared and visible light image fusion methods were selected, trained on the MSRS dataset, and tested on the MSRS test set. Six common image fusion evaluation metrics were chosen to assess the fusion results of each method, and the average value of all fusion results was used for metric calculation.

[0098]

[0099]

[0100] By comparing with the seven existing state-of-the-art (SOTA) methods, the method of this invention is effective and competitive, especially with high mutual information, demonstrating that the information extraction and retention mechanism in the feature extraction stage is effective.

[0101] In summary, the advantages of this invention lie in its ability to extract multi-level and multi-scale features from infrared and visible light images through a feature extraction module, followed by feature enhancement through a TAFEM module to reduce information loss. In the feature fusion module, attention fusion methods such as DIIM and CMIFM are used to fuse the common and different components between the two modalities. Finally, the image reconstruction module refines and upsamples the fused features to generate a high-quality fused image. The overall architecture, through the collaborative work of multiple modules, extracts the common and different modal components of features from different modalities and then fuses them, fully utilizing the complementarity of infrared and visible light images to ensure detail preservation and information accuracy, resulting in a significantly improved visual effect in the final fused image.

[0102] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multi-modal feature difference based infrared and visible image fusion system, characterized in that, The system comprises: M1, a feature extraction module: input the original infrared image and the visible light image into the edge enhancement feature extraction unit and the multi-scale convolution feature fusion unit respectively, and perform feature extraction on the two images, and the extracted features of the original infrared and visible light images are spliced in the channel dimension, and the output of the feature extraction module is the spliced infrared image features and visible light image features; M2, a pre-reconstruction module: the input of the pre-reconstruction module is the infrared and visible light image features extracted by the feature extraction module, the extracted infrared and visible light image features are reversely restored into the original input infrared and visible light images, the pixel-by-pixel loss calculation is performed on the original input images, and the loss function is optimized, the infrared and visible light features are respectively subjected to grouped convolution, 3*3 convolution, layer normalization and Hardwish activation function, and embedded features are obtained; M3, a feature fusion module: the input of the feature fusion module is the embedded features obtained by the pre-reconstruction and subjected to grouped convolution, 3*3 convolution, layer normalization and Hardwish activation function, the same components and difference components between the infrared and visible light images are fused through the feature fusion module, and the feature fusion is completed; M4, a reconstruction module: the fusion features output by the feature fusion module are up-sampled through a pixel rearrangement operation, the high resolution of the image is restored, and the reconstruction module generates a high-quality fusion image through multi-level feature processing, optimization and reconstruction; The feature extraction of M1 is specifically: the feature extraction module mainly comprises two core parts, i.e., an edge enhancement feature extraction unit (ER-FEM) and a multi-scale convolution feature fusion unit (MS-CFFM), The edge enhancement feature extraction unit extracts edge information in the image through Laplace operator and Sobel operator; The feature fusion module specifically implements the following steps: Based on the difference information injection unit (DIIM) and the alternating common information injection unit (ACIIM) in the ATFuse network, a common mode injection fusion unit (CMIFM) is constructed, the common mode injection fusion unit is replaced by the alternating common information injection unit in the original ATFuse network, and feature fusion is performed, The feature fusion is: the feature fusion module comprises a difference information injection unit (DIIM) and a common mode injection fusion unit (CMIFM), the same components and difference components between the infrared and visible light images are fused through the difference information injection unit and the common mode injection fusion unit, and the common mode injection fusion unit is applied to complete the feature fusion.

2. The multi-modal feature difference based infrared and visible image fusion system of claim 1, wherein, The multi-scale convolution feature fusion unit uses convolution kernels of different scales to extract different scale features of the image, The infrared and visible light image features extracted by the edge enhancement feature extraction unit and the multi-scale convolution feature fusion unit are spliced in the channel dimension as the output of the feature extraction module, The edge enhancement feature extraction unit extracts edge features and outputs a feature map F1, and the multi-scale convolution feature fusion unit extracts multi-scale features and outputs a feature map F2: F1=H×W×C1 F2=H×W×C2 where H and W are the spatial dimensions of the image, C1 and C2 are the number of channels, F1 and F2 are spliced in the channel dimension to obtain a new feature map F concat The formula is: F concat = concat(F1,F2) The feature map F concat has a size of H x W x (C1 + C2).

3. The infrared and visible image fusion system based on multi-modal feature difference according to claim 1, characterized in that, The pre-reconstruction module is used for feature reconstruction in the training stage, and the pre-reconstruction module comprises one convolution layer and two residual blocks; in the network training stage, the infrared and visible image features extracted by the feature extraction module are sent to the pre-reconstruction module, and the pre-reconstruction module reversely restores the infrared and visible image features into the original input infrared and visible images, The steps in the training stage are as follows: First, the infrared and visible image features are subjected to one convolution operation, and local information is extracted for subsequent processing; then, the features are further transmitted to two residual blocks, and the expression ability of the features is further improved through deep modeling, Finally, the pre-reconstruction image generated by the pre-reconstruction module is subjected to pixel-by-pixel loss calculation with the original input image, and through the optimization of the loss function, the information loss is reduced, and the formula is as follows: wherein, Ivis is a visible image reconstructed by the model from the visible features extracted by the feature extraction module, vi Ivis is an original visible image, Lvis is an L1 loss between the pre-reconstructed visible image and the original visible image, Iir is an infrared image reconstructed by the model from the infrared features extracted by the feature extractor, ir Iir is an original infrared image, Lir is an L1 loss between the reconstructed infrared image and the input infrared image, reconstruct Lpre is a pre-reconstruction module loss; The infrared and visible features are respectively subjected to grouped convolution, 3*3 convolution, layer normalization and Hardwish activation function to obtain embedded features, The pre-reconstruction module is optimized according to the loss function, and the pre-reconstruction module only participates in calculation in the training stage, and in the inference stage, the module is removed and no longer participates in calculation.

4. The multi-modal feature difference based infrared and visible image fusion system according to claim 1 or 2, characterized in that, The up-sampling of the M4 is performed through the pixel rearrangement operation of the up-sampling technology to restore the high resolution of the image, The image reconstruction module generates high-quality fusion images through multi-level feature processing, optimization and reconstruction, The image reconstruction module introduces the loss function of the ATFuse architecture training, and adds the SSIM loss and the loss of the pre-reconstruction module as the total loss function: L total = l ATFuse + αL ssim (I f , I vi ) + (1 - α)L ssim (I f , I ir ) + βL reconstruct where l ATFuse is the loss function adopted for the ATfuse neural network training, I f is the fused image output by the model, I vi is the original visible light image, I ir is the original infrared image, a is a hyperparameter ranging from 0 to 1 used to balance the weight of the two items of structural similarity loss, and β is a hyperparameter describing the contribution of the reconstruction loss to the whole, which is required to be a positive number, L ssim (I f vi )=1-SSIM(I f ,I vi )L ssim (I f ,I ir )=1-SSIM(I f ,I ir )​ where SSIM(I f ,I vi ) is the structural similarity between the fused image and the visible light image, and SSIM(I f ,I ir ) is the structural similarity between the fused image and the infrared image.

Images