Infrared and visible light image fusion system based on multi-modal characteristic difference

Abstract

The invention provides an infrared and visible light image fusion system based on multi-modal characteristic difference. According to the system, the quality and definition of infrared and visible light image fusion are improved, and firstly, multi-level and multi-scale features are extracted through a feature extraction module; the method comprises the following steps: firstly, carrying out feature fusion on two modal images, then, carrying out feature enhancement through a TAFEM module, in a feature fusion module, carrying out fusion of same components and different components between the two modal images through DIIM and CMIFM attention fusion modes, and finally, carrying out refining and up-sampling on the fused features through an image reconstruction module to generate a high-quality fused image. According to the method, the texture information and multi-scale feature parallel extraction mode is innovatively adopted, the feature extraction quality of the model is improved, the TAFEM module is further provided, information loss in the feature extraction stage is reduced, the CMIFM module is obtained through improvement on the basis of a traditional DIIM module and an ACIIM module, and the method is suitable for being applied to the field of feature extraction. And the defect of common information extraction of a traditional cross attention mechanism is overcome.

Description

Technical Field

[0001] The present invention relates to the field of infrared and visible light image fusion, and in particular to an infrared and visible light image fusion system based on multimodal feature differences. Background Art

[0002] Infrared and visible light image fusion technology aims to effectively combine the information of infrared images and visible light images to generate a new image that incorporates the advantages of both. Infrared images can capture heat source information and are suitable for low-light or nighttime environments, while visible light images provide rich environmental details and are suitable for daytime or well-lit scenes. By fusing these two images, detection and navigation of concealed targets can be achieved, thereby improving the system's safety and combat capability.

[0003] In recent years, with the rapid development of deep learning technology, infrared and visible light image fusion systems based on multimodal feature differences have gradually become a research hotspot. These methods utilize the powerful feature extraction and representation capabilities of deep neural networks to automatically learn complex features in images and achieve high-quality image fusion.

[0004] In addition, infrared and visible light image fusion technology has broad application prospects in multiple fields. In intelligent monitoring, fusion technology can provide clearer images in complex environments and improve the reliability of the monitoring system. In medical image analysis, fusion technology helps to improve the accuracy of diagnosis. In autonomous driving, fusion technology can enhance the vehicle's perception of the surrounding environment and improve driving safety. However, existing fusion methods still face some challenges, such as how to retain the information in the source image, reduce information loss, and how to better focus on and process the difference components and common components between images of different modalities. Summary of the Invention

[0005] In order to solve the above problems, the present invention starts from infrared and visible light images, extracts multi-level and multi-scale features through the feature extraction module, and then enhances the features through the TAFEM module to reduce information loss. In the feature fusion module, the same components and difference components between the two modal images are fused through the attention fusion method of DIIM and CMIFM. Finally, the image reconstruction module refines and upsamples the fused features to generate a high-quality fused image. Based on this, the present invention proposes an infrared and visible light image fusion system based on multimodal feature differences.

[0006] To this end, the technical solution adopted in the present invention is as follows:

[0007] M1, Feature Extraction Module: Input the original infrared image and visible light image into the edge enhancement feature extraction unit and the multi-scale convolution feature fusion unit respectively, and perform feature extraction on the two images. The extracted original infrared and visible light image features are spliced ​​in the channel dimension. The output of the feature extraction module is the spliced ​​infrared image features and visible light image features.

[0008] M2. Pre-reconstruction module: The input of the pre-reconstruction module is the infrared and visible light image features extracted by the feature extraction module. The extracted infrared and visible light image features are reversely restored to the original input infrared and visible light images, and pixel-by-pixel loss calculation is performed with the original input images. The loss function is used for optimization. The infrared and visible light features are respectively subjected to group convolution, 3×3 convolution, layer normalization and Hardwish activation function to obtain embedded features.

[0009] M3, feature fusion module: The input of the feature fusion module is the embedded features obtained by the pre-reconstruction through group convolution, 3×3 convolution, layer normalization and Hardwish activation function. The feature fusion module fuses the same components and difference components between the infrared and visible light images to complete feature fusion;

[0010] M4, reconstruction module: upsamples the fusion features output by the feature fusion module through pixel rearrangement operation to restore the high resolution of the image. The reconstruction module generates a high-quality fusion image through multi-level feature processing, optimization and reconstruction.

[0011] Furthermore, the feature extraction of M1 is specifically as follows: the feature extraction module mainly consists of two core parts, namely the edge enhancement feature extraction unit (ER-FEM) and the multi-scale convolutional feature fusion unit (MS-CFFM).

[0012] The edge enhancement feature extraction unit extracts features from edge information in the image using a Laplace operator and a Sobel operator.

[0013] Furthermore, the multi-scale convolution feature fusion unit uses convolution kernels of different scales to extract features of different scales of the image.

[0014] The infrared and visible light image features extracted by the edge enhancement feature extraction unit and the multi-scale convolution feature fusion unit are spliced ​​in the channel dimension as the output of the feature extraction module.

[0015] The ER-FEM unit extracts edge features and outputs a feature map F1. The MS-CFFM unit extracts multi-scale features and outputs a feature map F2.

[0016] F1=H×W×C1 F2=H×W×C2

[0017] Among them, H and W are the spatial dimensions of the image, C1 and C2 are the number of channels,

[0018] Concatenate F1 and F2 in the channel dimension to obtain a new feature map F concat , the formula is:

[0019] F concat =concat(F1,F2)

[0020] The feature map F concat The dimensions are H×W×(C1+C2).

[0021] Furthermore, the pre-reconstruction module is used for feature reconstruction in the training phase. The pre-reconstruction module includes a convolution layer and two residual blocks. During the network training phase, the infrared and visible light image features extracted by the feature extraction module are fed into the pre-reconstruction module, which reversely restores the infrared and visible light image features to the original input infrared and visible light images.

[0022] The steps of the training phase are as follows:

[0023] First, the infrared and visible light image features undergo a convolution operation to extract local information for subsequent processing. After that, the features are further passed to two residual blocks to further enhance the feature expression capability through deep modeling.

[0024] Finally, the pre-reconstructed image generated by the pre-reconstruction module is subjected to pixel-by-pixel loss calculation with the original input image. By optimizing the loss function, information loss is reduced. The formula is:

[0025]

[0026] in, is the visible light image reconstructed by the model based on the visible light features extracted by the feature extraction module, I vi is the original visible light image, is the L1 loss between the pre-reconstructed visible light image and the original visible light image, is the infrared image reconstructed by the model based on the infrared features extracted by the feature extractor, I ir is the original infrared image, is the L1 loss between the reconstructed infrared image and the input infrared image, L reconstruct is the loss of pre-reconstruction module;

[0027] The infrared and visible light features are subjected to group convolution, 3×3 convolution, layer normalization and Hardwish activation function respectively to obtain embedded features.

[0028] The pre-reconstruction module is optimized according to the loss function. The pre-reconstruction module only participates in the calculation during the training phase. During the inference phase, the module will be removed and no longer participates in the calculation.

[0029] Furthermore, the feature fusion module is specifically implemented as follows:

[0030] Based on the difference information injection unit (DIIM) and the alternating common information injection unit (ACIIM) in the ATFuse network, a common mode injection fusion unit (CMIFM) is constructed. The common mode injection fusion module replaces the alternating common information injection module in the original ATFuse network to perform feature fusion.

[0031] The feature fusion is as follows: the feature fusion module consists of a difference information injection unit (DIIM) and a common mode injection fusion unit (CMIFM). The same components and difference components between the infrared and visible light images are fused through the difference information injection unit and the common mode injection fusion unit, and are applied to the common mode injection fusion unit to complete the feature fusion.

[0032] Furthermore, the upsampling of M4 performs pixel rearrangement operation through upsampling technology to restore the high resolution of the image.

[0033] The image reconstruction module generates high-quality fused images through multi-level feature processing, optimization and reconstruction.

[0034] The image reconstruction module introduces the loss function of ATFuse architecture training and adds SSIM loss and TAFEM loss as the total loss function:

[0035] L total =L ATFuse +αL ssim (I f ,I vi )+(1-α)L ssim (I f ,I ir )+βL reconstruct

[0036] Among them, L ATFuse is the loss function used in ATFuse neural network training, I f is the fused image output by the model, I vi is the original visible light image, I ir is the original infrared image, α is a hyperparameter ranging from 0 to 1, which is used to balance the weights of the two structural similarity losses, L reconstruct is the reconstruction loss, β is a hyperparameter that describes the contribution of the reconstruction loss to the overall performance and is required to be a positive number.

[0037] L ssim(I f ,I vi )=1-SSIM(I f ,I vi )L ssim (I f ,I ir )=1-SSIM(I f ,I ir )

[0038] Among them, SSIM(I f ,I vi ) is the structural similarity between the fusion image and the visible light image, SSIM(I f ,I ir ) is the structural similarity between the fused image and the infrared image.

[0039] Compared with the prior art, the advantages of the present invention are:

[0040] 1. The present invention adopts a parallel extraction method of texture information and multi-scale features, using two parallel branches to extract and integrate features at different levels of the image, while not affecting the efficiency of model reasoning, while improving the feature extraction quality of the model.

[0041] 2. The pre-reconstruction module designed in the present invention restores the source image in the feature extraction stage, performs supervised training on the feature extraction module, enhances the network's ability to retain the feature information of the source image, minimizes information loss in the feature extraction stage, and retains as much information from the source image as possible in the fused image.

[0042] 3. This invention draws on the concepts of common mode and differential mode in signal analysis, and applies the method of extracting the common mode components of two signals to the traditional cross-attention mechanism, significantly enhancing the cross-attention module's ability to extract common features between the two modalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0044] Figure 1 This is a flowchart of the infrared and visible light image fusion of the present invention;

[0045] Figure 2 Schematic diagram of a visible light image reconstructed by the inference stage model of the present invention based on the extracted features;

[0046] Figure 3A schematic diagram of an infrared image reconstructed by the inference stage model of the present invention based on the extracted features;

[0047] Figure 4 Schematic diagram of the final fused image in the reasoning stage of the present invention. DETAILED DESCRIPTION

[0048] To achieve the above objectives, the present invention is implemented through the following technical solutions. The present invention provides an infrared and visible light image fusion system based on multimodal feature differences, the method comprising:

[0049] M1: Input the original infrared image and visible light image into the edge enhancement feature extraction unit and the multi-scale convolution feature fusion unit respectively, and perform feature extraction on the two images. The extracted original infrared and visible light image features are spliced ​​in the channel dimension respectively. The output of the feature extraction module is the spliced ​​infrared image features and visible light image features.

[0050] In the feature extraction module, an innovative multi-module collaborative working mode is adopted, combined with different feature extraction strategies, to achieve the interaction and fusion of infrared and visible light image features at different scales and levels.

[0051] The feature extraction module mainly consists of two core parts: edge enhancement feature extraction unit (ER-FEM) and multi-scale convolutional feature fusion unit (MS-CFFM)

[0052] First, the edge enhancement feature extraction unit extracts edge information from the image using the Laplacian operator and the Sobel operator. This unit focuses on capturing details and edge features in the image. In particular, edge information is often ignored or difficult to distinguish in infrared images and low-contrast visible light images. Therefore, by enhancing the edge information of the image, the ER-FEM module improves the image recognition.

[0053] Laplace operator: As a second-order differential operator, the Laplace operator can efficiently capture changes in edges and details. It is used to initialize the second-order derivative of the image and can detect sudden changes in the grayscale value in the image, thereby highlighting the edge information. By cornering the image, an image reflecting the second-order gradient information of the image can be obtained, thereby highlighting the changes in the edges and details in the image.

[0054] In infrared images, after applying the Laplace operator, the edges of heat sources in the infrared images will be clearly detected, which is helpful for the positioning and segmentation of targets.

[0055] In visible light images, the Laplace operator can be used to extract the simulation and great changes of the image and enhance the structural features of the image.

[0056] Sobel operator: By calculating the first-order derivative of the image in the horizontal and vertical directions, the edge information in the image is highlighted.

[0057] The horizontal Sobel operator is used to detect horizontal edges:

[0058]

[0059] Vertical Sobel operator, used to detect vertical edges:

[0060]

[0061] Through these two operators, the gradient of the image in the horizontal and vertical directions can be calculated to obtain the edge information of the image.

[0062] The horizontal direction G x and vertical direction G y The gradients of are combined to obtain the strength of the edge, which is usually reflected by calculating the amplitude of the gradient:

[0063]

[0064] Through the Sobel operator, important edge features can be extracted from infrared and visible light images to help complete subsequent image processing tasks.

[0065] Secondly, the multi-scale convolutional feature fusion unit uses convolution kernels of different scales, such as 3×3 and 5×5, to extract features of different scales of the image. In the image fusion task, features of different scales can help the model fully understand the overall structure and local details of the image. Multi-scale feature fusion enables the model to simultaneously focus on large-scale structural features and subtle local changes in the image, thereby enhancing the global consistency and local detail expression of the image.

[0066] Small-scale convolution kernels, such as 3×3, can capture local details and small-scale texture information in the image. For details such as edges, textures, and small objects in the image, the 3×3 convolution kernel can provide better extraction effects.

[0067] Large-scale convolution kernels, such as 5×5, can capture a wider range of contextual information and global structural features, helping to extract macro information such as large objects and backgrounds in images.

[0068] Through this multi-scale feature extraction, the model can simultaneously obtain local details and global structural information of the image.

[0069] Finally, the features extracted by the two units are concatenated in the channel dimension as the output of the feature extraction module.

[0070] The ER-FEM module extracts edge features and outputs a feature map F1. The MS-CFFM module extracts multi-scale features and outputs a feature map F2.

[0071] F1=H×W×C1 F2=H×W×C2

[0072] Among them, H and W are the spatial dimensions of the image, C1 and C2 are the number of channels,

[0073] Concatenate F1 and F2 in the channel dimension to obtain a new feature map F concat , the formula is:

[0074] F concat =concat(F1,F2)

[0075] The concatenated feature map F concat The dimensions are H×W×(C1+C2).

[0076] M2. The infrared and visible light image features extracted by the feature extraction module are sent to the pre-reconstruction module. The pre-reconstruction module reversely restores the extracted infrared and visible light image features to the original input infrared and visible light images, performs pixel-by-pixel loss calculation with the original input images, and optimizes them through the loss function. The infrared and visible light features are respectively subjected to group convolution, 3×3 convolution, layer normalization and Hardwish activation function to obtain embedded features.

[0077] In order to optimize the performance of the feature extraction module and minimize information loss, a pre-reconstruction module, referred to as the TAFEM module, is designed for feature reconstruction in the training phase, thereby enhancing the perception and expression capabilities of the feature extraction module through a supervision mechanism. This module consists of a layer of convolution and two residual blocks. During the network training phase, the infrared and visible light features extracted by the feature extraction module are fed into the pre-reconstruction module, which reversely restores these features to the original input infrared and visible light images.

[0078] First, the infrared and visible light features undergo a convolution operation to adjust their dimension and scale and extract local information for subsequent processing. After that, the infrared and visible light features are further passed to two residual blocks. The expressive power of the features is further improved through deep modeling. The design of the residual block effectively alleviates the gradient vanishing problem in deep networks and can capture high-order semantic information, providing stronger expressiveness and robustness for the infrared and visible light features.

[0079] Finally, the pre-reconstructed image generated by the pre-reconstruction module is subjected to pixel-by-pixel loss calculation with the original input image. By optimizing the loss function, the feature extractor is guided to be more comprehensive and information-rich in the feature extraction process, thereby reducing information loss.

[0080]

[0081] in, is the visible light image reconstructed by the model based on the features extracted by the feature extraction module, I vi is the original visible light image, is the L1 loss between the pre-reconstructed visible light image and the original visible light image, is the infrared image reconstructed by the model based on the features extracted by the feature extractor, I ir is the original infrared image, is the L1 loss between the pre-reconstructed infrared image and the input infrared image,

[0082] The two input features, infrared features and visible light features extracted by the feature extraction module, undergo feature embedding. First, a group convolution with a convolution kernel size of 3×3 and a group size of 64 is performed. Then, 3×3 convolution, layer normalization, and Hardwish activation function are performed to obtain embedded features. This operation converts the input features into tokens, preparing for subsequent feature fusion.

[0083] The pre-reconstruction module is optimized based on the loss function. It participates in calculations only during the training phase, optimizing the performance of the feature extractor through supervision. During the inference phase, this module is removed and no longer participates in calculations, thereby improving the network's operation speed. Through this design, we not only enhance the network's learning ability, but also ensure the efficiency of the inference process.

[0084] S3. Input the embedded features into the feature fusion module. Through the feature fusion module, the common mode information and differential mode information between the infrared and visible light images are extracted to complete the feature fusion.

[0085] Based on the difference information injection unit (DIIM) and the alternating common information injection unit (ACIIM) in the ATFuse network, a common mode injection fusion unit (CMIFM) is constructed. The common mode injection fusion unit replaces the alternating common information injection unit in the original ATFuse network to perform feature fusion. The specific structure of the common mode injection fusion unit is shown in Figure 1 ,

[0086] Based on the common-mode and differential-mode signal decomposition method in analog circuit signal analysis, in traditional signal analysis, signals can be divided into common-mode signals and differential-mode signals, which represent the similar and different parts of the two signals respectively. In the image fusion task, the present invention adopts a similar decomposition method to fuse the same components and different components between infrared and visible light images, thereby optimizing the information fusion process.

[0087] This innovative idea is applied to the feature fusion module, which consists of a difference information injection unit (DIIM) and a common mode injection fusion unit (CMIFM). The common mode information and differential mode information between the infrared and visible light images are extracted through the difference information injection unit and the common mode injection fusion unit, and the extracted common mode information and differential mode information are applied to the common mode injection fusion unit to complete the feature fusion.

[0088] M4, upsampling is performed through pixel rearrangement operation to restore the high resolution of the image. The reconstruction module generates a high-quality fused image through multi-level feature processing, optimization and reconstruction.

[0089] In this step, pixel rearrangement is performed through upsampling technology to restore the high resolution of the image. This operation effectively reduces information loss. The residual block further enhances the expressiveness of image features through deep modeling and optimizes the image reconstruction effect.

[0090] Ultimately, the entire image reconstruction module generates a high-quality fused image through multi-level feature processing, optimization, and reconstruction. This module not only effectively integrates the information in the infrared and visible light images, but also ensures the highest level of image quality and detail through meticulous feature refinement and upsampling operations.

[0091] The loss function of ATFuse architecture training is introduced, and SSIM loss and TAFEM loss are added as the total loss function:

[0092] L total =l ATFuse +αL ssim (I f ,I vi )+(1-α)L ssim (I f ,I ir )+βL reconstruct

[0093] Among them, l ATFuse is the loss function used in ATFuse neural network training, I f is the fused image output by the model, I vi is the original visible light image, I ir is the original infrared image, α is a hyperparameter ranging from 0 to 1, which is used to balance the weights of the two structural similarity losses, L reconstruct is the reconstruction loss, β is a hyperparameter that describes the contribution of the reconstruction loss to the overall performance and is required to be a positive number.

[0094] L ssim (I f ,I vi )=1-SSIM(I f ,Ivi )L ssim (I f ,I ir )=1-SSIM(I f ,I ir )

[0095] Among them, SSIM(I f ,I vi ) is the structural similarity between the fusion image and the visible light image, SSIM(I f ,I ir ) is the structural similarity between the fused image and the infrared image.

[0096] Comparative test:

[0097] Five existing classic infrared and visible light image fusion methods were selected, trained on the MSRS dataset and tested on the MSRS test set. Six common image fusion evaluation indicators were selected to evaluate the fusion results of each method. The average value of all fusion results was taken for the calculation of the indicators.

[0098]

[0099]

[0100] By comparing with the existing seven SOTA methods, i.e. the most advanced methods, the method of the present invention is effective and competitive. In particular, the mutual information is high, which proves that the information extraction and retention mechanism in the feature extraction stage is effective.

[0101] In summary, the advantages of the present invention are that it starts from infrared and visible light images, extracts multi-level and multi-scale features through the feature extraction module, then enhances the features through the TAFEM module to reduce information loss, and in the feature fusion module, the same components and difference components between the two modal images are fused through the attention fusion method of DIIM and CMIFM. Finally, the image reconstruction module refines and upsamples the fused features to generate a high-quality fused image. The overall architecture extracts the common-mode components and differential-mode components of different modal features through the collaborative work of multiple modules, and finally fuses them with each other, making full use of the complementarity of infrared and visible light images, ensuring the detail retention and information accuracy of the image, so that the final generated fused image has a significant improvement in visual effect.

[0102] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An infrared and visible light image fusion system based on multimodal feature differences, characterized in that: The system includes: M1, Feature Extraction Module: Input the original infrared image and visible light image into the edge enhancement feature extraction unit and the multi-scale convolution feature fusion unit respectively, and perform feature extraction on the two images. The extracted original infrared and visible light image features are spliced ​​in the channel dimension. The output of the feature extraction module is the spliced ​​infrared image features and visible light image features. M2, Pre-reconstruction Module: The input of the pre-reconstruction module is the infrared and visible light image features extracted by the feature extraction module. The extracted infrared and visible light image features are reversely restored to the original input infrared and visible light images, and pixel-by-pixel loss calculation is performed with the original input images. The loss function is optimized and the infrared and visible light features are subjected to group convolution, 3×3 convolution, layer normalization and Hardwish activation function respectively to obtain embedded features. M3, feature fusion module: The input of the feature fusion module is the embedded features obtained by the pre-reconstruction through group convolution, 3×3 convolution, layer normalization and Hardwish activation function. The feature fusion module fuses the same components and difference components between the infrared and visible light images to complete feature fusion; M4, reconstruction module: upsamples the fusion features output by the feature fusion module through pixel rearrangement operation to restore the high resolution of the image. The reconstruction module generates a high-quality fusion image through multi-level feature processing, optimization and reconstruction.

2. The infrared and visible light image fusion system based on multimodal feature differences according to claim 1, characterized in that: The feature extraction of M1 is specifically as follows: the feature extraction module mainly consists of two core parts, namely the edge enhancement feature extraction unit (ER-FEM) and the multi-scale convolutional feature fusion unit (MS-CFFM). The edge enhancement feature extraction unit extracts features from edge information in the image using a Laplace operator and a Sobel operator.

3. The infrared and visible light image fusion system based on multimodal feature differences according to claim 2, characterized in that: The multi-scale convolution feature fusion unit uses convolution kernels of different scales to extract features of different scales of the image. The infrared and visible light image features extracted by the edge enhancement feature extraction unit and the multi-scale convolution feature fusion unit are spliced ​​in the channel dimension as the output of the feature extraction module. The edge enhancement feature extraction unit extracts edge features and outputs feature map F1. The multi-scale convolution feature fusion unit extracts multi-scale features and outputs feature map F2: F1=H×W×C1 F2=H×W×C2 Among them, H and W are the spatial dimensions of the image, C1 and C2 are the number of channels, Concatenate F1 and F2 in the channel dimension to obtain a new feature map F concat , the formula is: F concat =concat(F1,F2) The feature map F concat The dimensions are H×W×(C1+C2).

4. The infrared and visible light image fusion system based on multimodal feature differences according to claim 1, characterized in that: The pre-reconstruction module is used for feature reconstruction in the training phase. The pre-reconstruction module includes a convolution layer and two residual blocks. During the network training phase, the infrared and visible light image features extracted by the feature extraction module are fed into the pre-reconstruction module, which reversely restores the infrared and visible light image features to the original input infrared and visible light images. The steps of the training phase are as follows: First, the infrared and visible light image features undergo a convolution operation to extract local information for subsequent processing. After that, the features are further passed to two residual blocks to further enhance the feature expression capability through deep modeling. Finally, the pre-reconstructed image generated by the pre-reconstruction module is subjected to pixel-by-pixel loss calculation with the original input image. By optimizing the loss function, information loss is reduced. The formula is: in, is the visible light image reconstructed by the model based on the visible light features extracted by the feature extraction module, I vi is the original visible light image, is the L1 loss between the pre-reconstructed visible light image and the original visible light image, is the infrared image reconstructed by the model based on the infrared features extracted by the feature extractor, I ir is the original infrared image, is the L1 loss between the reconstructed infrared image and the input infrared image, L reconstruct is the loss of pre-reconstruction module; The infrared and visible light features are subjected to group convolution, 3×3 convolution, layer normalization and Hardwish activation function respectively to obtain embedded features. The pre-reconstruction module is optimized according to the loss function. The pre-reconstruction module only participates in the calculation during the training phase. During the inference phase, the module will be removed and no longer participate in the calculation.

5. The infrared and visible light image fusion system based on multimodal feature difference according to claim 1, characterized in that: The specific implementation steps of the feature fusion module are: Based on the difference information injection unit (DIIM) and the alternating common information injection unit (ACIIM) in the ATFuse network, a common mode injection fusion unit (CMIFM) is constructed. The common mode injection fusion unit replaces the alternating common information injection unit in the original ATFuse network to perform feature fusion. The feature fusion is as follows: the feature fusion module consists of a difference information injection unit (DIIM) and a common mode injection fusion unit (CMIFM). The same components and difference components between the infrared and visible light images are fused through the difference information injection unit and the common mode injection fusion unit, and are applied to the common mode injection fusion unit to complete the feature fusion.

6. The infrared and visible light image fusion system based on multimodal feature differences according to claim 1 or 3, characterized in that: The upsampling of M4 performs pixel rearrangement operation through upsampling technology to restore the high resolution of the image. The image reconstruction module generates high-quality fused images through multi-level feature processing, optimization and reconstruction. The image reconstruction module introduces the loss function of ATFuse architecture training, and adds the SSIM loss and the loss of the pre-reconstruction module as the total loss function: L total =l ATFuse +αL ssim (I f ,I vi )+(1-α)L ssim (I f ,I ir )+βL reconstruct Among them, l ATFuse is the loss function used in ATFuse neural network training, I f is the fused image output by the model, I vi is the original visible light image, I ir is the original infrared image, α is a hyperparameter ranging from 0 to 1, which is used to balance the weights of the two structural similarity losses, and β is a hyperparameter that describes the contribution of the reconstruction loss to the overall loss, which is required to be a positive number. L ssim (I f ,I vi )=1-SSIM(I f ,I vi )L ssim (I f ,I ir )=1-SSIM(I f ,I ir ) Among them, SSIM(I f ,I vi ) is the structural similarity between the fusion image and the visible light image, SSIM(I f ,I ir ) is the structural similarity between the fused image and the infrared image.

Images