Image fusion method and system based on depth estimation and dual-module attention

Abstract

The application relates to the technical field of image processing, in particular to an image fusion method and system based on depth estimation and double-module attention; the method comprises the following steps: processing infrared images and visible light images through a depth estimation module, respectively performing feature mapping on the infrared images and the visible light images through Restormer blocks, respectively performing frequency domain modeling and enhancing on shallow features of the infrared images and shallow features of the visible light images through a Fourier neural operator module, respectively processing frequency domain enhanced features of the infrared images and frequency domain enhanced features of the visible light images through a basic feature extraction module, respectively processing the frequency domain enhanced features of the infrared images and the frequency domain enhanced features of the visible light images through a detail feature extraction module, performing fusion processing through a double-module attention fusion layer, decoding and reconstructing through a decoder, and outputting a fusion image; the application can improve the quality of the fusion image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to an image fusion method and system based on depth estimation and dual-module attention. Background Technology

[0002] Multimodal information fusion, as a key technology, has a profound impact on many cutting-edge fields of computer vision. By integrating data sources from different modalities (such as infrared, visible light, and depth information), it significantly improves the perception capabilities and decision-making quality of a system. In many fusion applications, tasks such as 3D reconstruction, autonomous driving, and medical diagnosis particularly rely on the collaborative processing of multimodal information to address the limitations of single-modal perception in complex scenes. Among these, multimodal image fusion, as a fundamental task, not only has significant theoretical value but also plays a crucial role in enhancing image information richness and improving the robustness of target recognition.

[0003] In the early stages of image fusion technology development, researchers typically decomposed source images according to specific rules. These rules mainly included traditional signal processing techniques such as edge detection operators and multi-scale transformations. A common implementation involved using fixed convolution kernels or signal processing operators to process the image and extract useful structural information. While these methods achieved some success in certain applications, their reliance on manually designed fixed operators made them difficult to adapt to complex and changing real-world environments, posing significant challenges in terms of versatility and robustness. To overcome these limitations, various representation learning-based decomposition methods have been introduced into the image fusion field in recent years. For example, sparse representation (SR) and low-rank representation (LRR) have become two representative decomposition strategies in image fusion tasks. These methods decompose images into low-rank components (foundation) and sparse components (details) using learned dictionaries or projection matrices, achieving significant progress in improving fusion performance. However, these methods usually require complex optimization calculations to solve the SR or LRR model, resulting in high computational costs and making it difficult to meet the efficiency requirements of practical applications.

[0004] With the continuous development of deep learning technology, an increasing number of learning-based decomposition methods have been proposed and applied to image fusion tasks. These methods typically employ lightweight network modules capable of efficiently extracting detailed features from images. Compared to traditional methods, deep learning-based decomposition models can automatically optimize the loss function through end-to-end training, thereby improving fusion results. Nevertheless, with the increasing complexity of network structures and the introduction of new modules, the number of model parameters and training difficulty also increase. Furthermore, while some new modules improve feature representation capabilities, their structure and loss function design still face certain challenges.

[0005] Existing image fusion methods still suffer from several limitations. First, most methods focus only on fusing infrared and visible light images, neglecting the crucial role of depth estimation in scene understanding, resulting in a lack of spatial structural information in the fusion results. Second, traditional attention mechanisms often employ single spatial or channel attention, failing to fully capture the complex relationships between cross-modal features and limiting the effectiveness of feature representation. Furthermore, existing methods typically employ simple concatenation or addition operations during feature fusion, lacking adaptive feature recalibration mechanisms and failing to dynamically adjust the fusion strategy based on the importance of input features, thus affecting fusion quality. As network depth increases, gradient vanishing and feature degradation become increasingly prominent, making the design of effective residual connections and feature enhancement mechanisms a pressing technical challenge. Simultaneously, pure spatial domain-based feature extraction methods struggle to effectively capture the global structure and periodic patterns of images, limiting their ability to represent details such as texture and edges, thus hindering further improvements in fusion performance. Finally, existing fusion methods are primarily limited to processing two-dimensional image features, failing to fully utilize the three-dimensional structural information of the scene, leading to structural inconsistencies and spatial distribution deviations in fusion results in weakly textured regions and complex scenes with occlusion. Therefore, designing an adaptive multi-scale feature extraction mechanism, enhancing global context awareness, and effectively utilizing depth information to guide the fusion process have become key technical challenges in improving the quality of infrared and visible light image fusion. Summary of the Invention

[0006] To address the aforementioned issues, this invention provides an image fusion method and system based on depth estimation and dual-module attention. By introducing a depth estimation module comprising a depth estimation network and a depth feature extraction network, the depth estimation network provides structured depth information. The depth feature extraction network generates channel representations through two-stage convolution and depth-specific channel attention, injecting depth information early on using lightweight residuals. Simultaneously, in the later stages of encoding, a dual-module attention fusion layer deeply fuses the data with basic features, enhancing spatial geometric perception. The dual-module attention fusion layer includes a feature recalibration module and a global enhancement fusion module: the feature recalibration module achieves deep fusion of depth and basic features through spatial-channel cross-attention and adaptive fusion; the global enhancement fusion module enhances high-frequency details such as texture and edges through cross-modal attention. The encoder also includes a Fourier neural operator module for frequency domain global dependency modeling. This mechanism significantly improves the adaptive fusion and detail preservation of multimodal features, enhancing robustness and structural consistency in complex scenes with weak textures and occlusions, generating fusion results with clear hierarchy and rich detail.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] On one hand, embodiments of the present invention provide an image fusion method based on depth estimation and dual-module attention, the method comprising the following steps:

[0009] S100, acquire infrared image, visible light image and pre-trained image fusion model; wherein, the image fusion model includes a depth estimation module, a Restormer block, a Fourier neural operator module, a basic feature extraction module based on Transformer architecture, a detail feature extraction module based on reversible neural network architecture, a dual-module attention fusion layer and a decoder.

[0010] S200, the infrared image and the visible light image are input into a pre-trained image fusion model, and the infrared image and the visible light image are processed by the depth estimation module to output the depth estimation information of the infrared image and the depth estimation information of the visible light image;

[0011] S300, the feature mapping of the infrared image and the visible light image is performed by the Restormer block respectively, and the shallow features of the infrared image and the shallow features of the visible light image are output.

[0012] S400, the shallow features of the infrared image and the shallow features of the visible light image are modeled and enhanced in the frequency domain through the Fourier neural operator module, and the frequency domain enhancement features of the infrared image and the visible light image are output.

[0013] S500, the basic feature extraction module processes the frequency domain enhancement features of the infrared image and the frequency domain enhancement features of the visible light image respectively, and outputs the basic features of the infrared image and the basic features of the visible light image;

[0014] S600, the detail feature extraction module processes the frequency domain enhancement features of the infrared image and the frequency domain enhancement features of the visible light image respectively, and outputs the detail features of the infrared image and the detail features of the visible light image.

[0015] S700 performs fusion processing on the basic features of the infrared image, the basic features of the visible light image, the detailed features of the infrared image, the detailed features of the visible light image, the depth estimation information of the infrared image, and the depth estimation information of the visible light image through the dual-module attention fusion layer, and outputs the basic features and detailed features after cross-modal fusion.

[0016] S800: The decoder decodes and reconstructs the basic features and detailed features after cross-modal fusion, and outputs the fused image.

[0017] On the other hand, embodiments of the present invention provide an image fusion system based on depth estimation and dual-module attention, comprising:

[0018] At least one processor;

[0019] At least one memory for storing at least one program;

[0020] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0021] On the other hand, embodiments of the present invention provide a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the above-described method.

[0022] The embodiments of the present invention have the following beneficial effects:

[0023] (1) A depth information estimation mechanism is proposed: the depth estimation network and the depth feature extraction network of the depth estimation module obtain the depth structure representation through the depth image, perform early depth fusion before the Fourier neural operator module, and perform deep fusion with the basic features through the feature recalibration module in the dual-module attention fusion layer, so that the network can better utilize the three-dimensional structural information of the scene and enhance the perception of spatial geometric relationships.

[0024] (2) A dual-module attention fusion layer is proposed: the feature recalibration module realizes the deep fusion of basic features and deep features, and the channel attention and spatial attention mechanisms are used for adaptive weight allocation; the global enhancement fusion module realizes the cross-attention fusion of basic features and detailed features, and the global context enhancement strategy is adopted to effectively improve the model's adaptive fusion ability and detail preservation ability for multimodal features.

[0025] (3) The method of this invention achieves the best fusion performance compared with the 32 mainstream methods on four datasets: RoadScene, LLVIP, M3FD and TNO. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating the image fusion method based on depth estimation and dual-module attention in an embodiment of the present invention.

[0028] Figure 2 This is a network structure diagram of the image fusion method based on depth estimation and dual-module attention in an embodiment of the present invention;

[0029] Figure 3 This is a framework diagram of the depth estimation module proposed in this invention;

[0030] Figure 4 This is a framework diagram of the feature recalibration module of the dual-module attention fusion layer proposed in this invention;

[0031] Figure 5 This is a framework diagram of the global enhancement fusion module of the dual-module attention fusion layer proposed in this invention. Detailed Implementation

[0032] The following will provide a clear and complete description of the concept, specific structure, and technical effects of the present invention in conjunction with embodiments and accompanying drawings, so as to fully understand the purpose, solution, and effects of the present invention. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0034] It is understood that the terms “first,” “second,” etc., used in this invention may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, first information may also be referred to as second information without departing from the scope of embodiments of the invention, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to determination” as used herein may be interpreted as “when…” or “when…” or “in response to determination.”

[0035] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0037] refer to Figure 1 and Figure 2 ,like Figure 1 The image shown is an image fusion method based on depth estimation and dual-module attention provided by an embodiment of the present invention. The method includes the following steps:

[0038] S100, acquire infrared image, visible light image and pre-trained image fusion model; wherein, the image fusion model includes a depth estimation module, a Restormer block, a Fourier neural operator module, a basic feature extraction module based on Transformer architecture, a detail feature extraction module based on reversible neural network architecture, a dual-module attention fusion layer and a decoder.

[0039] S200, the infrared image and the visible light image are input into a pre-trained image fusion model, and the infrared image and the visible light image are processed by the depth estimation module to output the depth estimation information of the infrared image and the depth estimation information of the visible light image;

[0040] S300, the feature mapping of the infrared image and the visible light image is performed by the Restormer block respectively, and the shallow features of the infrared image and the shallow features of the visible light image are output.

[0041] S400, the shallow features of the infrared image and the shallow features of the visible light image are modeled and enhanced in the frequency domain through the Fourier neural operator module, and the frequency domain enhancement features of the infrared image and the visible light image are output.

[0042] S500, the basic feature extraction module processes the frequency domain enhancement features of the infrared image and the frequency domain enhancement features of the visible light image respectively, and outputs the basic features of the infrared image and the basic features of the visible light image;

[0043] S600, the detail feature extraction module processes the frequency domain enhancement features of the infrared image and the frequency domain enhancement features of the visible light image respectively, and outputs the detail features of the infrared image and the detail features of the visible light image.

[0044] S700 performs fusion processing on the basic features of the infrared image, the basic features of the visible light image, the detailed features of the infrared image, the detailed features of the visible light image, the depth estimation information of the infrared image, and the depth estimation information of the visible light image through the dual-module attention fusion layer, and outputs the basic features and detailed features after cross-modal fusion.

[0045] S800: The decoder decodes and reconstructs the basic features and detailed features after cross-modal fusion, and outputs the fused image.

[0046] In the embodiments provided by this invention, firstly, the depth estimation module consists of a depth estimation network and a depth feature extraction network, effectively extracting depth estimation information from the image. Infrared and visible light images undergo feature mapping via Restormer blocks to extract shallow spatial features. Subsequently, the depth estimation information is used for early depth enhancement in the shallow layer in the form of lightweight residuals to enhance shallow spatial features. Then, it enters the Fourier neural operator module for frequency domain enhancement, obtaining more discriminative backbone features. Next, the encoder divides the backbone features into two branches: basic features are extracted by a basic feature extraction module based on a Transformer architecture, and detailed features are extracted by a detailed feature extraction module based on a reversible neural network architecture. Then, the basic and detailed features enter a dual-module attention fusion layer. Within the basic branch of the dual-module attention fusion layer, the basic features are deeply fused with the corresponding backbone basic features through a feature recalibration module, thereby enhancing the spatial structure perception capability of the basic features and forming an enhanced basic feature representation. Finally, the enhanced basic features are aggregated across modalities. The detailed features are processed by the global enhancement fusion module in the dual-module attention fusion layer, achieving effective fusion of detailed features while remaining unaffected by depth to preserve texture and edge information as much as possible, resulting in a detailed feature representation. Subsequently, the fused basic and detailed features are concatenated along the channel dimension and fed into the decoder. After passing through the Restormer block and then through a convolutional layer, the features are restored to the image. Finally, the residual is superimposed with the visible light image, and the image is normalized by the Sigmoid function to output the final high-quality fused image.

[0047] refer to Figure 2The image fusion model comprises two main parts: input mapping and encoder / decoder. The encoder consists of six modules: a depth estimation module, a Restormer block, a Fourier neural operator module, a basic feature extraction module, a detail feature extraction module, and a dual-module attention fusion layer. The depth estimation module, composed of a depth estimation network and a deep feature extraction network, effectively extracts depth estimation information from the image. On one hand, it performs early depth enhancement in the shallow layers using lightweight residuals; on the other hand, in the later stages of encoding, it performs deep fusion with basic features through the feature recalibration module in the dual-module attention fusion layer to enhance the spatial structure perception capability of the basic features. The basic feature extraction module, based on a Transformer structure, is responsible for obtaining low-frequency and structural backbone information; the detail feature extraction module, based on a reversible neural network architecture, is responsible for extracting texture and high-frequency information. The dual-module attention fusion layer includes a feature recalibration module and a global enhancement fusion module: the feature recalibration module fuses deep features with basic features to generate enhanced basic features, which are then further fused across modalities; the global enhancement fusion module fuses detail features across modalities, thus providing a more hierarchical and complete feature representation for subsequent image reconstruction. The decoder consists of multiple layers of Restormer blocks and convolutional reconstruction layers, used to restore the abstract representation output by the encoder into a high-quality fused image.

[0048] The input consists of infrared and visible light images. The infrared and visible light images are first processed... Convolution completes feature mapping and shallow feature extraction, then feeds into the Restormer block to obtain local texture and context. Subsequently, depth estimation information is used to enhance spatial features in the form of lightweight residuals for early depth enhancement. Then, it enters the Fourier neural operator module, which extracts shallow and contextual information in the spatial domain and injects it into the frequency domain response in a residual manner to form the backbone features. The backbone features then split into two branches: basic features are extracted by a Transformer-based basic feature extraction module, and detailed features are extracted by a reversible neural network-based detailed feature extraction module. The basic and detailed features then enter a dual-module attention fusion layer. The basic features are fused with the depth estimation information and the corresponding backbone basic features through a feature recalibration module, generating enhanced basic features. These enhanced basic features are then fused across modalities again to form the final basic feature representation. Detail features are processed by the global enhancement fusion module in the dual-module attention fusion layer, achieving effective fusion of detailed features while remaining unaffected by depth to preserve texture and edge information as much as possible, resulting in a detailed feature representation. Subsequently, the fused basic and detailed features are concatenated along the channel dimension and fed into the decoder. After passing through the Restormer block and then through the convolutional layer, the features are restored to the image. Finally, the residual is superimposed with the visible light image, and the final high-quality fused image is output after normalization by the Sigmoid function.

[0049] The Restormer block, Fourier neural operator module, basic feature extraction module, detail feature extraction module, and dual-module attention fusion layer together constitute the main feature extraction path; the depth estimation module is independent of the main path and acts as a parallel module within the encoder to represent the depth map. Its output depth estimation information is not only subjected to early depth fusion after passing through the Restormer block, but also deeply fused with the basic features in the feature recalibration module of the dual-module feature fusion layer.

[0050] In some embodiments, in S200, the process of processing the infrared image and the visible light image through the depth estimation module to output depth estimation information for the infrared image and depth estimation information for the visible light image includes:

[0051] S210, The input infrared image and visible light image are fed into the depth estimation network. The depth estimation network performs joint depth modeling on the infrared image and visible light image to generate a preliminary depth map.

[0052] S220: The initial depth map is input into the depth feature extraction network. The depth feature extraction network first performs feature dimensionality reduction and abstraction through two convolutional layers to obtain depth features. The first convolutional layer extracts shallow depth features, and the second convolutional layer adjusts the channel dimensions.

[0053] S230, the depth features are weighted by a depth-specific channel attention mechanism to obtain weighted depth features;

[0054] S240, the weighted depth features are used as depth estimation information for the infrared image and the visible light image, respectively.

[0055] refer to Figure 3 The depth estimation module specifically performs feature representation on the input depth image, extracting high-level depth structure information:

[0056]

[0057] in, This represents a deep feature extraction network. This represents the depth map generation model. This represents the input infrared and visible light images. These represent the extracted depth features. This module uses a two-layer convolutional network structure combined with a depth-specific channel attention mechanism to extract high-level depth structure representations from depth images, providing important spatial structure information for subsequent depth fusion.

[0058] In some embodiments, in S300, the step of performing feature mapping on the infrared image and the visible light image respectively through the Restormer block, and outputting the shallow features of the infrared image and the shallow features of the visible light image, includes:

[0059] S310 uses overlapping convolution to perform feature mapping and shallow feature extraction on the input infrared image and visible light image respectively, to obtain the feature-mapped infrared image features and visible light image features;

[0060] S320: The infrared image features and visible light image features after feature mapping are respectively fed into several stacked Restormer blocks. Through the multi-head self-attention mechanism, feedforward neural network, layer normalization and residual connection contained in the Restormer block, the local texture and long-range dependency of the image are modeled at the same time, and the shallow features of the infrared image and the shallow features of the visible light image are output.

[0061] Specifically, each sub-block within the Restormer block contains core components such as a multi-head self-attention mechanism, a feedforward neural network, layer normalization, and residual connections. It aims to extract shallow features from infrared and visible light inputs.

[0062]

[0063] in, Represents the Restormer block. This represents the input infrared and visible light images. This represents the shallow features of the processed visible light and infrared images.

[0064] The Restormer block is used to map infrared and visible light inputs to a unified shallow feature space. In its implementation, it first uses... Convolution completes the input mapping while preserving resolution and local context. Then, several Restormer blocks are stacked to simultaneously model local texture and long-range dependencies. The advantages are: processing with the same encoder can align the two modalities to a consistent representation domain, reducing modality differences and facilitating subsequent Fourier neural operator modeling and feature decomposition; overlapping convolutions and Restormer structures preserve edges and details without downsampling, providing a high-quality shallow semantic and texture foundation.

[0065] After the Restormer block, the depth estimation module specifically represents the input depth image, and its output performs early depth enhancement in the shallow layers as a lightweight residual:

[0066]

[0067] in This represents the shallow features of infrared and visible light images after early depth enhancement. The value is 0.1. After the Restormer block, early depth enhancement is performed. This operation injects depth features into shallow infrared and visible light features in the form of lightweight residuals, providing the network with stable structural priors. It can significantly improve the geometric and hierarchical perception capabilities of shallow features, laying a more reliable structural foundation for subsequent frequency domain modeling, extraction of basic and detailed features, and dual-module attention fusion. It also helps to better maintain the consistency of edges and contours in the fusion result, reduce structural distortion, and improve overall quality.

[0068] In some embodiments, S400, the step of performing frequency domain modeling and enhancement on the shallow features of the infrared image and the shallow features of the visible light image through the Fourier neural operator module, and outputting the frequency domain enhancement features of the infrared image and the frequency domain enhancement features of the visible light image, includes:

[0069] S410, Perform two-dimensional Fourier transform on the shallow features of the infrared image and the shallow features of the visible light image after early depth enhancement to obtain their respective complex spectra, wherein the complex spectra contain real and imaginary parts;

[0070] S420, extract the low-order spectral coefficients of the complex spectrum, apply convolution operations to the real and imaginary parts of the extracted low-order spectral coefficients in the frequency domain, and introduce a frequency domain attention mechanism, which performs weighted adjustment of frequency domain features through amplitude gating.

[0071] S430 combines the real and imaginary spectral coefficients adjusted by the frequency domain attention mechanism, and then performs a two-dimensional inverse Fourier transform to convert from the frequency domain back to the spatial domain, thus obtaining the preliminary frequency domain enhancement features.

[0072] S440, the preliminary frequency domain enhancement features are weighted and fused with the shallow features of the infrared image or the shallow features of the visible light image after early depth enhancement according to a learnable scaling factor;

[0073] The S450 inputs the weighted fused features into a lightweight feedforward network for further optimization, outputting frequency domain enhancement features for infrared images and frequency domain enhancement features for visible light images.

[0074] Specifically, the Fourier neural operator module performs frequency domain modeling and enhancement of shallow features using Fourier neural operators:

[0075]

[0076] in, This represents the Fourier neural operator module. This represents the frequency domain enhancement features after processing by the Fourier neural operator module.

[0077] The Fourier neural operator module enhances the global representation by performing Fourier domain modeling and residual injection on shallow features: its operation involves first performing a two-dimensional Fourier transform on the shallow features output by the Restormer block, selecting / truncating low-order spectral coefficients, and applying them to the real and imaginary parts in the frequency domain, respectively. The algorithm employs convolution and incorporates frequency domain attention (amplitude gating). The updated spectrum is then restored to the spatial domain via inverse Fourier transform. Finally, the processed features are weighted and fused with the original shallow features according to a learnable ratio, and further optimized using a lightweight feedforward network. The advantages of this design are: capturing long-range dependencies and global structure in the frequency domain with low computational overhead; explicitly enhancing regular textures and edge periodic components; complementing spatial attention / convolution to improve the joint perception of large-scale layouts and fine-grained textures; maintaining resolution; and stabilizing training through residuals, avoiding the high complexity and gradient instability associated with pure spatial attention.

[0078] The aforementioned basic feature extraction module based on the Transformer architecture is used to extract low-frequency basic features from frequency domain enhanced features, which includes deep information fusion.

[0079]

[0080] in, This represents the basic feature extraction module. These represent the basic features of the infrared image and the visible light image after processing by the basic feature extraction module, respectively.

[0081] The detailed feature extraction module based on the reversible neural network architecture is used to extract high-frequency detailed features from the frequency domain enhanced features, capturing the texture, edge, and fine structure information of the image.

[0082]

[0083] in, This indicates the detailed feature extraction module. These represent the detailed features of the infrared and visible light images after processing by the detail feature extraction module, respectively.

[0084] In some embodiments, the dual-module attention fusion layer includes a feature recalibration module and a global enhancement fusion module. In S700, the dual-module attention fusion layer performs fusion processing on the basic features of the infrared image, the basic features of the visible light image, the detailed features of the infrared image, the detailed features of the visible light image, the depth estimation information of the infrared image, and the depth estimation information of the visible light image, outputting the cross-modal fused basic features and the cross-modal fused detailed features, including:

[0085] S710, the feature recalibration module fuses the basic features of the infrared image with the depth estimation information of the infrared image to generate basic features for infrared image enhancement; the feature recalibration module also fuses the basic features of the visible light image with the depth estimation information of the visible light image to generate basic features for visible light image enhancement; wherein, the feature recalibration module achieves the fusion of basic features and depth estimation information through feature recalibration, channel aggregation, local attention, and global attention mechanisms;

[0086] S720, the global enhancement fusion module fuses the detailed features of the infrared image with the detailed features of the visible light image to generate cross-modal fused detailed features; the basic features of the infrared image enhancement and the basic features of the visible light image enhancement are cross-modal aggregated to generate cross-modal fused basic features; wherein, the global enhancement fusion module achieves cross-modal fusion of detailed features through channel dimensionality reduction, spatial downsampling, local convolution enhancement and attention mechanism.

[0087] refer to Figure 4 and Figure 5 The dual-module attention fusion layer includes a feature recalibration module and a global enhancement fusion module, which are responsible for effectively fusing basic features, detailed features, and deep features.

[0088] First, in the feature recalibration module, the basic features and depth estimation information are fused:

[0089]

[0090] in, These are the basic features of infrared and visible light images, respectively. These are depth estimation information from infrared and visible light images, respectively. This represents the feature recalibration module. This module deeply fuses basic features with deep estimation information through feature recalibration, channel aggregation, local attention, and global attention mechanisms to generate enhanced basic features. This provides enhanced multi-level feature representations for subsequent image reconstruction.

[0091] Then, in the feature recalibration module and the global enhancement fusion module, the enhancement fusion of basic and detailed features is performed respectively.

[0092] Using the recalibrated base features as priors, two cross-modal fusion processes are performed: one is cross-modal aggregation at the base layer, enhancing structural consistency and brightness / contrast robustness; the other is cross-modal fusion at the detail layer, utilizing global context and bidirectional attention to enhance texture, edge, and high-frequency information. These two fusion processes are performed in parallel, outputting a multi-layered fusion representation for decoding.

[0093]

[0094] in, These are the basic features for enhancing infrared and visible light images, respectively. For detailed features, This represents the global enhancement and fusion module. This module effectively fuses basic and detailed features through channel dimensionality reduction, spatial downsampling, local convolutional enhancement, and attention mechanisms. This represents the basic features after cross-modal fusion. It represents the detailed features after cross-modal fusion, providing enhanced multi-level feature representation for subsequent image reconstruction.

[0095] In some embodiments, in S800, the step of decoding and reconstructing the basic features and detailed features after cross-modal fusion using the decoder, and outputting the fused image, includes:

[0096] S810, the basic features and detailed features after cross-modal fusion are spliced ​​together in the channel dimension to obtain the spliced ​​features;

[0097] S820 feeds the stitched features into a decoder layer based on Restormer blocks. After processing by the Restormer blocks, the features are then restored to image-dimensional features through a convolutional layer.

[0098] The S830 performs residual superposition of the restored image dimensional features and the original visible light image, and then normalizes the superposition result using the Sigmoid function to output the fused image.

[0099] Specifically, the final features enter the decoder. In the decoder, the fused basic features and detail features are concatenated along the channel dimension and used as input. This fused input is then used for decoding and reconstruction.

[0100] ;

[0101] in, This represents the features after decoding and reconstruction. Indicates channel splicing. This indicates that the concatenated features are decoded and reconstructed through a decoder layer based on Restormer blocks.

[0102] Finally, the decoded and reconstructed features are residually joined with the original visible light image:

[0103] ;

[0104] in, For the final fused image, This is a non-linear activation function operation that ensures the pixel values ​​of the fused image are within a reasonable range. This is a visible light image used for residual joining.

[0105] In some embodiments, the image fusion model employs a comprehensive loss function during training, which includes reconstruction loss, decomposition loss, fusion loss, depth consistency loss, and frequency domain preservation loss.

[0106] The formula for calculating the reconstruction loss is:

[0107] ;

[0108] ;

[0109] ;

[0110] in, and These are the weighting coefficients. Represents the structural similarity index. This represents the mean square error. and These represent visible light images, infrared images, and fused images, respectively.

[0111] The depth consistency loss The calculation formula is as follows:

[0112] ;

[0113] in, These are the weighting coefficients. This represents the deep consistency calculation function. These represent the depth features extracted from infrared and visible light depth images, respectively. These represent the basic features of infrared and visible light images, respectively.

[0114] Specifically, during the model training process, a comprehensive loss function is used, including reconstruction loss, decomposition loss, fusion loss, depth consistency loss, and frequency domain preservation loss, the specific expression of which is as follows:

[0115] ;

[0116] in, Indicates the reconstruction loss. Indicates the decomposition loss. Indicates fusion loss, This represents the loss of deep consistency. This indicates frequency domain preservation loss.

[0117] The model's inference process is as follows: First, the depth estimation module consists of a depth estimation network and a depth feature extraction network, effectively extracting depth estimation information from the image. Infrared and visible light images undergo feature mapping via Restormer blocks to extract shallow spatial features. Subsequently, the depth estimation information is used for early depth enhancement in the shallow layer as a lightweight residual, enhancing shallow spatial features. Then, it enters the Fourier neural operator module for frequency domain enhancement, resulting in more discriminative backbone features. Next, the encoder divides the backbone features into two branches: basic features are extracted by a Transformer-based basic feature extraction module, and detailed features are extracted by a reversible neural network-based detailed feature extraction module. Then, the basic and detailed features enter a dual-module attention fusion layer. Within the basic branch of this layer, the basic features are fused with the corresponding backbone basic features through a feature recalibration module, thereby enhancing the spatial structure perception capability of the basic features and forming an enhanced basic feature representation. Finally, the enhanced basic features are aggregated across modalities. The detailed features are processed by the global enhancement fusion module in the dual-module attention fusion layer, achieving effective fusion of detailed features while remaining unaffected by depth to preserve texture and edge information as much as possible, resulting in a detailed feature representation. Subsequently, the fused basic and detailed features are concatenated along the channel dimension and fed into the decoder. After passing through the Restormer block and then through a convolutional layer, the features are restored to the image. Finally, the residual is superimposed with the visible light image, and the image is normalized by the Sigmoid function to output the final high-quality fused image.

[0118] The following are specific embodiments provided by the present invention:

[0119] Model structure:

[0120] Step S1: Input mapping;

[0121] The infrared and visible light images input into the model will first pass through a Convolution extracts shallow features to obtain preliminary feature representations.

[0122] Step S2: Depth estimation and early depth fusion;

[0123] The depth estimation module consists of a depth estimation network and a depth feature extraction network, extracting depth estimation information from the image. Initial features are fed into the Restormer block to extract shallow spatial features. Through normalization, attention mechanisms, and feedforward networks, multi-scale spatial features are extracted to obtain richer feature representations. Further, the depth estimation module represents the input depth image, and its output is used for early depth enhancement in the form of lightweight residuals, improving the perception of geometric structures and weakly textured regions. This is then fed into the Fourier neural operator module. This module performs a Fourier transform on the feature map, enhancing the global structure and periodic information representation of the features in the frequency domain through learnable convolutions and attention mechanisms. Subsequently, an inverse Fourier transform maps the enhanced features back to the spatial domain, and residual connections are made with the original features. This provides a rich and high-quality feature foundation for subsequent feature decomposition and fusion.

[0124] Step S3: Feature decomposition and dual-module attention fusion;

[0125] Following this is feature decomposition. Basic features are extracted by the Transformer-based basic branch, while detailed features are extracted by the detail branch based on a reversible neural network. Both basic and detailed features then enter a dual-module attention fusion layer. Within the basic branch of this layer, the basic features are deeply fused with the corresponding backbone basic features through a feature recalibration module, thereby enhancing the spatial structure perception capability of the basic features and forming an enhanced basic feature representation. This enhanced basic feature is then aggregated across modalities. The detailed features, on the other hand, are processed by the global enhancement fusion module in the dual-module attention fusion layer, achieving effective fusion while remaining unaffected by depth to preserve texture and edge information as much as possible, resulting in a detailed feature representation.

[0126] Step S4: Decoder reconstruction;

[0127] The fusion of fundamental and detailed features enters the decoder, where channel splicing is first performed, followed by... After dimensionality reduction via convolution, the input consists of the decoder backbone composed of several Restormer blocks. Several more convolutional layers then reconstruct the features into an image, generating a single-channel reconstruction result. This result is then added to the visible light image as a residual, and finally normalized using Sigmoid to obtain a high-quality fused image. This design effectively preserves multimodal detail information while maintaining global structural consistency.

[0128] Training phase:

[0129] During model training, a comprehensive loss function is used, and the form of the loss function is as follows:

[0130] ;

[0131] in, Indicates the reconstruction loss. Indicates the decomposition loss. Indicates fusion loss, This represents the loss of deep consistency. This indicates frequency domain preservation loss.

[0132] Reasoning stage:

[0133] Reasoning stage, such as Figure 2 As shown, during inference, infrared and visible light images are input. First, the depth estimation module consists of a depth estimation network and a depth feature extraction network, effectively extracting depth estimation information from the images. Infrared and visible light images undergo feature mapping via Restormer blocks to extract shallow spatial features. Subsequently, the depth estimation information is used for early depth enhancement in the shallow layer as a lightweight residual, enhancing shallow spatial features. This is followed by frequency domain enhancement via a Fourier neural operator module, resulting in more discriminative backbone features. Next, the encoder divides the backbone features into two branches: basic features are extracted by a Transformer-based basic feature extraction module, and detailed features are extracted by a reversible neural network-based detailed feature extraction module. Then, the basic and detailed features enter a dual-module attention fusion layer. Within the basic branch of this layer, the basic features are fused with the corresponding backbone basic features via a feature recalibration module, enhancing the spatial structure perception capability of the basic features and forming an enhanced basic feature representation. Finally, the enhanced basic features are aggregated across modalities. The detailed features are processed by the global enhancement fusion module in the dual-module attention fusion layer, achieving effective fusion of detailed features while remaining unaffected by depth to preserve texture and edge information as much as possible, resulting in a detailed feature representation. Subsequently, the aggregated basic features and detailed features are concatenated along the channel dimension and fed into the decoder. After passing through the Restormer block and then through a convolutional layer, the features are restored to the image. Finally, the residual is superimposed with the visible light image, and the image is normalized by the Sigmoid function to output the final high-quality fused image.

[0134] Basic features are extracted by a Transformer-based basic feature extraction module, while detailed features are extracted by a reversible neural network-based detailed feature extraction module. Both then enter a dual-module attention fusion layer. The basic features are deeply fused with the corresponding backbone basic features through a feature recalibration module to obtain enhanced basic features. These enhanced basic features are then aggregated across modalities. Detailed features are processed by the global enhancement fusion module in the dual-module attention fusion layer, achieving effective fusion of detailed features.

[0135] For the basic features, the fusion strategy can be expressed as:

[0136] ;

[0137] in, This refers to the feature recalibration module. This module deeply fuses basic features and deep features through feature recalibration, channel aggregation, local attention, and global attention mechanisms to generate enhanced basic features. .

[0138] In a formal sense, for detailed features, the fusion strategy can be expressed as:

[0139] ;

[0140] in, This represents the global enhancement and fusion module. This module effectively fuses basic and detailed features through channel reduction, spatial downsampling, local convolutional enhancement, and attention mechanisms to generate the final fused features. This provides enhanced multi-level feature representations for subsequent image reconstruction.

[0141] Performance evaluation:

[0142] This invention uses infrared and visible light images from the MSRS (Multi-Spectral Road Scenarios) dataset and generates corresponding depth images for both. These images are used to train the model. During the testing phase, the model performance is evaluated using the standard test set of the MSRS dataset.

[0143] Simulation experiment:

[0144] This invention utilizes paired infrared and visible light images from the MSRS dataset as training samples during the training phase. A total of 1083 pairs are used. All images are uniformly adjusted to... The training data is organized in HDF5 format, and the model is loaded using a custom H5PatchDataset class to ensure a one-to-one correspondence between infrared, visible light, and depth images. During training, infrared and visible light images are used as input, passing through the Restormer block, early depth enhancement, and Fourier neural operator modules before entering the basic feature extraction module and the detail feature extraction module to generate basic and detail features. Training settings: epoch=120, batch_size=16, lr=1e-4; mixed precision and gradient clipping are used; four independent optimizers and StepLR (step size 20, decay 0.5) are used; the total loss is a weighted sum of reconstruction loss, feature decomposition correlation loss, fusion loss, frequency domain loss, and depth consistency loss; multi-GPU training is supported, and checkpoints are saved periodically.

[0145] Comparative experiment:

[0146] This paper selects 32 of the most advanced existing fusion methods for comparative experiments:

[0147] This includes image fusion techniques based on dual-tree complex wavelet transform (DTCWT). [1] Remote sensing image fusion based on curve transform (CVT) [2] Image fusion techniques based on multi-resolution singular value decomposition (MSVD) [3] An image fusion algorithm based on a non-subsampled contour wave transform domain spatial frequency excitation pulse coupled neural network (NSCT-SF-PCNN) [4] Image fusion (ADF) of infrared and visible light sensors based on anisotropic diffusion and KL transform [5] Infrared and visible light image perceptual fusion (MLGCF) is achieved through multi-level Gaussian curvature filtering image decomposition. [6] Infrared and visible light image fusion (Quadtree) through infrared feature extraction and preservation of visual information. [7] Infrared and Visible Image Fusion Based on Target Enhancement Multi-Scale Transform Decomposition (TE-MST) [8] A general framework for image fusion based on multi-scale transformation and sparse representation (DTCWT-SR) [9] Texture-sharp multimodal image fusion based on joint sparse model (JSM)

[10] Infrared and Visible Image Fusion Based on Lat Low-Rank Representation (LatLRR)

[11] A novel infrared and visible light image fusion decomposition method (MDLatLRR)

[12] Infrared and Visible Image Fusion (GTF) Based on Gradient Transfer and Total Variation Minimization

[13] DRTV (Different Resolution Infrared and Visible Image Fusion) Based on a Total Variation Model

[14] Infrared and Visible Image Fusion Using a Deep Learning Framework (VggML)

[15] Infrared and visible light image fusion based on ResNet and zero phase component analysis (ResNet-ZCA)

[16] An infrared and visible light image fusion method based on a dense connection encoding and decoding architecture (DenseFuse)

[17] A method for fusion of infrared and visible light images based on nested connectivity and spatial channel attention model (NestFuse)

[18] An end-to-end residual fusion network (RFN-Nest) for infrared and visible light images.

[19] A Decoupled Representation-Based Infrared and Visible Image Fusion Method (DRF)

[20] Generative Adversarial Network (FusionGAN) for fusing infrared and visible light images

[21] A Generative Adversarial Network (GANMCC) for Infrared and Visible Image Fusion with Multi-Classification Constraints

[22] A CSF (Category Saliency) method for infrared and visible light image fusion.

[23] A fast unified image fusion network (PMGI) based on gradient and intensity ratio preservation.

[24] A method for fusing infrared and visible light images based on depth image decomposition (DIDFuse)

[25] An unsupervised depth image fusion method based on structure tensor representation (DIFNet)

[26] A self-supervised decomposition-based image fusion method (DeFusion)

[27] A novel infrared and visible light image fusion network based on salient target detection (STDFusionNet)

[28] A deep image fusion model based on task guidance, implicit search, and meta-initialization (TIMFusion)

[29] A target detection-oriented infrared and visible light image fusion method (TarDAL)

[30] A general image fusion method based on cross-domain long-range learning and Swin Transformer (SwinFusion)

[31] And a cross-attention mechanism-based infrared and visible light image fusion method (CrossFuse).

[32] .

[0148] This paper uses nine metrics to quantitatively evaluate fusion performance, including:

[33] ,

[34] ,

[35] ,

[36] ,

[37] ,

[37] ,

[37] ,

[38] ,

[39] Multiple metrics were used for quantitative evaluation to comprehensively verify the effectiveness and superiority of the proposed method in multimodal image fusion tasks. Furthermore, the effectiveness of the proposed method was validated using 83 pairs of RoadScene datasets, 46 pairs of LLVIP datasets, 88 pairs of M3FD datasets, and 52 pairs of TNO datasets.

[0149] Table 1: Comparison of the average fusion results obtained by existing fusion methods and the method proposed in this invention on the RoadScene dataset.

[0150]

[0151] Table 2: Comparison of the average fusion results obtained by existing fusion methods and the method proposed in this invention on the LLVIP dataset.

[0152]

[0153] Table 3: Comparison of the average fusion results obtained by existing fusion methods and the method proposed in this invention on the M3FD dataset.

[0154]

[0155] Table 4: Comparison of the average fusion results obtained by existing fusion methods and the method proposed in this invention on the TNO dataset;

[0156]

[0157] As shown in Tables 1, 2, 3, and 4, the method proposed in this invention achieved optimal values ​​on nine metrics, indicating that the fused images obtained by the method of this invention on the RoadScene, LLVIP, M3FD, and TNO datasets have better image quality than those obtained by 32 other methods such as FusionGAN, DRF, and VggML.

[0158] These metrics also demonstrate that the proposed fusion network achieves better fusion performance in complex scenarios (TNO, RoadScene, LLVIP, and M3FD). On four datasets, the method of this invention achieved nine best-in-class metrics. These observations indicate that the fusion method of this invention achieves higher robustness against interference compared to state-of-the-art fusion methods.

[0159] The ablation experiment of this invention will be described next:

[0160] Table 5: Comparison of fusion performance of the ablation experiment conducted by the fusion method of this invention on the RoadScene dataset. This ablation experiment removed the depth information estimation mechanism.

[0161]

[0162] These metrics also demonstrate the effectiveness of the innovations of this invention in the fusion network. Compared with the depth information removal estimation mechanism, the innovations added in this invention enable the model to achieve better fusion performance.

[0163] Table 6: Comparison of fusion performance of the ablation experiment conducted by the fusion method of the present invention on the RoadScene dataset. This ablation experiment removed the dual-module attention fusion layer.

[0164]

[0165] These metrics also demonstrate the effectiveness of the innovations of this invention in the fusion network. Compared with removing the dual-module attention fusion layer, the innovations added in this invention enable the model to achieve better fusion performance.

[0166] References comparing algorithms and evaluation metrics:

[0167] [1] DTCWT: Lewis JJ, O'Callaghan RJ, Nikolov SG, et al. Pixel-andregion-based image fusion with complex wavelets[J]. Information fusion, 2007,8(2): 119-130.

[0168] [2] CVT: Nencini F, Garzelli A, Baronti S, et al. Remote sensingimage fusion using the curvelet transform[J]. Information fusion, 2007, 8(2):143-156.

[0169] [3] MSVD: Naidu VP S. Image fusion technique using multi-resolutionsingular value decomposition[J]. Defense Science Journal, 2011, 61(5): 479.

[0170] [4] NSCT-SF-PCNN: Xiao-Bo Q, Jing-Wen Y, Hong-Zhi X, et al. Imagefusion algorithm based on spatial frequency-motivated pulse coupled neural networks in nonsubsampled contourlet transform domain[J]. Acta AutomaticaSinica, 2008, 34(12): 1508-1514.

[0171] [5] ADF: Bavirisetti D P, Dhuli R. Fusion of infrared and visiblesensor images based on anisotropic diffusion and Karhunen-Loeve transform[J].IEEE Sensors Journal, 2015, 16(1): 203-209.

[0172] [6] MLGCF: Tan W, Zhou H, Song J, et al. Infrared and visible imageperceptive fusion through multi-level Gaussian curvature filtering imagedecomposition[J]. Applied optics, 2019, 58(12): 3064-3073.

[0173] [7] Quadtree: Zhang Y, Zhang L, Bai X, et al. Infrared and visualimage fusion through infrared feature extraction and visual informationpreservation[J]. Infrared Physics & Technology, 2017, 83: 227-237.

[0174] [8] TE-MST: Chen J, Li X, Luo L, et al. Infrared and visible imagefusion based on target-enhanced multiscale transform decomposition[J].Information Sciences, 2020, 508: 64-78.

[0175] [9] DTCWT-SR: Liu Y, Liu S, Wang Z. A general framework for imagefusion based on multi-scale transform and sparse representation[J].Information fusion, 2015, 24: 147-164.

[0176]

[10] JSM: Gao Z, Zhang C. Texture clear multi-modal image fusion withjoint sparsity model[J]. Optik, 2017, 130: 255-265.

[0177]

[11] LatLRR: Li H, Wu X J. Infrared and visible image fusion usinglatent low-rank representation[J]. arXiv preprint arXiv:1804.08992, 2018.

[0178]

[12] MDLatLRR: Li H, Wu X J, Kittler J. MDLatLRR: A noveldecomposition method for infrared and visible image fusion[J]. IEEETransactions on Image Processing, 2020, 29: 4733-4746.

[0179]

[13] GTF: Ma J, Chen C, Li C, et al. Infrared and visible imagefusion via gradient transfer and total variation minimization[J]. InformationFusion, 2016, 31: 100-109.

[0180]

[14] DRTV: Du Q, Xu H, Ma Y, et al. Fusing infrared and visibleimages of different resolutions via total variation model[J]. Sensors, 2018,18(11): 3827.

[0181]

[15] VggML: Li H, Wu X J, Kittler J. Infrared and visible imagefusion using a deep learning framework[C] / / 2018 24th international conferenceon pattern recognition (ICPR). IEEE, 2018: 2705-2710.

[0182]

[16] ResNet-ZCA: Li H, Wu X, Durrani T S. Infrared and visible imagefusion with ResNet and zero-phase component analysis[J]. Infrared Physics &Technology, 2019, 102: 103039.

[0183]

[17] DenseFuse: Li H, Wu X J. DenseFuse: A fusion approach toinfrared and visible images[J]. IEEE Transactions on Image Processing, 2018,28(5): 2614-2623.

[0184]

[18] NestFuse: Li H, Wu X J, Durrani T. NestFuse: An infrared andvisible image fusion architecture based on nest connection and spatial / channel attention models[J]. IEEE Transactions on Instrumentation andMeasurement, 2020, 69(12): 9645-9656.

[0185]

[19] RFN-Nest: Li H, Wu X J, Kittler J. RFN-Nest: An end-to-endresidual fusion network for infrared and visible images[J]. InformationFusion, 2021, 73: 72-86.

[0186]

[20] DRF: Xu H, Wang X, Ma J. DRF: Disentangled representation forvisible and infrared image fusion[J]. IEEE Transactions on Instrumentationand Measurement, 2021, 70: 1-13.

[0187]

[21] FusionGAN: Ma J, Yu W, Liang P, et al. FusionGAN: A generativeadversarial network for infrared and visible image fusion[J]. Informationfusion, 2019, 48: 11-26.

[0188]

[22] GANMcC: Ma J, Zhang H, Shao Z, et al. GANMcC: A generativeadversarial network with multiclassification constraints for infrared andvisible image fusion[J]. IEEE Transactions on Instrumentation andMeasurement, 2020, 70: 1-14.

[0189]

[23] CSF: Xu H, Zhang H, Ma J. Classification saliency-based rule forvisible and infrared image fusion[J]. IEEE Transactions on ComputationalImaging, 2021, 7: 824-836.

[0190]

[24] PMGI: Zhang H, Xu H, Xiao Y, et al. Rethinking the image fusion:A fast unified image fusion network based on proportional maintenance ofgradient and intensity[C] / / Proceedings of the AAAI conference on artificialintelligence. 2020, 34(07): 12797-12804.

[0191]

[25] DIDFuse: Zhao Z, Xu S, Zhang C, et al. DIDFuse: Deep imagedecomposition for infrared and visible image fusion[J]. arXiv preprint arXiv:2003.09210, 2020.

[0192]

[26] DIFNet: Jung H, Kim Y, Jang H, et al. Unsupervised deep imagefusion with structure tensor representations[J]. IEEE Transactions on ImageProcessing, 2020, 29: 3845-3858.

[0193]

[27] DeFusion: Liang P, Jiang J, Liu X, et al. Fusion fromdecomposition: A self-supervised decomposition approach for image fusion[C] / / European conference on computer vision. Cham: Springer Nature Switzerland,2022: 719-735.

[0194]

[28] STDFusionNet: Ma J, Tang L, Xu M, et al. STDFusionNet: Aninfrared and visible image fusion network based on salient target detection[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-13.

[0195]

[29] TIMFusion: Liu R, Liu Z, Liu J, et al. A task-guided,implicitly-searched and meta-initialized deep model for image fusion[J]. IEEETransactions on Pattern Analysis and Machine Intelligence, 2024, 46(10):6594-6609.

[0196]

[30] TarDAL: Liu J, Fan X, Huang Z, et al. Target-aware dualadversarial learning and a multi-scenario multi-modality benchmark to fuseinfrared and visible for object detection[C] / / Proceedings of the IEEE / CVFconference on computer vision and pattern recognition. 2022: 5802-5811.

[0197]

[31] SwinFusion: Ma J, Tang L, Fan F, et al. SwinFusion: Cross-domainlong-range learning for general image fusion via swin transformer[J]. IEEE / CAA Journal of Automatica Sinica, 2022, 9(7): 1200-1217.

[0198]

[32] CrossFuse: Li H, Wu X J. CrossFuse: A novel cross attentionmechanism based infrared and visible image fusion approach[J]. InformationFusion, 2024, 103: 102147.

[0199]

[33] EI: Xydeas C S, Petrovic V. Objective image fusion performancemeasure[J]. Electronics letters, 2000, 36(4): 308-309.

[0200]

[34] SF: Eskicioglu A M, Fisher P S. Image quality measures and theirperformance[J]. IEEE Transactions on communications, 2002, 43(12): 2959-2965.

[0201]

[35] SSIM: Wang Z, Bovik A C, Sheikh H R, et al. Image qualityassessment: from error visibility to structural similarity[J]. IEEEtransactions on image processing, 2004, 13(4): 600-612.

[0202]

[36] QAB / F: Qu G, Zhang D, Yan P. Information measure for performanceof image fusion[J]. Electronics letters, 2002, 38(7): 313-315.

[0203]

[37] QS,QW,QE: Piella G, Heijmans H. A new quality metric for imagefusion[C] / / Proceedings 2003 international conference on image processing(Cat. No. 03CH37429). IEEE, 2003, 3: III-173.

[0204]

[38] AG: Cui G, Feng H, Xu Z, et al. Detail preserved fusion ofvisible and infrared images using regional saliency extraction and multi-scale image decomposition[J]. Optics Communications, 2015, 341: 199-209.

[0205]

[39] MS-SSIM: Ma K, Zeng K, Wang Z. Perceptual quality assessment for multi-exposure image fusion[J]. IEEE Transactions on Image Processing, 2015,24(11): 3345-3356.

[0206] This invention also provides an image fusion system based on depth estimation and dual-module attention, comprising:

[0207] At least one processor;

[0208] At least one memory for storing at least one program;

[0209] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0210] The content of the above method embodiments is applicable to this embodiment. The specific functions implemented in this embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments. Therefore, they will not be repeated here.

[0211] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0212] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0213] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0214] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0215] This invention also provides a computer program product, including a computer program or computer instructions, which are stored in a memory. A processor of a computer device reads the computer program or computer instructions from the memory and executes the computer program or computer instructions, causing the computer device to perform the above-described method.

[0216] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0217] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0218] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0219] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

Claims

1. An image fusion method based on depth estimation and dual module attention, characterized in that, The method comprises the following steps: S100, acquiring an infrared image, a visible light image, and a pre-trained image fusion model; wherein the image fusion model comprises a depth estimation module, a Restormer block, a Fourier neural operator module, a basic feature extraction module based on a Transformer architecture, a detailed feature extraction module based on a reversible neural network architecture, a double-module attention fusion layer, and a decoder; S200, inputting the infrared image and the visible light image into the pre-trained image fusion model, processing the infrared image and the visible light image through the depth estimation module, and outputting depth estimation information of the infrared image and depth estimation information of the visible light image; S300, performing feature mapping on the infrared image and the visible light image respectively through the Restormer block, and outputting shallow features of the infrared image and shallow features of the visible light image; S400, performing frequency domain modeling and enhancement on the shallow features of the infrared image and the shallow features of the visible light image respectively through the Fourier neural operator module, and outputting frequency domain enhanced features of the infrared image and frequency domain enhanced features of the visible light image; S500, processing the frequency domain enhanced features of the infrared image and the frequency domain enhanced features of the visible light image respectively through the basic feature extraction module, and outputting basic features of the infrared image and basic features of the visible light image; S600, processing the frequency domain enhanced features of the infrared image and the frequency domain enhanced features of the visible light image respectively through the detailed feature extraction module, and outputting detailed features of the infrared image and detailed features of the visible light image; S700, performing fusion processing on the basic features of the infrared image, the basic features of the visible light image, the detailed features of the infrared image, the detailed features of the visible light image, the depth estimation information of the infrared image, and the depth estimation information of the visible light image through the double-module attention fusion layer, and outputting cross-modal fused basic features and cross-modal fused detailed features; S800, decoding and reconstructing the cross-modal fused basic features and the cross-modal fused detailed features through the decoder, and outputting a fused image.

2. The method of claim 1, wherein, In S300, the Restormer block is used to perform feature mapping on the infrared image and the visible light image respectively, and output shallow features of the infrared image and shallow features of the visible light image, comprising: S310, using overlapping convolution to perform feature mapping and shallow feature extraction on the input infrared image and visible light image respectively, to obtain feature-mapped infrared image features and visible light image features; S320, sending the feature-mapped infrared image features and visible light image features into a plurality of stacked Restormer blocks respectively, simultaneously modeling local texture and long-range dependency of the image through multi-head self-attention mechanisms, feedforward neural networks, layer normalization, and residual connections contained in the Restormer blocks, and outputting shallow features of the infrared image and shallow features of the visible light image.

3. The method of claim 1, wherein, In S400, the shallow features of the infrared image and the shallow features of the visible light image are respectively modeled and enhanced in the frequency domain by the Fourier neural operator module, and the frequency domain enhanced features of the infrared image and the visible light image are output, including: In S410, two-dimensional Fourier transform is performed on the shallow features of the infrared image and the shallow features of the visible light image respectively to obtain corresponding complex spectra, which contain real parts and imaginary parts. In S420, low-order spectral coefficients of the complex spectra are intercepted, and convolution operations are respectively applied to the real parts and the imaginary parts of the intercepted low-order spectral coefficients in the frequency domain, and a frequency domain attention mechanism is introduced to weight and adjust the frequency domain features through amplitude gating. In S430, after the real part and the imaginary part of the spectral coefficient are adjusted by the frequency domain attention mechanism, two-dimensional inverse Fourier transform is performed to convert from the frequency domain to the spatial domain to obtain preliminary frequency domain enhanced features. In S440, the preliminary frequency domain enhanced features are weighted and fused with the infrared image shallow features or the visible light image shallow features after early depth enhancement according to a learnable proportion coefficient. In S450, the weighted and fused features are input into a lightweight feedforward network for further optimization, and the frequency domain enhanced features of the infrared image and the visible light image are output.

4. The method of claim 1, wherein, The dual-module attention fusion layer includes a feature re-calibration module and a global enhancement fusion module. In S700, the basic features of the infrared image, the basic features of the visible light image, the detailed features of the infrared image, the detailed features of the visible light image, the depth estimation information of the infrared image, and the depth estimation information of the visible light image are fused by the dual-module attention fusion layer to output the cross-modal fused basic features and the cross-modal fused detailed features, including: In S710, the feature re-calibration module is used to fuse the basic features of the infrared image with the depth estimation information of the infrared image to generate infrared image enhanced basic features, and to fuse the basic features of the visible light image with the depth estimation information of the visible light image to generate visible light image enhanced basic features. The feature re-calibration module realizes the fusion of the basic features and the depth estimation information through feature re-calibration, channel aggregation, local attention, and global attention mechanism. In S720, the global enhancement fusion module is used to fuse the detailed features of the infrared image with the detailed features of the visible light image to generate cross-modal fused detailed features, and to cross-modally aggregate the infrared image enhanced basic features with the visible light image enhanced basic features to generate cross-modal fused basic features. The global enhancement fusion module realizes the cross-modal fusion of the detailed features through channel dimension reduction, spatial downsampling, local convolution enhancement, and attention mechanism.

5. The method of claim 1, wherein, In S800, the cross-modal fused basic features and the cross-modal fused detailed features are decoded and reconstructed by the decoder to output a fused image, including: S810, concatenating the cross-modal fused base feature and the cross-modal fused detail feature in the channel dimension to obtain a concatenated feature; S820, sending the concatenated feature into a decoder layer based on a Restormer block, processing the concatenated feature through the Restormer block, and restoring the feature to an image dimension feature through a convolution layer; S830, performing residual superposition on the restored image dimension feature and the original visible light image, performing normalization processing on the superposition result through a Sigmoid function, and outputting a fusion image.

6. The method of claim 1, wherein, In the training process of the image fusion model, a comprehensive loss function is used, and the comprehensive loss function includes a reconstruction loss and a depth consistency loss. The calculation formula of the reconstruction loss is: ； ； ； wherein, and are weight coefficients, denotes a structural similarity index, denotes a mean square error, and denote a visible light image, an infrared image and a fused image, respectively. the depth consistency loss The calculation formula is as follows: ； wherein, are weight coefficients, denotes a depth consistency computation function, denote depth features extracted from infrared and visible light depth images, respectively, denote base features of infrared and visible light images, respectively.

7. An image fusion system based on depth estimation and dual module attention, characterized in that, including: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method of any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to implement the method of any one of claims 1 to 6.

Images