Infrared and visible light image enhancement fusion method and system for low-illumination scene

Abstract

The invention provides an infrared and visible light image enhancement fusion method and system for a low-illumination scene, and the method comprises the steps: obtaining an infrared image and a visible light image, and carrying out the preprocessing of the infrared image and the visible light image; inputting the brightness image and the illumination prior image into an encoder, and performing single-mode reconstruction pre-training through a minimization reconstruction loss function; performing deep feature extraction on the dual-channel visible light image and the normalized infrared image by using a pre-trained encoder, performing bidirectional feature guidance and fusion in a fusion module in a manner of exchanging query vectors by using a cross-modal cross attention mechanism, and reconstructing the fused features by using a pre-trained decoder; according to the method, low-illumination image enhancement and cross-modal feature fusion are integrated into a unified framework, brightness improvement, detail reservation and modal consistency synchronous optimization are realized through end-to-end joint modeling, and information loss caused by enhancement and fusion process separation is avoided.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision technology, and in particular to a method and system for enhancing and fusing infrared and visible light images in low-light scenes. Background Technology

[0002] In low-light environments such as nighttime, backlighting, and fog, traditional image acquisition methods often suffer from insufficient brightness, blurred details, and increased noise, leading to a significant decline in visual information quality. This restricts applications in scenarios requiring high image perception accuracy, such as intelligent security, nighttime driving, border monitoring, and military reconnaissance. Visible light images possess rich texture and color information, providing clear scene structure, but they are prone to brightness attenuation, image blurring, and low signal-to-noise ratio under insufficient lighting conditions. In contrast, infrared images, based on target thermal radiation imaging, do not rely on external visible light illumination and can stably present target outlines and thermal target areas even in dark or backlighting conditions. However, infrared images typically lack color and fine texture, with relatively limited detail representation, making it difficult to meet the high-fidelity detail and color consistency requirements of the human eye or machine vision.

[0003] To combine the advantages of both modalities, infrared and visible light image fusion has been extensively studied. Existing methods can be broadly categorized into two types: one is traditional methods based on manually designed rules, such as image pyramids, multi-scale decomposition, sparse representation, and wavelet transform. These methods rely on pre-defined feature decomposition and fusion strategies, performing reasonably well when the scene structure is stable, but prone to problems such as detail loss, blurred edges, or insufficient fusion in complex backgrounds, dynamic changes, or situations with significant modal differences. The other type is image fusion methods based on deep learning. End-to-end models, represented by convolutional neural networks (CNNs), can automatically learn multi-level features, improving the clarity and information integrity of the fused image. Some studies have also introduced residual structures and generative adversarial networks (GANs) to enhance the synergy between multimodal features. However, CNNs have limited effective receptive fields and insufficient ability to model long-range dependencies and global context, limiting their performance in complex fusion tasks.

[0004] In recent years, the Transformer architecture has demonstrated superior performance in multimodal image processing tasks due to its powerful global modeling capabilities and flexible feature interaction methods. Compared with CNNs, Transformers are better at capturing long-range dependencies and complex modal interactions, making them a powerful tool for improving image fusion quality. Although the Transformer architecture has made some progress in the field of image fusion, its direct application to infrared and visible light images still faces several challenges.

[0005] 1) The two modalities differ significantly in data distribution, feature structure, and dynamic range, making it difficult for feature alignment and fusion strategies to simultaneously ensure global consistency and local detail fidelity; 2) Under low illumination conditions, visible light images have low signal-to-noise ratios, insufficient brightness, and missing details, making it difficult to extract effective features from the visible light side, thus affecting fusion stability; 3) Mechanisms based on global attention lack targeted compensation for local distortions introduced by degradation, easily leading to insufficient recovery of local details.

[0006] On the other hand, existing research has attempted to improve low-light image quality by introducing image enhancement before fusion, but most of these studies employ a sequential process of "enhance first, then fuse." This sequential process lacks joint optimization and is prone to the following problems: i) artifacts, color deviations, or texture distortions generated during the enhancement stage are further amplified in subsequent fusion; ii) the fusion network struggles to distinguish high-frequency signals generated by both the real scene and enhanced noise, resulting in loss of detail or color distortion; iii) the lack of joint constraints on brightness, contrast, and color consistency makes it difficult to achieve a balance between brightness enhancement and detail preservation. Meanwhile, stacking a large number of parameterized modules in pursuit of performance leads to high computational and storage overhead, hindering real-time deployment on edge devices such as monitoring terminals and automotive components. Summary of the Invention

[0007] In view of the above, the main objective of this invention is to propose a method and system for enhancing and fusing infrared and visible light images in low-light scenes, so as to solve the above-mentioned technical problems.

[0008] This invention proposes a method for enhancing and fusing infrared and visible light images in low-light scenes, the method comprising the following steps: Step 1: Construct an encoder and decoder containing multi-adaptive Transformer units based on the Transformer structure. The encoder, decoder, and fusion module together constitute an image enhancement fusion model. Step 2: Acquire infrared and visible light images, preprocess the infrared and visible light images to obtain normalized infrared images, brightness images and illumination prior images respectively; Step 3: Stitch the brightness image and the prior illumination image into a dual-channel visible light image, and input it along with the normalized infrared image into the network composed of the encoder and decoder. Perform single-modal reconstruction pre-training by minimizing the reconstruction loss function to obtain the pre-trained encoder and pre-trained decoder. Step 4: Use the pre-trained encoder to perform deep feature extraction on the dual-channel visible light image and normalized infrared image to generate their respective query vectors, key vectors and value vectors. Then, adopt a cross-modal cross-attention mechanism and perform bidirectional feature guidance and fusion by exchanging query vectors in the fusion module to obtain fused features. Finally, reconstruct the fused image of the brightness channel through the pre-trained decoder. Step 5: Concatenate the fused image of the luminance channel with the chroma channel extracted from the visible light image and convert it to RGB space to obtain a color fused image; construct texture loss, contrast loss and color consistency loss based on the color fused image, jointly optimize the decoder and fusion module to obtain an optimized image enhancement fusion model, and input the infrared image and visible light image into the optimized image enhancement fusion model to obtain the final color fused image.

[0009] This invention also proposes an infrared and visible light image enhancement and fusion system for low-light scenes, the system comprising: The image acquisition and preprocessing module is used for: An encoder and decoder containing multiple adaptive Transformer units are constructed based on the Transformer structure. The encoder, decoder, and fusion module together constitute an image enhancement fusion model. Infrared and visible light images are acquired, and the infrared and visible light images are preprocessed to obtain normalized infrared images, brightness images, and illumination prior images, respectively. The feature extraction network pre-training module is used for: The brightness image and the prior illumination image are stitched together to form a dual-channel visible light image, which is then input into a network consisting of an encoder and a decoder along with a normalized infrared image. Single-modal reconstruction pre-training is performed by minimizing the reconstruction loss function to obtain a pre-trained encoder and a pre-trained decoder. The cross-modal feature fusion module is used for: A pre-trained encoder is used to perform deep feature extraction on dual-channel visible light images and normalized infrared images to generate their respective query vectors, key vectors and value vectors. A cross-modal cross-attention mechanism is adopted and bidirectional feature guidance and fusion are performed by exchanging query vectors in the fusion module to obtain fused features. The fused image of the brightness channel is then reconstructed by a pre-trained decoder. The color restoration and output optimization module is used for: The fused image of the luminance channel is concatenated with the chroma channel extracted from the visible light image and converted to RGB space to obtain a color fused image. Based on the color fused image, texture loss, contrast loss and color consistency loss are constructed. The decoder and fusion module are jointly optimized to obtain an optimized image enhancement fusion model. The infrared image and the visible light image are input into the optimized image enhancement fusion model to obtain the final color fused image.

[0010] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention integrates low-light image enhancement and cross-modal feature fusion into a unified architecture. Through end-to-end joint modeling, it achieves simultaneous optimization of brightness enhancement, detail preservation, and modal consistency, avoiding information loss caused by the fragmentation of the enhancement and fusion process.

[0011] 2. The multivariate adaptive Transformer module designed in this invention takes into account both global modeling and local detail restoration capabilities. It can effectively adapt to the distortion introduced by the enhancement process and strengthen the features of key regions, thereby improving the structural fidelity and detail representation of the fused image.

[0012] 3. This invention introduces a joint perceptual loss function that integrates texture preservation, contrast enhancement, and color consistency guidance mechanisms into a unified optimization objective, guiding the model to generate a fusion result that achieves balanced optimization in terms of brightness distribution, structural clarity, and color naturalness.

[0013] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by means of embodiments of the invention. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating the steps of an infrared and visible light image enhancement and fusion method for low-light scenes proposed in this invention.

[0015] Figure 2 This is a network diagram illustrating an infrared and visible light image enhancement and fusion method for low-light scenes proposed in this invention.

[0016] Figure 3 This is a schematic diagram of a multivariate adaptive Transformer block for an infrared and visible light image enhancement and fusion method for low-light scenes proposed in this invention.

[0017] Figure 4 This is a schematic diagram of the hybrid adaptive attention mechanism in the multivariate adaptive Transformer block of the infrared and visible light image enhancement and fusion method for low-light scenes proposed in this invention.

[0018] Figure 5This is a schematic diagram of the feature refinement feedforward network in the multivariate adaptive Transformer block of the infrared and visible light image enhancement and fusion method for low-light scenes proposed in this invention.

[0019] Figure 6 This is a framework diagram of an infrared and visible light image enhancement and fusion system for low-light scenes proposed in this invention. Detailed Implementation

[0020] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0021] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to provide some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0022] Please see Figure 1 This invention proposes a method for enhancing and fusing infrared and visible light images in low-light scenes, which includes the following steps: Step 1: Construct an encoder and decoder containing multiple adaptive Transformer units based on the Transformer structure. The encoder, decoder and fusion module together constitute an image enhancement fusion model.

[0023] It should be noted that the encoder contains one convolutional layer and four multivariate adaptive Transformer modules, and the encoder and decoder have symmetrical structures, both consisting of convolutional layers and multivariate adaptive Transformer modules; among them, the multivariate adaptive Transformer modules include a hybrid adaptive attention module and a feedforward refinement module.

[0024] Step 2: Acquire infrared and visible light images, and preprocess the infrared and visible light images to obtain normalized infrared images, brightness images, and illumination prior images, respectively.

[0025] In step 2, infrared and visible light images are acquired, and preprocessed to obtain normalized infrared, brightness, and illumination prior images, respectively. The specific steps include: Infrared and visible light images are acquired, the visible light image is converted to the YCbCr color space, and the brightness channel image is extracted based on the YCbCr color space. The brightness channel image is enhanced based on the illumination prior estimation method to obtain the illumination prior image; The infrared image, brightness channel image, and illumination prior image are linearly normalized to obtain the normalized infrared image, brightness image, and illumination prior image.

[0026] Furthermore, the infrared image is kept in single-channel format, and the illumination prior image is used to enhance the image brightness and detail. The infrared image, brightness image, and illumination prior image are linearly normalized to normalize their pixel values ​​to the [0,1] range to ensure consistency in numerical distribution among different images. Then, data augmentation operations are performed on the normalized image data, including random cropping and random flipping, to improve the robustness and generalization ability of the model during training. The augmented image samples are divided into training and validation sets at a ratio of 80% and 20% for subsequent network training and performance evaluation.

[0027] Step 3: Stitch the brightness image and the prior illumination image into a dual-channel visible light image, and input it along with the normalized infrared image into the network composed of the encoder and decoder. Perform single-modal reconstruction pre-training by minimizing the reconstruction loss function to obtain the pre-trained encoder and pre-trained decoder.

[0028] Please see Figure 2 In step 3, the brightness image and the prior illumination image are stitched together to form a dual-channel visible light image, which is then input into the network consisting of the encoder and decoder along with the normalized infrared image. Single-modal reconstruction pre-training is performed by minimizing the reconstruction loss function to obtain the pre-trained encoder and pre-trained decoder. The specific steps include the following: The brightness image and the prior illumination image are stitched together in the channel dimension to obtain a dual-channel visible light image; The dual-channel visible light image and the normalized infrared image are input into the encoder respectively to extract deep features, thus obtaining the features extracted from the visible light image and the features extracted from the infrared image. The features extracted from the visible light image and the features extracted from the infrared image are respectively input into the decoder for image reconstruction to obtain the visible light image reconstruction result and the infrared image reconstruction result; The reconstruction loss of visible light images is constructed based on the reconstruction results of visible light images, and the reconstruction loss of infrared images is constructed based on the reconstruction results of infrared images; The encoder and decoder are pre-trained using the reconstruction loss of visible light images and the reconstruction loss of infrared images to obtain pre-trained encoders and pre-trained decoders.

[0029] The brightness image and the prior illumination image are input into the encoder and stitched together along the channel dimension to obtain a dual-channel visible light image. The corresponding relationship in this process is as follows: ; in, This represents a dual-channel visible light image. This indicates a splicing operation. Represents a brightness image. Represents the prior image of illumination; In the steps of inputting the dual-channel visible light image and the normalized infrared image into the encoder to extract deep features and obtain the features extracted from the visible light image and the infrared image, the corresponding relationship in the process is as follows: ; in, This represents the features extracted from a visible light image. Indicates encoder, This represents the features extracted from infrared images. This represents the normalized infrared image. In the steps of inputting features extracted from visible light images and infrared images into the decoder for image reconstruction, respectively, to obtain the visible light image reconstruction result and the infrared image reconstruction result, the corresponding process relationships are as follows: ; in, This represents the result of visible light image reconstruction. Indicates decoder, This indicates the result of infrared image reconstruction; In the step of constructing the reconstruction loss of a visible light image based on the visible light image reconstruction results, the corresponding relationship in the process is as follows: ; in, This represents the reconstruction loss of a visible light image. Represents the square of the L2 norm. This represents the weighting coefficient of the SSIM loss term. Indicates structural similarity in visible light images; It should be noted that, In this embodiment, the value is 5; In the step of constructing the reconstruction loss of an infrared image based on the infrared image reconstruction results, the corresponding relationship in the process is as follows: ; in, This represents the reconstruction loss of the infrared image. Indicates the structural similarity of infrared images.

[0030] In the process of constructing the reconstruction loss of infrared images and the reconstruction loss of visible light images, there is also a total reconstruction loss, and the corresponding relationship in the process is as follows: ; in, This indicates the total reconstruction losses.

[0031] Furthermore, by minimizing the reconstruction loss, the parameters in the encoder and decoder can be effectively optimized, enabling the network to maintain structural and detailed information while learning image features, thus providing a stable and semantically rich information foundation for subsequent fusion tasks.

[0032] Step 4: Use the pre-trained encoder to perform deep feature extraction on the dual-channel visible light image and the normalized infrared image to generate their respective query vectors, key vectors and value vectors. Then, adopt a cross-modal cross-attention mechanism and perform bidirectional feature guidance and fusion by exchanging query vectors in the fusion module to obtain fused features. Finally, reconstruct the fused image of the brightness channel by the pre-trained decoder.

[0033] Please see Figure 3 , Figure 4 and Figure 5 In step 4, a pre-trained encoder is used to perform deep feature extraction on the dual-channel visible light image and the normalized infrared image, generating their respective query vectors, key vectors, and value vectors. A cross-modal cross-attention mechanism is then employed, and bidirectional feature guidance and fusion are performed in the fusion module by exchanging query vectors to obtain fused features. Finally, the fused image of the brightness channel is reconstructed by a pre-trained decoder. The specific steps include the following: In the multivariate adaptive Transformer unit of the pre-trained encoder, the input feature map of the current processing layer of the pre-trained encoder is given; wherein, the multivariate adaptive Transformer unit includes a hybrid adaptive attention module and a feature refinement feedforward network; The input feature map is transformed linearly to generate the original query vector, the original key vector, and the original value vector. Apply discrete wavelet transform to the input feature map to obtain the low-frequency subband coefficients and high-frequency subband coefficients; Convolution operations are applied to the low-frequency subband coefficients and the high-frequency subband coefficients respectively to obtain the frequency domain features after low-frequency processing and the frequency domain features after high-frequency processing. The frequency domain features after low-frequency processing and the frequency domain features after high-frequency processing are restored to the spatial domain using inverse wavelet transform to obtain the repaired spatial features. The repaired spatial features and the original query vector are then fused using an adaptive weighted fusion mechanism to obtain the fused enhanced query features. The fused enhanced query features and the original key vector are combined using a hybrid adaptive attention mechanism to calculate attention weights, resulting in the final calculated attention weights. The final calculated attention weights are multiplied by the original value vector to obtain the attention-weighted feature map. The attention-weighted feature map is input into the feature refinement feedforward network, divided into two parts according to the channel dimension, and one of the two parts is subjected to local convolution to extract spatial structure information, while the other part remains unchanged. The two parts are then merged to obtain the refined feature. The refined features are mapped to a high-dimensional space through convolution, and then further divided by channel to obtain two parts of features after secondary division. A depthwise separable convolution and GeLU activation are applied to one of the features after secondary division to obtain a gated weight. The gated weight is then multiplied element-wise with the other feature of the two parts after secondary division to obtain a gated feature. The gated feature is used as the output of the feature refinement feedforward network. The output of the feature refinement feedforward network includes the output of the infrared branch and the output of the visible light branch. The outputs of the infrared branch and the visible light branch are respectively input into the hybrid adaptive attention module to extract the query vector, key vector, and value vector of the infrared branch, as well as the query vector, key vector, and value vector of the visible light branch. For the query vector, key vector, and value vector of the infrared branch, as well as the query vector, key vector, and value vector of the visible light branch, a cross-modal cross-attention mechanism is used to perform bidirectional feature guidance and fusion in the fusion module by exchanging query vectors, so as to obtain the fused features. The fused features are input into a feature refinement feedforward network to eliminate redundancy. After feature redundancy is eliminated, the image is reconstructed through a pre-trained decoder to obtain the fused image of the brightness channel.

[0034] In the multivariate adaptive Transformer unit of the pre-trained encoder, given the input feature map of the current processing layer of the pre-trained encoder, the corresponding process has the following relationship: ; in, Indicates the input feature map, Indicates batch size Number of channels Space height and space width The feature map set; In the steps of applying convolution operations to the low-frequency subband coefficients and high-frequency subband coefficients respectively to obtain the frequency domain features after low-frequency processing and the frequency domain features after high-frequency processing, the corresponding relationship in the process is as follows: ; in, This represents the processed frequency domain characteristics. This represents the spatial coordinates of the feature map in the height direction (Y-axis). This represents the spatial coordinates of the feature map in the width direction (X-axis). This represents the relative offset index of the convolution kernel in the height direction. This represents the relative offset index of the convolution kernel in the width direction. Indicates the convolution kernel weights, This indicates that the input feature map is at the location , pixels on; In the process of restoring the low-frequency and high-frequency frequency domain features to the spatial domain using inverse wavelet transform to obtain the repaired spatial features, and then using an adaptive weighted fusion mechanism to fuse the repaired spatial features with the original query vector to obtain the fused enhanced query features, the corresponding relationship in the process is as follows: ; in, This indicates enhanced query features that are integrated. and These represent two different adaptive fusion weights. Represents the original query vector. This indicates the inverse wavelet transform operation. This represents the convolution operation. This represents the discrete wavelet transform operation; It should be noted that, and The sum is 1.

[0035] In the step of combining the fused enhanced query features with the original key vector and calculating the attention weights through a hybrid adaptive attention mechanism to obtain the final calculated attention weights, the corresponding relationship in the process is as follows: ; in, This represents the final calculated attention weights. and These represent two different fusion coefficients. Represents the normalization function. This represents the transpose of the original key vector. This represents the ReLU activation function.

[0036] It should be noted that, and The sum is 1.

[0037] The attention-weighted feature map is input into the feature refinement feedforward network, where it is divided into two parts according to the channel dimension. One of the features is then subjected to local convolution to extract spatial structure information, while the other feature remains unchanged. Finally, the two parts are merged to obtain the refined feature. The corresponding relationship in this process is as follows: ; in, Indicates a refined characteristic. This indicates a local convolution operation. This represents the first part of the channel dimension division. This represents the second part of the channel dimension division; The process involves mapping refined features to a high-dimensional space via convolution, then further partitioning them by channel to obtain two sub-features. A depthwise separable convolution and activation process is applied to one of these sub-features to obtain a gating weight. Element-wise multiplication is then performed between this gating weight and the other sub-feature to obtain a gated feature. This gated feature is then used as the output of the feature refinement feedforward network. The corresponding relationships in this process are as follows: ; in, Indicates gating features, This represents the GeLU activation function. This indicates a depthwise separable convolution operation. This represents one of the two features after the secondary partitioning. This indicates element-wise multiplication. This represents the other feature among the two parts of features after the secondary partitioning; In the step of bidirectional feature guidance and fusion in the fusion module by exchanging query vectors for the query vectors, key vectors, and value vectors of the infrared branch, and the query vectors, key vectors, and value vectors of the visible light branch, a cross-modal cross-attention mechanism is used to obtain the fused features. The corresponding relationship in the process is as follows: ; in, Indicates the characteristics after fusion. This indicates an adaptive multi-head attention operation. The query vector represents the visible light branch. The key vector representing the infrared branch. The value vector representing the infrared branch. The query vector representing the infrared branch. The key vector representing the visible light branch, The value vector representing the visible light branch; It should be noted that, By exchanging query vectors, semantic complementarity and feature enhancement between different modalities are guided while maintaining structural information alignment. The first term enhances the visible light region of interest with infrared features, while the second term reverses the infrared semantic information with visible light features, thereby achieving bidirectional semantic coupling and spatial detail matching.

[0038] In the steps of inputting the fused features into a feature refinement feedforward network to eliminate redundancy, and then reconstructing the fused image of the brightness channel using a pre-trained decoder after feature redundancy elimination, the corresponding relationship in the process is as follows: ; in, This represents a fused image of the luminance channels. This represents the pre-trained decoder.

[0039] Furthermore, effective signals are enhanced through gating mechanisms while suppressing redundant features, achieving refined contextual modeling. The aforementioned multivariate adaptive attention and feature refinement feedforward network combination unit serves as a basic component of the encoder and decoder structure and is repeatedly superimposed throughout the training process, improving the robustness and generalization ability of the model in low-light, degradation, and multi-source scenarios.

[0040] Step 5: Concatenate the fused image of the luminance channel with the chroma channel extracted from the visible light image and convert it to RGB space to obtain a color fused image; construct texture loss, contrast loss and color consistency loss based on the color fused image, jointly optimize the decoder and fusion module to obtain an optimized image enhancement fusion model, and input the infrared image and visible light image into the optimized image enhancement fusion model to obtain the final color fused image.

[0041] In step 5, the fused image of the luminance channel is concatenated with the chroma channel extracted from the visible light image and converted to the RGB space to obtain a color fused image, specifically including: The fused image of the luminance channel and the extracted chroma channel of the visible light image are concatenated to obtain the concatenated features. The concatenated features are then converted from YCbCr space to RGB space using a color space conversion function to obtain the color fused image. The corresponding relationship in this process is as follows: ; in, Represents a color-blended image. This indicates a color space conversion operation. This represents the first chroma channel extracted from the visible light image. This represents the second chromaticity channel extracted from the visible light image.

[0042] Based on the color fused image, texture loss, contrast loss, and color consistency loss are constructed. The relationship between the texture loss and the corresponding process is as follows: ; in, Indicates texture loss. This indicates that the Sobel operator is used to extract the image gradient. Describing the L1 norm, This indicates taking the absolute value of the pixels. This indicates the calculation of the maximum value; The relationship between contrast loss and the corresponding process is as follows: ; in, Indicates contrast loss. This represents the function for calculating standard deviation. The relationship between color consistency loss and the process is as follows: ; in, Indicates loss of color consistency. Indicates the arccosine operation. The operation representing the inner product, This represents the L2 norm.

[0043] In the process of constructing texture loss, contrast loss, and color consistency loss, there is also a joint loss, and the corresponding relationship is as follows: ; in, Indicates joint loss, , and These represent the three hyperparameters.

[0044] Furthermore, guided by joint loss, this embodiment improves the color consistency and perceptual quality of the fused image while maintaining structural clarity and natural contrast; the final output image has both infrared sensing capability and visible light visual realism, meeting the multimodal image enhancement needs in complex low-light scenes.

[0045] Please see Figure 6 The present invention also provides an infrared and visible light image enhancement and fusion system for low-light scenes, the system comprising: The image acquisition and preprocessing module is used for: An encoder and decoder containing multiple adaptive Transformer units are constructed based on the Transformer structure. The encoder, decoder, and fusion module together constitute an image enhancement fusion model. Infrared and visible light images are acquired, and the infrared and visible light images are preprocessed to obtain normalized infrared images, brightness images, and illumination prior images, respectively. The feature extraction network pre-training module is used for: The brightness image and the prior illumination image are stitched together to form a dual-channel visible light image, which is then input into a network consisting of an encoder and a decoder along with a normalized infrared image. Single-modal reconstruction pre-training is performed by minimizing the reconstruction loss function to obtain a pre-trained encoder and a pre-trained decoder. The cross-modal feature fusion module is used for: A pre-trained encoder is used to perform deep feature extraction on dual-channel visible light images and normalized infrared images to generate their respective query vectors, key vectors and value vectors. A cross-modal cross-attention mechanism is adopted and bidirectional feature guidance and fusion are performed by exchanging query vectors in the fusion module to obtain fused features. The fused image of the brightness channel is then reconstructed by a pre-trained decoder. The color restoration and output optimization module is used for: The fused image of the luminance channel is concatenated with the chroma channel extracted from the visible light image and converted to RGB space to obtain a color fused image. Based on the color fused image, texture loss, contrast loss and color consistency loss are constructed. The decoder and fusion module are jointly optimized to obtain an optimized image enhancement fusion model. The infrared image and the visible light image are input into the optimized image enhancement fusion model to obtain the final color fused image.

[0046] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0047] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0048] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method for enhancing and fusing infrared and visible light images in low-light scenes, characterized in that, The method includes the following steps: Step 1: Construct an encoder and decoder containing multi-adaptive Transformer units based on the Transformer structure. The encoder, decoder, and fusion module together constitute an image enhancement fusion model. Step 2: Acquire infrared and visible light images, preprocess the infrared and visible light images to obtain normalized infrared images, brightness images and illumination prior images respectively; Step 3: Stitch the brightness image and the prior illumination image into a dual-channel visible light image, and input it along with the normalized infrared image into the network composed of the encoder and decoder. Perform single-modal reconstruction pre-training by minimizing the reconstruction loss function to obtain the pre-trained encoder and pre-trained decoder. Step 4: Use the pre-trained encoder to perform deep feature extraction on the dual-channel visible light image and normalized infrared image to generate their respective query vectors, key vectors and value vectors. Then, use a cross-modal cross-attention mechanism to perform bidirectional feature guidance and fusion in the fusion module by exchanging query vectors to obtain fused features. Finally, use the pre-trained decoder to reconstruct the fused image of the brightness channel. Step 5: Concatenate the fused image of the luminance channel with the chroma channel extracted from the visible light image and convert it to RGB space to obtain a color fused image; construct texture loss, contrast loss and color consistency loss based on the color fused image, jointly optimize the decoder and fusion module to obtain an optimized image enhancement fusion model, and input the infrared image and visible light image into the optimized image enhancement fusion model to obtain the final color fused image.

2. The infrared and visible light image enhancement and fusion method for low-light scenes according to claim 1, characterized in that, In step 2, infrared and visible light images are acquired, and the infrared and visible light images are preprocessed to obtain normalized infrared images, brightness images, and illumination prior images, respectively. Specifically, the steps include: Infrared and visible light images are acquired, the visible light image is converted to the YCbCr color space, and the brightness channel image is extracted based on the YCbCr color space. The brightness channel image is enhanced based on the illumination prior estimation method to obtain the illumination prior image; The infrared image, brightness channel image, and illumination prior image are linearly normalized to obtain the normalized infrared image, brightness image, and illumination prior image.

3. The infrared and visible light image enhancement and fusion method for low-light scenes according to claim 2, characterized in that, In step 3, the brightness image and the prior illumination image are stitched together to form a dual-channel visible light image, which is then input into the network consisting of the encoder and decoder along with the normalized infrared image. Single-modal reconstruction pre-training is performed by minimizing the reconstruction loss function to obtain the pre-trained encoder and pre-trained decoder. Specifically, the steps are as follows: The brightness image and the prior illumination image are stitched together in the channel dimension to obtain a dual-channel visible light image; The dual-channel visible light image and the normalized infrared image are input into the encoder to extract deep features, so as to obtain the features extracted from the visible light image and the features extracted from the infrared image. The features extracted from the visible light image and the features extracted from the infrared image are respectively input into the decoder for image reconstruction, and the visible light image reconstruction result and the infrared image reconstruction result are obtained. The reconstruction loss of visible light images is constructed based on the reconstruction results of visible light images, and the reconstruction loss of infrared images is constructed based on the reconstruction results of infrared images; The encoder and decoder are pre-trained using the reconstruction loss of visible light images and the reconstruction loss of infrared images to obtain pre-trained encoders and pre-trained decoders.

4. The infrared and visible light image enhancement and fusion method for low-light scenes according to claim 3, characterized in that, The brightness image and the prior illumination image are input into the encoder and stitched together along the channel dimension to obtain a dual-channel visible light image. The corresponding relationship in this process is as follows: ； in, This represents a dual-channel visible light image. This indicates a splicing operation. Represents a brightness image. Represents the prior image of illumination; In the steps of inputting the dual-channel visible light image and the normalized infrared image into the encoder to extract deep features and obtain the features extracted from the visible light image and the infrared image, the corresponding relationship in the process is as follows: ； in, This represents the features extracted from a visible light image. Indicates encoder, This represents the features extracted from infrared images. This represents the normalized infrared image. In the steps of inputting features extracted from visible light images and infrared images into the decoder for image reconstruction, respectively, to obtain the visible light image reconstruction result and the infrared image reconstruction result, the corresponding process relationships are as follows: ； in, This represents the result of visible light image reconstruction. Indicates decoder, This indicates the result of infrared image reconstruction; In the step of constructing the reconstruction loss of a visible light image based on the visible light image reconstruction results, the corresponding relationship in the process is as follows: ； in, This represents the reconstruction loss of a visible light image. Represents the square of the L2 norm. This represents the weighting coefficient of the SSIM loss term. Indicates structural similarity in visible light images; In the step of constructing the reconstruction loss of an infrared image based on the infrared image reconstruction results, the corresponding relationship in the process is as follows: ； in, This represents the reconstruction loss of the infrared image. Indicates the structural similarity of infrared images.

5. The infrared and visible light image enhancement and fusion method for low-light scenes according to claim 4, characterized in that, In step 4, a pre-trained encoder is used to perform deep feature extraction on the dual-channel visible light image and the normalized infrared image, generating their respective query vectors, key vectors, and value vectors. A cross-modal cross-attention mechanism is then employed, and bidirectional feature guidance and fusion are performed in the fusion module by exchanging query vectors to obtain fused features. Finally, the fused image of the brightness channel is reconstructed by a pre-trained decoder. The specific steps include the following: In the multivariate adaptive Transformer unit of the pre-trained encoder, the input feature map of the current processing layer of the pre-trained encoder is given; wherein, the multivariate adaptive Transformer unit includes a hybrid adaptive attention module and a feature refinement feedforward network; The input feature map is transformed linearly to generate the original query vector, the original key vector, and the original value vector. Apply discrete wavelet transform to the input feature map to obtain the low-frequency subband coefficients and high-frequency subband coefficients; Convolution operations are applied to the low-frequency subband coefficients and the high-frequency subband coefficients respectively to obtain the frequency domain features after low-frequency processing and the frequency domain features after high-frequency processing. The frequency domain features after low-frequency processing and the frequency domain features after high-frequency processing are restored to the spatial domain using inverse wavelet transform to obtain the repaired spatial features. The repaired spatial features and the original query vector are then fused using an adaptive weighted fusion mechanism to obtain the fused enhanced query features. The fused enhanced query features and the original key vector are combined using a hybrid adaptive attention mechanism to calculate attention weights, resulting in the final calculated attention weights. The final calculated attention weights are multiplied by the original value vector to obtain the attention-weighted feature map. The attention-weighted feature map is input into the feature refinement feedforward network, divided into two parts according to the channel dimension, and one of the two parts is subjected to local convolution to extract spatial structure information, while the other part remains unchanged. The two parts are then merged to obtain the refined feature. The refined features are mapped to a high-dimensional space through convolution, and then further divided by channel to obtain two parts of features after secondary division. A depthwise separable convolution and GeLU activation are applied to one of the features after secondary division to obtain a gated weight. The gated weight is then multiplied element-wise with the other feature of the two parts after secondary division to obtain a gated feature. The gated feature is used as the output of the feature refinement feedforward network. The output of the feature refinement feedforward network includes the output of the infrared branch and the output of the visible light branch. The outputs of the infrared branch and the visible light branch are respectively input into the hybrid adaptive attention module to extract the query vector, key vector, and value vector of the infrared branch, as well as the query vector, key vector, and value vector of the visible light branch. For the query vector, key vector, and value vector of the infrared branch, as well as the query vector, key vector, and value vector of the visible light branch, a cross-modal cross-attention mechanism is used to perform bidirectional feature guidance and fusion in the fusion module by exchanging query vectors, so as to obtain the fused features. The fused features are input into a feature refinement feedforward network to eliminate redundancy. After feature redundancy is eliminated, the image is reconstructed through a pre-trained decoder to obtain the fused image of the brightness channel.

6. The infrared and visible light image enhancement and fusion method for low-light scenes according to claim 5, characterized in that, In the multivariate adaptive Transformer unit of the pre-trained encoder, given the input feature map of the current processing layer of the pre-trained encoder, the corresponding process has the following relationship: ； in, Indicates the input feature map, Indicates batch size Number of channels Space height and space width The feature map set; In the steps of applying convolution operations to the low-frequency subband coefficients and high-frequency subband coefficients respectively to obtain the frequency domain features after low-frequency processing and the frequency domain features after high-frequency processing, the corresponding relationship in the process is as follows: ； in, This represents the processed frequency domain characteristics. This represents the spatial coordinates of the feature map in the height direction. This represents the spatial coordinates of the feature map in the width direction. This represents the relative offset index of the convolution kernel in the height direction. This represents the relative offset index of the convolution kernel in the width direction. Indicates the convolution kernel weights. This indicates that the input feature map is at the location , pixels on; In the process of restoring the low-frequency and high-frequency frequency domain features to the spatial domain using inverse wavelet transform to obtain the repaired spatial features, and then using an adaptive weighted fusion mechanism to fuse the repaired spatial features with the original query vector to obtain the fused enhanced query features, the corresponding relationship in the process is as follows: ； in, This indicates enhanced query features that are integrated. and These represent two different adaptive fusion weights. Represents the original query vector. This indicates the inverse wavelet transform operation. This represents the convolution operation. This represents the discrete wavelet transform operation; In the step of combining the fused enhanced query features with the original key vector and calculating the attention weights through a hybrid adaptive attention mechanism to obtain the final calculated attention weights, the corresponding relationship in the process is as follows: ； in, This represents the final calculated attention weights. and These represent two different fusion coefficients. Represents the normalization function. This represents the transpose of the original key vector. This represents the ReLU activation function.

7. The infrared and visible light image enhancement and fusion method for low-light scenes according to claim 6, characterized in that, The attention-weighted feature map is input into the feature refinement feedforward network, where it is divided into two parts according to the channel dimension. One of the features is then subjected to local convolution to extract spatial structure information, while the other feature remains unchanged. Finally, the two parts are merged to obtain the refined feature. The corresponding relationship in this process is as follows: ； in, Indicates a refined characteristic. This indicates a local convolution operation. This represents the first part of the channel dimension division. This represents the second part of the channel dimension division; The process involves mapping refined features to a high-dimensional space via convolution, then further partitioning them by channel to obtain two sub-features. A depthwise separable convolution and activation process is applied to one of these sub-features to obtain a gating weight. Element-wise multiplication is then performed between this gating weight and the other sub-feature to obtain a gated feature. This gated feature is then used as the output of the feature refinement feedforward network. The corresponding relationships in this process are as follows: ； in, Indicates gating features, This represents the GeLU activation function. This indicates a depthwise separable convolution operation. This represents one of the two features after the secondary partitioning. This indicates element-wise multiplication. This represents the other feature among the two parts of features after the secondary partitioning; In the step of bidirectional feature guidance and fusion in the fusion module by exchanging query vectors for the query vectors, key vectors, and value vectors of the infrared branch, and the query vectors, key vectors, and value vectors of the visible light branch, a cross-modal cross-attention mechanism is used to obtain the fused features. The corresponding relationship in the process is as follows: ； in, Indicates the characteristics after fusion. This indicates an adaptive multi-head attention operation. The query vector represents the visible light branch. The key vector representing the infrared branch, The value vector representing the infrared branch. The query vector representing the infrared branch. The key vector representing the visible light branch, The value vector representing the visible light branch; In the steps of inputting the fused features into a feature refinement feedforward network to eliminate redundancy, and then reconstructing the fused image of the brightness channel using a pre-trained decoder after feature redundancy elimination, the corresponding relationship in the process is as follows: ； in, This represents a fused image of the luminance channels. This represents the pre-trained decoder.

8. The infrared and visible light image enhancement and fusion method for low-light scenes according to claim 7, characterized in that, In step 5, the fused image of the luminance channel is concatenated with the chroma channel extracted from the visible light image and converted to the RGB space to obtain a color fused image, specifically including: The fused image of the luminance channel and the extracted chroma channel of the visible light image are concatenated to obtain the concatenated features. The concatenated features are then converted from YCbCr space to RGB space using a color space conversion function to obtain the color fused image. The corresponding relationship in this process is as follows: ； in, Represents a color-blended image. This indicates a color space conversion operation. This represents the first chroma channel extracted from the visible light image. This represents the second chromaticity channel extracted from the visible light image.

9. The infrared and visible light image enhancement and fusion method for low-light scenes according to claim 8, characterized in that, Based on the color fused image, texture loss, contrast loss, and color consistency loss are constructed. The relationship between the texture loss and the corresponding process is as follows: ； in, Indicates texture loss, This indicates that the Sobel operator is used to extract image gradients. Describing the L1 norm, This indicates taking the absolute value of the pixels. This indicates the calculation of the maximum value; The relationship between contrast loss and the corresponding process is as follows: ； in, Indicates contrast loss. This represents the function for calculating standard deviation. The relationship between color consistency loss and the process is as follows: ； in, Indicates loss of color consistency. Indicates the arccosine operation. The operation representing the inner product, This represents the L2 norm.

10. A system for enhancing and fusing infrared and visible light images in low-light scenes, characterized in that, The system employs an infrared and visible light image enhancement and fusion method for low-light scenes as described in any one of claims 1 to 9, and the system comprises: The image acquisition and preprocessing module is used for: An encoder and decoder containing multiple adaptive Transformer units are constructed based on the Transformer structure. The encoder, decoder, and fusion module together constitute an image enhancement fusion model. Infrared and visible light images are acquired, and the infrared and visible light images are preprocessed to obtain normalized infrared images, brightness images, and illumination prior images, respectively. The feature extraction network pre-training module is used for: The brightness image and the prior illumination image are stitched together to form a dual-channel visible light image, which is then input into a network consisting of an encoder and a decoder along with a normalized infrared image. Single-modal reconstruction pre-training is performed by minimizing the reconstruction loss function to obtain a pre-trained encoder and a pre-trained decoder. The cross-modal feature fusion module is used for: A pre-trained encoder is used to perform deep feature extraction on dual-channel visible light images and normalized infrared images to generate their respective query vectors, key vectors and value vectors. A cross-modal cross-attention mechanism is adopted and bidirectional feature guidance and fusion are performed by exchanging query vectors in the fusion module to obtain fused features. The fused image of the brightness channel is then reconstructed by a pre-trained decoder. The color restoration and output optimization module is used for: The fused image of the luminance channel is concatenated with the chroma channel extracted from the visible light image and converted to RGB space to obtain a color fused image. Based on the color fused image, texture loss, contrast loss and color consistency loss are constructed. The decoder and fusion module are jointly optimized to obtain an optimized image enhancement fusion model. The infrared image and the visible light image are input into the optimized image enhancement fusion model to obtain the final color fused image.

Images