A method and system for enhancing and fusing infrared and visible light images in low-light scenes

Abstract

This invention proposes a method and system for enhancing and fusing infrared and visible light images in low-light scenes. The method includes: acquiring infrared and visible light images and preprocessing them; inputting brightness images and prior illumination images into an encoder and performing single-modal reconstruction pre-training by minimizing the reconstruction loss function; extracting deep features from the dual-channel visible light image and the normalized infrared image using the pre-trained encoder; employing a cross-modal cross-attention mechanism to perform bidirectional feature guidance and fusion in the fusion module by exchanging query vectors; and reconstructing the fused features using a pre-trained decoder. This invention integrates low-light image enhancement and cross-modal feature fusion into a unified architecture, achieving simultaneous optimization of brightness enhancement, detail preservation, and modal consistency through end-to-end joint modeling, thus avoiding information loss caused by the separation of enhancement and fusion processes.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing and computer vision, and particularly relates to an infrared and visible light image enhancement fusion method and system for low-illumination scenes. BACKGROUND

[0002] In low-light environments such as night, backlight, haze, etc., traditional image acquisition methods often face problems such as insufficient brightness, blurred details, and rising noise, which leads to a significant decrease in visual information quality, and further restricts the application of scenes with high requirements for image perception accuracy such as intelligent security, night driving, border monitoring, and military reconnaissance. Visible light images have rich texture and color information and can provide clear scene structure, but under insufficient light conditions, they are prone to problems such as brightness attenuation, image blurring, and low signal-to-noise ratio. In contrast, infrared images are based on target thermal radiation imaging and do not rely on external visible light illumination, and can still stably present target outlines and thermal target regions under dark or backlight conditions. However, infrared images usually lack color and fine texture, and the details are relatively limited, making it difficult to meet the needs of human eyes or machine vision for high-fidelity details and color consistency.

[0003] To integrate the advantages of the two modalities, infrared and visible light image fusion has been widely studied. Existing methods can be roughly divided into two categories: one is the traditional method based on artificial design rules, such as image pyramid, multi-scale decomposition, sparse representation, and wavelet transform. This type of method relies on pre-set feature decomposition and fusion strategies, and the effect is acceptable when the scene structure is stable, but in the case of complex background, dynamic change, or significant modal difference, it is easy to have problems such as detail loss, edge blurring, or insufficient fusion. The other is the image fusion method based on deep learning, with the end-to-end model represented by the convolutional neural network (CNN) being able to automatically learn multi-level features and improve the clarity and information integrity of the fused image. Some research also introduces residual structures and generative adversarial networks (GAN) to strengthen the cooperation between multi-modal features. However, the effective receptive field of CNN is limited, and its modeling ability for long-range dependencies and global context is insufficient, which limits its performance in complex fusion tasks.

[0004] In recent years, the Transformer architecture has shown superior performance in multi-modal image processing tasks due to its strong global modeling ability and flexible feature interaction. Compared with CNN, Transformer is better at capturing long-range dependencies and complex modal interaction relationships, and has become a powerful tool for improving image fusion quality. Although the Transformer structure has made some progress in the field of image fusion, its direct application to infrared and visible light images still faces multiple challenges.

[0005] 1) Two modalities differ significantly in data distribution, feature structure and dynamic range, and the feature alignment and fusion strategy is difficult to simultaneously consider global consistency and local detail fidelity; 2) Under low-light conditions, the signal-to-noise ratio of the visible light image is low, the brightness is insufficient and the details are missing, which makes it difficult to extract effective features on the visible light side, thereby affecting the stability of the fusion; 3) The mechanism mainly based on global attention lacks targeted compensation for local distortion introduced by degradation, and local detail recovery is prone to be insufficient.

[0006] On the other hand, existing research attempts to introduce image enhancement before fusion to improve the quality of low-light images, but most of them use a serial process of "enhancement first, then fusion". Such a serial process lacks joint optimization and is prone to the following problems: i) artifacts, color deviations or texture damage generated in the enhancement stage are further amplified in the subsequent fusion; ii) the fusion network has difficulty in distinguishing high-frequency signals generated by the real scene and enhanced noise, resulting in loss of details or color distortion; iii) lack of joint constraints for brightness, contrast and color consistency, making it difficult to balance between brightness enhancement and detail fidelity. At the same time, the practice of stacking a large number of parameterized modules in pursuit of performance will bring high computational and storage overhead, which is not conducive to real-time deployment on edge devices such as monitoring terminals and vehicles. SUMMARY

[0007] In view of the above situation, the main purpose of the present application is to propose an infrared and visible light image enhancement fusion method and system for low-light scenes to solve the above technical problems.

[0008] The present application proposes an infrared and visible light image enhancement fusion method for low-light scenes, which comprises the following steps:

[0009] Step 1, an encoder and a decoder containing multiple adaptive Transformer units are constructed based on the Transformer structure, and the encoder, the decoder and the fusion module jointly constitute an image enhancement fusion model;

[0010] Step 2, an infrared image and a visible light image are obtained, and the infrared image and the visible light image are preprocessed to obtain a normalized infrared image, a brightness image and an illumination prior image, respectively;

[0011] Step 3, the brightness image and the illumination prior image are spliced into a two-channel visible light image, and input into the network composed of the encoder and the decoder, and single-modality reconstruction pre-training is performed by minimizing the reconstruction loss function, to obtain a pre-trained encoder and a pre-trained decoder;

[0012] Step 4, deep feature extraction is performed on the dual-channel visible light image and the normalized infrared image by using the pre-trained encoder to generate respective query vectors, key vectors and value vectors, and a cross-modal cross-attention mechanism is adopted and bidirectional feature guidance and fusion are performed by exchanging the query vectors in the fusion module to obtain fusion features, and then the pre-trained decoder is used for reconstruction to obtain a fusion image of the luminance channel;

[0013] Step 5, the fusion image of the luminance channel and the chrominance channel extracted from the visible light image are spliced and converted to the RGB space to obtain a color fusion image; texture loss, contrast loss and color consistency loss are constructed based on the color fusion image, the decoder and the fusion module are jointly optimized to obtain an optimized image enhancement fusion model, and the infrared image and the visible light image are input into the optimized image enhancement fusion model to obtain a final color fusion image.

[0014] The application also provides an infrared and visible light image enhancement fusion system for a low-illumination scene.

[0015] An image acquisition and preprocessing module is configured to:

[0016] An encoder and a decoder containing multiple adaptive Transformer units are constructed based on a Transformer structure, and the encoder, the decoder and the fusion module jointly constitute an image enhancement fusion model.

[0017] An infrared image and a visible light image are acquired, and the infrared image and the visible light image are preprocessed to obtain a normalized infrared image, a luminance image and an illumination prior image respectively.

[0018] A feature extraction network pretraining module is configured to:

[0019] The luminance image and the illumination prior image are spliced into a dual-channel visible light image, and the dual-channel visible light image and the normalized infrared image are input into a network composed of the encoder and the decoder to perform single-modal reconstruction pretraining by minimizing a reconstruction loss function, so that the pre-trained encoder and the pre-trained decoder are obtained.

[0020] A cross-modal feature fusion module is configured to:

[0021] Deep feature extraction is performed on the dual-channel visible light image and the normalized infrared image by using the pre-trained encoder to generate respective query vectors, key vectors and value vectors, and a cross-modal cross-attention mechanism is adopted and bidirectional feature guidance and fusion are performed by exchanging the query vectors in the fusion module to obtain fusion features, and then the pre-trained decoder is used for reconstruction to obtain a fusion image of the luminance channel.

[0022] A color recovery and output optimization module is configured to:

[0023] The fused image of the brightness channel is spliced with the chroma channel extracted from the visible light image and converted to the RGB space to obtain a color fused image; texture loss, contrast loss and color consistency loss are constructed based on the color fused image, the decoder and the fusion module are jointly optimized to obtain an optimized image enhancement fusion model, and the infrared image and the visible light image are input into the optimized image enhancement fusion model to obtain a final color fused image.

[0024] Compared with the prior art, the present application has the following advantages:

[0025] 1、The present application integrates low-illumination image enhancement and cross-modal feature fusion into a unified architecture, realizes synchronous optimization of brightness improvement, detail preservation and modal consistency through end-to-end joint modeling, and avoids information loss caused by the separation of enhancement and fusion processes.

[0026] 2、The multi-element adaptive Transformer module designed in the present application takes into account the global modeling and local detail repair capabilities, can effectively adapt to the distortion introduced in the enhancement process and strengthen the key area features, thereby improving the structural fidelity and detail performance of the fused image.

[0027] 3、The present application introduces a joint perception loss function, which integrates texture preservation, contrast enhancement and color consistency guiding mechanism into a unified optimization target, guiding the model to generate a fusion result that is balanced in brightness distribution, structural clarity and color naturalness.

[0028] Additional aspects and advantages of the present application will be partially given in the following description, partially become obvious from the following description, or be understood by embodiments of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 A step flow chart of an infrared and visible light image enhancement fusion method for a low-illumination scene proposed by the present application.

[0030] Figure 2 A network schematic diagram of an infrared and visible light image enhancement fusion method for a low-illumination scene proposed by the present application.

[0031] Figure 3 A multi-element adaptive Transformer block schematic diagram of an infrared and visible light image enhancement fusion method for a low-illumination scene proposed by the present application.

[0032] Figure 4 A mixed adaptive attention mechanism schematic diagram in a multi-element adaptive Transformer block of an infrared and visible light image enhancement fusion method for a low-illumination scene proposed by the present application.

[0033] Figure 5A low-illumination scene infrared and visible light image enhancement fusion method multi-element adaptive Transformer block feature refining feedforward network schematic diagram is provided for the present application.

[0034] Figure 6 A low-illumination scene infrared and visible light image enhancement fusion system framework diagram is provided for the present application. DETAILED DESCRIPTION

[0035] Embodiments of the present application are described below in detail, examples of which are shown in the drawings, wherein the same or similar reference numbers are the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary, only for explaining the present application, and cannot be understood as limiting the present application.

[0036] These and other aspects of embodiments of the present application will become clear from the following description and drawings. In these descriptions and drawings, some specific embodiments of embodiments of the present application are specifically disclosed to be some ways to implement the principles of embodiments of the present application, but it should be understood that the scope of embodiments of the present application is not limited thereto.

[0037] Please refer to Figure 1 The present application provides a low-illumination scene infrared and visible light image enhancement fusion method, which comprises the following steps:

[0038] Step 1, based on the Transformer structure, an encoder and a decoder containing multi-element adaptive Transformer units are constructed, and the encoder, the decoder and the fusion module jointly constitute an image enhancement fusion model.

[0039] It should be noted that the encoder contains one convolutional layer and four multi-element adaptive Transformer modules, and the encoder and the decoder structure are symmetrical, both composed of convolutional layers and multi-element adaptive Transformer modules; wherein the multi-element adaptive Transformer module group comprises a hybrid adaptive attention module and a feedforward refining module.

[0040] Step 2, obtain infrared image and visible light image, and preprocess the infrared image and visible light image to obtain normalized infrared image, luminance image and illumination prior image respectively.

[0041] In step 2, the infrared image and the visible light image are obtained, and the infrared image and the visible light image are preprocessed to obtain normalized infrared image, luminance image and illumination prior image respectively, which comprises the following steps:

[0042] Obtaining an infrared image and a visible light image, converting the visible light image to a YCbCr color space, and extracting a luminance channel image based on the YCbCr color space;

[0043] Enhancing the luminance channel image based on a light prior estimation method to obtain a light prior image;

[0044] Performing linear normalization processing on the infrared image, the luminance channel image, and the light prior image respectively to obtain a normalized infrared image, a luminance image, and a light prior image.

[0045] Further, the infrared image remains in a single channel format, and the light prior image is used to improve image brightness and detail performance. The infrared image, the luminance image, and the light prior image are respectively subjected to linear normalization processing to normalize the pixel values to the [0, 1] interval, so as to ensure the consistency of different images in numerical distribution. Then, data enhancement operations are performed on the normalized image data, including random cropping, random flipping, etc., so as to improve the robustness and generalization ability of the model in the training process. The enhanced image samples are divided into a training set and a validation set according to a ratio of 80% and 20% for subsequent network training and performance evaluation.

[0046] Step 3, the luminance image and the light prior image are spliced into a dual-channel visible light image, and the normalized infrared image is input into a network composed of an encoder and a decoder to perform single-modal reconstruction pre-training by minimizing a reconstruction loss function, to obtain a pre-trained encoder and a pre-trained decoder.

[0047] Please refer to Figure 2 In step 3, the luminance image and the light prior image are spliced into a dual-channel visible light image, and the normalized infrared image is input into a network composed of an encoder and a decoder to perform single-modal reconstruction pre-training by minimizing a reconstruction loss function, to obtain a pre-trained encoder and a pre-trained decoder, which specifically includes the following steps:

[0048] The luminance image and the light prior image are spliced in the channel dimension to obtain a dual-channel visible light image;

[0049] The dual-channel visible light image and the normalized infrared image are respectively input into the encoder to extract deep features, to obtain features extracted from the visible light image and features extracted from the infrared image;

[0050] The features extracted from the visible light image and the features extracted from the infrared image are respectively input into the decoder for image reconstruction to obtain visible light image reconstruction results and infrared image reconstruction results;

[0051] A reconstruction loss of the visible light image is constructed based on the visible light image reconstruction result, and a reconstruction loss of the infrared image is constructed based on the infrared image reconstruction result.

[0052] The encoder and the decoder are pre-trained by using the reconstruction loss of the visible light image and the reconstruction loss of the infrared image, to obtain the pre-trained encoder and the pre-trained decoder.

[0053] The luminance image and the illumination prior image are input into the encoder, and a splicing process is performed in the channel dimension to obtain a dual-channel visible light image, and a relationship formula existing in the corresponding process is as follows:

[0054] ;

[0055] wherein, denotes the dual-channel visible light image, denotes a splicing operation, denotes the luminance image, denotes the illumination prior image;

[0056] In the step of inputting the dual-channel visible light image and the normalized infrared image into the encoder respectively to extract deep features to obtain the visible light image extracted feature and the infrared image extracted feature, a relationship formula existing in the corresponding process is as follows:

[0057] ;

[0058] wherein, denotes the visible light image extracted feature, denotes the encoder, denotes the infrared image extracted feature, denotes the normalized infrared image;

[0059] In the step of inputting the visible light image extracted feature and the infrared image extracted feature into the decoder respectively to perform image reconstruction to obtain the visible light image reconstruction result and the infrared image reconstruction result, a relationship formula existing in the corresponding process is as follows:

[0060] ;

[0061] wherein, denotes the visible light image reconstruction result, denotes the decoder, denotes the infrared image reconstruction result;

[0062] In the step of constructing the reconstruction loss of the visible light image based on the visible light image reconstruction result, a relationship formula existing in the corresponding process is as follows:

[0063] ;

[0064] wherein, denotes the reconstruction loss of the visible light image, denotes the L2 norm square, denotes the weight coefficient of the SSIM loss term, denotes the structural similarity of the visible light image;

[0065] It should be noted that, In the embodiment, the value is 5;

[0066] In the step of constructing the reconstruction loss of the infrared image based on the reconstruction result of the infrared image, the corresponding process exists the following relationship:

[0067] ;

[0068] wherein, denotes the reconstruction loss of the infrared image, denotes the structural similarity of the infrared image.

[0069] In the process of constructing the reconstruction loss of the infrared image and the reconstruction loss of the visible light image, there is also a total reconstruction loss, and the corresponding process exists the following relationship:

[0070] ;

[0071] wherein, denotes the total reconstruction loss.

[0072] Further, by minimizing the reconstruction loss, the parameters in the encoder and the decoder can be effectively optimized, so that the network can learn the image features while maintaining the structure and detail information, providing a stable and rich semantic information basis for subsequent fusion tasks.

[0073] Step 4, using the pre-trained encoder to extract deep features of the dual-channel visible light image and the normalized infrared image to generate respective query vectors, key vectors and value vectors, and using the cross-modal cross-attention mechanism and exchanging query vectors in the fusion module to guide and fuse the features in both directions to obtain fusion features, and then using the pre-trained decoder to reconstruct the fusion image of the luminance channel.

[0074] Please refer to Figure 3 , Figure 4 and Figure 5 In step 4, the pre-trained encoder is used to extract deep features of the dual-channel visible light image and the normalized infrared image to generate respective query vectors, key vectors and value vectors, and the cross-modal cross-attention mechanism is used and the query vectors are exchanged in the fusion module to guide and fuse the features in both directions to obtain fusion features, and then the pre-trained decoder is used to reconstruct the fusion image of the luminance channel, which specifically includes the following steps:

[0075] In the multi-element adaptive Transformer unit in the pre-trained encoder, an input feature map of a current processing layer of the pre-trained encoder is given; wherein the multi-element adaptive Transformer unit comprises a hybrid adaptive attention module and a feature refining feedforward network;

[0076] The input feature map is transformed by a linear transformation to generate an original query vector, an original key vector and an original value vector;

[0077] A discrete wavelet transform is applied to the input feature map to obtain low-frequency sub-band coefficients and high-frequency sub-band coefficients;

[0078] Convolution operations are respectively applied to the low-frequency sub-band coefficients and the high-frequency sub-band coefficients to obtain low-frequency processed frequency domain features and high-frequency processed frequency domain features;

[0079] The low-frequency processed frequency domain features and the high-frequency processed frequency domain features are restored to the spatial domain by an inverse wavelet transform operation to obtain repaired spatial features; the repaired spatial features and the original query vector are adaptively fused by an adaptive weighting fusion mechanism to obtain fused enhanced query features;

[0080] The fused enhanced query features and the original key vector are calculated by a hybrid adaptive attention mechanism to obtain final calculated attention weights;

[0081] The final calculated attention weights are multiplied by the original value vector to obtain an attention weighted feature map;

[0082] The attention weighted feature map is input into the feature refining feedforward network, divided into two parts of features in the channel dimension, and one of the two parts of features is applied with a local convolution to extract spatial structure information, while the other part of feature remains unchanged, and then the two parts of features are merged to obtain refined features;

[0083] The refined features are mapped to a high-dimensional space by a convolution, and then divided into two parts of features again in the channel dimension to obtain the two parts of features after the second division, one of the two parts of features after the second division is applied with a depth separable convolution and a GeLU activation process to obtain a gating weight; the gating weight is multiplied by the other part of features after the second division by an element-wise multiplication operation to obtain a gated feature, and the gated feature is taken as the output of the feature refining feedforward network; wherein the output of the feature refining feedforward network comprises an output of an infrared branch and an output of a visible light branch;

[0084] The output of the infrared branch and the output of the visible light branch are respectively input into a mixed adaptive attention module, and query vectors, key vectors and value vectors of the infrared branch and query vectors, key vectors and value vectors of the visible light branch are extracted;

[0085] The query vectors, key vectors and value vectors of the infrared branch and the query vectors, key vectors and value vectors of the visible light branch are subjected to bidirectional feature guidance and fusion in the fusion module through the cross-modal cross-attention mechanism by exchanging the query vectors, to obtain fused features;

[0086] The fused features are input into a feature refining feedforward network to eliminate redundancy, and after the feature redundancy is eliminated, a pre-trained decoder is used for reconstruction to obtain a fused image of the luminance channel.

[0087] In the multi-element adaptive Transformer unit in the pre-trained encoder, given the input feature map of the current processing layer of the pre-trained encoder, the corresponding process exists the following relationship:

[0088] ;

[0089] wherein, represents the input feature map, represents the batch size , the number of channels , the spatial height and the spatial width of the feature map set;

[0090] In the step of applying convolution operations to the low-frequency subband coefficients and the high-frequency subband coefficients respectively to obtain the low-frequency processed frequency domain features and the high-frequency processed frequency domain features, the corresponding process exists the following relationship:

[0091] ;

[0092] wherein, represents the processed frequency domain feature, represents the spatial coordinates of the feature map in the height direction (Y axis), represents the spatial coordinates of the feature map in the width direction (X axis), represents the relative offset index of the convolution kernel in the height direction, represents the relative offset index of the convolution kernel in the width direction, represents the convolution kernel weight, represents the pixel of the input feature map at position , ;

[0093] In the step of restoring the low-frequency processed frequency domain features and the high-frequency processed frequency domain features to the spatial domain by inverse wavelet transform operation to obtain the repaired spatial features, adaptively fusing the repaired spatial features and the original query vector by the adaptive weighted fusion mechanism to obtain the fused enhanced query features, the corresponding process exists the following relationship:

[0094] ;

[0095] wherein, represents the fused enhanced query features, and respectively represent two different adaptive fusion weights, represents the original query vector, represents the inverse wavelet transform operation, represents the convolution operation, represents the discrete wavelet transform operation.

[0096] It should be noted that, and and are 1.

[0097] In the step of calculating the attention weight by mixing the adaptive attention mechanism between the fused enhanced query features and the original key vector to obtain the finally calculated attention weight, the corresponding process exists the following relationship:

[0098] ;

[0099] wherein, represents the finally calculated attention weight, and respectively represent two different fusion coefficients, represents a normalization function, represents the transpose of the original key vector, represents a ReLU activation function.

[0100] It should be noted that, and add up to 1.

[0101] The attention weighted feature map is input into the feature refining feedforward network, divided into two parts of features according to the channel dimension, and one of the two parts of features is applied to local convolution to extract spatial structure information, while the other part of feature remains unchanged, then the two parts of features are merged to obtain refined features, and the corresponding process exists the following relationship:

[0102] ;

[0103] wherein, represents the refined features, This indicates a local convolution operation. This represents the first part of the channel dimension division. This represents the second part of the channel dimension division;

[0104] The process involves mapping refined features to a high-dimensional space via convolution, then further partitioning them by channel to obtain two sub-features. A depthwise separable convolution and activation process is applied to one of these sub-features to obtain a gating weight. Element-wise multiplication is then performed between this gating weight and the other sub-feature to obtain a gated feature. This gated feature is then used as the output of the feature refinement feedforward network. The corresponding relationships in this process are as follows:

[0105] ;

[0106] in, Indicates gating features, This represents the GeLU activation function. This indicates a depthwise separable convolution operation. This represents one of the two features after the secondary partitioning. This indicates element-wise multiplication. This represents the other feature among the two parts of features after the secondary partitioning;

[0107] In the step of bidirectional feature guidance and fusion in the fusion module by exchanging query vectors for the query vectors, key vectors, and value vectors of the infrared branch, and the query vectors, key vectors, and value vectors of the visible light branch, a cross-modal cross-attention mechanism is used to obtain the fused features. The corresponding relationship in the process is as follows:

[0108] ;

[0109] in, Indicates the characteristics after fusion. This indicates an adaptive multi-head attention operation. The query vector represents the visible light branch. The key vector representing the infrared branch. The value vector representing the infrared branch. The query vector representing the infrared branch. The key vector representing the visible light branch, The value vector representing the visible light branch;

[0110] It should be noted that, By exchanging query vectors, the semantic complement and feature enhancement between different modalities are guided while maintaining the alignment of structural information; the first item realizes the enhancement guidance of infrared features to visible light attention regions, and the second item realizes the reverse compensation of visible light features to infrared semantic information, thereby realizing bidirectional semantic coupling and spatial detail matching.

[0111] In the step of inputting the fused features into the feature refining feedforward network to eliminate redundancy, and reconstructing through the pre-trained decoder after the feature redundancy elimination to obtain the fused image of the luminance channel, the relationship formula existing in the corresponding process is as follows:

[0112] ;

[0113] Wherein, Y represents the fused image of the luminance channel, Y represents the pre-trained decoder.

[0114] Further, the effective signal is reinforced and the redundant features are inhibited through the gating mechanism, so as to realize the refining context modeling effect; the above multi-element adaptive attention and feature refining feedforward network combination unit is used repeatedly in the whole training process as a basic component module of the encoder and decoder structure, so as to improve the robustness and generalization ability of the model in the weak light, degradation and multi-source scene.

[0115] In step 5, the luminance channel fused image and the chroma channel extracted from the visible light image are spliced and converted to RGB space to obtain a color fused image; texture loss, contrast loss and color consistency loss are constructed based on the color fused image, the decoder and the fusion module are jointly optimized, and an optimized image enhancement fusion model is obtained, and the infrared image and the visible light image are input into the optimized image enhancement fusion model to obtain the final color fused image.

[0116] In step 5, the luminance channel fused image and the chroma channel extracted from the visible light image are spliced and converted to RGB space to obtain a color fused image, which specifically includes:

[0117] The luminance channel fused image and the chroma channel extracted from the visible light image are spliced and converted to RGB space to obtain a color fused image, which specifically includes:

[0118] ;

[0119] Wherein, Y represents the color fused image, Y represents the color space conversion operation, Y represents the first chroma channel extracted from the visible light image, a second chroma channel representing extraction of the visible light image.

[0120] The texture loss, the contrast loss and the color consistency loss are constructed based on the color fusion image, wherein the texture loss corresponds to a relationship as follows:

[0121] ;

[0122] wherein, the texture loss is represented by, the image gradient is extracted using a Sobel operator, the L1 norm is represented by, the absolute value of the pixel is represented by, the maximum value calculation is represented by;

[0123] The contrast loss corresponds to a relationship as follows:

[0124] ;

[0125] wherein, the contrast loss is represented by, the standard deviation calculation function is represented by;

[0126] The color consistency loss corresponds to a relationship as follows:

[0127] ;

[0128] wherein, the color consistency loss is represented by, the inverse cosine operation is represented by, the inner product operation is represented by, the L2 norm is represented by.

[0129] In the process of constructing the texture loss, the contrast loss and the color consistency loss, there is also a joint loss, which corresponds to a relationship as follows:

[0130] ;

[0131] wherein, the joint loss is represented by, , and represent three hyperparameters respectively.

[0132] Further, through the guidance of the joint loss, the embodiment further improves the color consistency and the perceptual quality of the fusion image while maintaining clear structure and natural contrast; the final output image has both infrared perception ability and visible light visual realism, meeting the multi-modal image enhancement requirements in complex low-illumination scenes.

[0133] Referring to Figure 6 The application also provides an infrared and visible light image enhancement fusion system for a low-illumination scene, the system comprising:

[0134] an image acquisition and preprocessing module configured to:

[0135] an encoder and a decoder comprising multi-element adaptive Transformer units based on a Transformer structure, and a fusion module, wherein the encoder, the decoder and the fusion module jointly constitute an image enhancement fusion model;

[0136] obtaining an infrared image and a visible light image, and preprocessing the infrared image and the visible light image to obtain a normalized infrared image, a luminance image and an illumination prior image;

[0137] a feature extraction network pre-training module configured to:

[0138] concatenating the luminance image and the illumination prior image into a dual-channel visible light image, and inputting the dual-channel visible light image and the normalized infrared image into a network composed of the encoder and the decoder to perform single-modal reconstruction pre-training by minimizing a reconstruction loss function, thereby obtaining a pre-trained encoder and a pre-trained decoder;

[0139] a cross-modal feature fusion module configured to:

[0140] extracting deep features of the dual-channel visible light image and the normalized infrared image by using the pre-trained encoder to generate respective query vectors, key vectors and value vectors, and performing bidirectional feature guidance and fusion by exchanging the query vectors in the fusion module by using a cross-modal cross-attention mechanism, thereby obtaining fusion features, and reconstructing the fusion features by using the pre-trained decoder to obtain a fusion image of a luminance channel;

[0141] a color recovery and output optimization module configured to:

[0142] concatenating the fusion image of the luminance channel and a chroma channel extracted from the visible light image and converting the concatenated image to an RGB space to obtain a color fusion image, constructing a texture loss, a contrast loss and a color consistency loss based on the color fusion image, jointly optimizing the decoder and the fusion module based on the texture loss, the contrast loss and the color consistency loss to obtain an optimized image enhancement fusion model, and inputting the infrared image and the visible light image into the optimized image enhancement fusion model to obtain a final color fusion image.

[0143] It should be appreciated that parts of the present application can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, several steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and as in another embodiment, any of the following technologies, or a combination thereof, can be used: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0144] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0145] The above-described embodiments only express several implementation manners of the present application, which are described in a more specific and detailed manner, but should not be understood as a limitation on the patent scope of the present application. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, which are all within the protection scope of the present application. Therefore, the patent protection scope of the present application should be subject to the appended claims.

Claims

1. An infrared and visible light image enhancement fusion method for low-illumination scenes, characterized in that, The method comprises the following steps: Step 1, constructing an encoder and a decoder comprising multi-element adaptive Transformer units based on a Transformer structure, wherein the encoder, the decoder and a fusion module jointly constitute an image enhancement fusion model; Step 2, obtaining an infrared image and a visible light image, and pre-processing the infrared image and the visible light image to obtain a normalized infrared image, a luminance image and an illumination prior image respectively; Step 3, splicing the luminance image and the illumination prior image into a dual-channel visible light image, and inputting the dual-channel visible light image and the normalized infrared image into a network composed of the encoder and the decoder, performing single-modal reconstruction pre-training by minimizing a reconstruction loss function, and obtaining a pre-trained encoder and a pre-trained decoder; Step 4, using the pre-trained encoder to extract deep features of the dual-channel visible light image and the normalized infrared image, generating respective query vectors, key vectors and value vectors, and using a cross-modal cross-attention mechanism to exchange the query vectors in the fusion module to guide and fuse the features in both directions, obtaining fusion features, and then reconstructing the fusion features through the pre-trained decoder to obtain a luminance channel fusion image; Step 5, splicing the luminance channel fusion image and the chrominance channel extracted from the visible light image and converting them to an RGB space to obtain a color fusion image; constructing a texture loss, a contrast loss and a color consistency loss based on the color fusion image, jointly optimizing the decoder and the fusion module, and obtaining an optimized image enhancement fusion model; inputting the infrared image and the visible light image into the optimized image enhancement fusion model to obtain a final color fusion image.

2. The method for infrared and visible image enhancement fusion of low-illumination scene according to claim 1, characterized in that, In the step 2, the infrared image and the visible light image are obtained, and the infrared image and the visible light image are pre-processed to obtain a normalized infrared image, a luminance image and an illumination prior image respectively, which specifically comprises the following steps: Obtaining the infrared image and the visible light image, converting the visible light image to a YCbCr color space, and extracting a luminance channel image based on the YCbCr color space; Enhancing the luminance channel image based on an illumination prior estimation method to obtain an illumination prior image; Respectively performing linear normalization processing on the infrared image, the luminance channel image and the illumination prior image to obtain a normalized infrared image, a luminance image and an illumination prior image.

3. The method for infrared and visible image enhancement fusion of low-illumination scene according to claim 2, characterized in that, In the step 3, the luminance image and the illumination prior image are spliced into a dual-channel visible light image, and the dual-channel visible light image and the normalized infrared image are input into a network composed of the encoder and the decoder, single-modal reconstruction pre-training is performed by minimizing a reconstruction loss function, a pre-trained encoder and a pre-trained decoder are obtained, and the step specifically comprises the following steps: Splicing the luminance image and the illumination prior image in the channel dimension to obtain a dual-channel visible light image; Inputting the dual-channel visible light image and the normalized infrared image into the encoder respectively to extract deep features, so as to obtain features extracted from the visible light image and features extracted from the infrared image; Inputting the features extracted from the visible light image and the features extracted from the infrared image into the decoder respectively to reconstruct images, so as to obtain a visible light image reconstruction result and an infrared image reconstruction result; A reconstruction loss of the visible light image is constructed based on the visible light image reconstruction result, and a reconstruction loss of the infrared image is constructed based on the infrared image reconstruction result; The encoder and the decoder are pre-trained by using the reconstruction loss of the visible light image and the reconstruction loss of the infrared image, to obtain the pre-trained encoder and the pre-trained decoder.

4. The method for infrared and visible image enhancement fusion of low-illumination scene according to claim 3, characterized in that, The luminance image and the illumination prior image are input into the encoder, and are spliced in the channel dimension to obtain a dual-channel visible light image, and a relationship formula corresponding to the process is as follows: ； wherein, denotes a dual-channel visible light image, denotes a stitching operation, denotes a luminance image, denotes an illumination prior image; In the step of inputting the dual-channel visible light image and the normalized infrared image into the encoder respectively to extract deep features to obtain the visible light image extracted features and the infrared image extracted features, a relationship formula corresponding to the process is as follows: ； wherein, denotes features extracted from the visible light image, denotes the encoder, denotes features extracted from the infrared image, denotes the normalized infrared image; In the step of inputting the visible light image extracted features and the infrared image extracted features into the decoder respectively for image reconstruction to obtain the visible light image reconstruction result and the infrared image reconstruction result, a relationship formula corresponding to the process is as follows: ； wherein, denotes the visible light image reconstruction result, denotes the decoder, denotes the infrared image reconstruction result; In the step of constructing the reconstruction loss of the visible light image based on the visible light image reconstruction result, a relationship formula corresponding to the process is as follows: ； wherein, denotes a reconstruction loss of the visible light image, denotes an L2 norm square, denotes a weight coefficient of the SSIM loss term, denotes a structural similarity of the visible light image; In the step of constructing the reconstruction loss of the infrared image based on the infrared image reconstruction result, a relationship formula corresponding to the process is as follows: ； wherein, denotes a reconstruction loss of the infrared image, denotes a structural similarity of the infrared image.

5. The method for infrared and visible image enhancement fusion of low-illumination scene according to claim 4, characterized in that, In the step 4, the pre-trained encoder is used to extract deep features of the dual-channel visible light image and the normalized infrared image, to generate respective query vectors, key vectors and value vectors, and a cross-modal cross-attention mechanism is adopted to exchange the query vectors in the fusion module to guide and fuse the features in both directions, to obtain fusion features, and then the pre-trained decoder is used to reconstruct the fusion image of the luminance channel, which specifically includes the following steps: In the multi-element adaptive Transformer unit in the pre-trained encoder, an input feature map of a current processing layer of the pre-trained encoder is given; wherein the multi-element adaptive Transformer unit includes a hybrid adaptive attention module and a feature refining feedforward network; The input feature map is transformed by linear transformation to generate an original query vector, an original key vector and an original value vector; Discrete wavelet transform is applied to the input feature map to obtain low-frequency sub-band coefficients and high-frequency sub-band coefficients; Convolution operations are respectively applied to the low-frequency sub-band coefficients and the high-frequency sub-band coefficients to obtain low-frequency processed frequency domain features and high-frequency processed frequency domain features; The low-frequency processed frequency domain features and the high-frequency processed frequency domain features are restored to the spatial domain by inverse wavelet transform operation to obtain repaired spatial features; the repaired spatial features and the original query vector are adaptively fused by adaptive weighting fusion mechanism to obtain fused enhanced query features; The fused enhanced query features and the original key vector are calculated by the hybrid adaptive attention mechanism to obtain the finally calculated attention weight; The finally calculated attention weight is multiplied by the original value vector to obtain an attention weighted feature map. The attention-weighted feature map is input into a feature refining feedforward network, divided into two parts of features in the channel dimension, one of the two parts of features is applied with local convolution to extract spatial structure information, the other part of feature is kept unchanged, and then the two parts of features are merged to obtain refined features; The refined features are mapped to a high-dimensional space through convolution, and are divided into two parts of features again in the channel dimension, one of the two parts of features is applied with depth separable convolution and GeLU activation processing to obtain a gating weight; the gating weight is multiplied with the other part of features to obtain a gated feature, and the gated feature is taken as the output of the feature refining feedforward network; wherein the output of the feature refining feedforward network includes the output of the infrared branch and the output of the visible light branch; The output of the infrared branch and the output of the visible light branch are input into a hybrid adaptive attention module to extract the query vector, the key vector and the value vector of the infrared branch, and the query vector, the key vector and the value vector of the visible light branch; The query vector of the infrared branch, the key vector of the infrared branch and the value vector of the infrared branch, and the query vector of the visible light branch, the key vector of the visible light branch and the value vector of the visible light branch are subjected to bidirectional feature guidance and fusion in a fusion module through cross-modal cross-attention mechanism by exchanging the query vector to obtain fused features; The fused features are input into the feature refining feedforward network to eliminate redundancy, and the features after eliminating redundancy are reconstructed by a pre-trained decoder to obtain a fused image of the luminance channel.

6. The method for infrared and visible image enhancement fusion of low-illumination scene according to claim 5, characterized in that, In the multi-element adaptive Transformer unit in the pre-trained encoder, the relationship of the corresponding process exists as follows for a given input feature map of the current processing layer of the pre-trained encoder: ； wherein, represents an input feature map, represents a batch size , a number of channels , a spatial height , and a spatial width of the feature map set; In the step of applying convolution operation on the low-frequency subband coefficient and the high-frequency subband coefficient to obtain the low-frequency processed frequency domain feature and the high-frequency processed frequency domain feature, the relationship of the corresponding process exists as follows: ； wherein, represents the processed frequency domain feature, represents the spatial coordinate of the feature map in the height direction, represents the spatial coordinate of the feature map in the width direction, represents the relative offset index of the convolution kernel in the height direction, represents the relative offset index of the convolution kernel in the width direction, represents the convolution kernel weight, represents the pixel of the input feature map at position , ; In the step of restoring the repaired spatial feature to the spatial domain from the low-frequency processed frequency domain feature and the high-frequency processed frequency domain feature by using inverse wavelet transform operation, and adaptively fusing the repaired spatial feature and the original query vector by using adaptive weighting fusion mechanism to obtain fused enhanced query features, the relationship of the corresponding process exists as follows: ； wherein, denotes a fused enhanced query feature, and denote two different adaptive fusion weights, respectively, denotes an original query vector, denotes an inverse wavelet transform operation, denotes a convolution operation, denotes a discrete wavelet transform operation; In the step of calculating attention weights by using a hybrid adaptive attention mechanism for the fused enhanced query features and the original key vector to obtain finally calculated attention weights, the relationship of the corresponding process exists as follows: ； where, denotes the final computed attention weight, and denote two different fusion coefficients, denotes a normalization function, denotes the transpose of the original key vector, denotes a ReLU activation function.

7. The method for infrared and visible image enhancement fusion of low-illumination scene according to claim 6, characterized in that, The attention-weighted feature map is input into a feature refining feedforward network, divided into two parts of features in the channel dimension, one of the two parts of features is applied with local convolution to extract spatial structure information, the other part of feature is kept unchanged, and then the two parts of features are merged to obtain refined features, the relationship of the corresponding process exists as follows: ； wherein, denotes a refining feature, denotes a local convolution operation, denotes a first part of a channel dimension split, denotes a second part of a channel dimension split; In the step of mapping the refining feature to a high-dimensional space through convolution, and performing secondary division again by channel to obtain two features after secondary division, applying a depth separable convolution and an activation process to one of the two features after secondary division to obtain a gating weight, and performing element-wise multiplication calculation on the gating weight and the other feature after secondary division to obtain a gating feature, the relationship formula corresponding to the process is as follows: ； wherein, denotes a gating feature, denotes a GeLU activation function, denotes a depthwise separable convolution operation, denotes one of the two features after the second split, denotes an element-wise multiplication computation, denotes the other of the two features after the second split; In the step of performing bidirectional feature guidance and fusion in the fusion module by exchanging query vectors through the cross-modal cross-attention mechanism, the relationship formula corresponding to the process is as follows: ； wherein, denotes the fused features, denotes the adaptive multi-head attention operation, denotes the query vector of the visible light branch, denotes the key vector of the infrared branch, denotes the value vector of the infrared branch, denotes the query vector of the infrared branch, denotes the key vector of the visible light branch, denotes the value vector of the visible light branch; In the step of inputting the fused feature into the feature refining feedforward network to eliminate redundancy, and reconstructing the luminance channel fusion image through the pre-trained decoder after eliminating the feature redundancy, the relationship formula corresponding to the process is as follows: ； wherein, represents a fused image of the luminance channel, represents a pre-trained decoder.

8. The method for infrared and visible image enhancement fusion of low-illumination scene according to claim 7, characterized in that, In the step 5, the luminance channel fusion image and the extracted chroma channel of the visible light image are spliced and converted to the RGB space to obtain a color fusion image, which specifically includes: The luminance channel fusion image and the extracted chroma channel of the visible light image are spliced and converted to the RGB space to obtain a color fusion image, which specifically includes: ； wherein, denotes a color fused image, denotes a color space conversion operation, denotes a first chroma channel of the extraction of the visible light image, denotes a second chroma channel of the extraction of the visible light image.

9. The method for infrared and visible image enhancement fusion of low-illumination scene according to claim 8, characterized in that, Based on the color fusion image, texture loss, contrast loss, and color consistency loss are constructed, wherein the relationship formula corresponding to the process of the texture loss is as follows: ； wherein, denotes texture loss, denotes extraction of image gradient using Sobel operator, denotes L1 norm, denotes taking pixel absolute value, denotes max value computation; The relationship formula corresponding to the process of the contrast loss is as follows: ； wherein, denotes the contrast loss, denotes a standard deviation computation function; The relationship formula corresponding to the process of the color consistency loss is as follows: ； wherein, denotes a loss of color consistency, denotes an inverse cosine operation, denotes an operation of an inner product, denotes an L2 norm.

10. An infrared and visible light image enhancement fusion system for low-illumination scenes, characterized in that, The system applies the infrared and visible light image enhancement fusion method for low-illumination scenes according to any one of the above claims 1 to 9, and the system includes: An image acquisition and preprocessing module for: Based on the Transformer structure, an encoder and a decoder containing multiple adaptive Transformer units are constructed, and the encoder, the decoder, and the fusion module jointly constitute an image enhancement fusion model; An infrared image and a visible light image are obtained, and the infrared image and the visible light image are preprocessed to obtain a normalized infrared image, a luminance image, and an illumination prior image; A feature extraction network pre-training module for: The luminance image and the illumination prior image are spliced into a dual-channel visible light image, and are input into the network composed of the encoder and the decoder to perform single-modal reconstruction pre-training by minimizing the reconstruction loss function, to obtain a pre-trained encoder and a pre-trained decoder; A cross-modal feature fusion module for: Deep feature extraction is performed on the dual-channel visible light image and the normalized infrared image by using a pre-trained encoder to generate respective query vectors, key vectors and value vectors, and a cross-modal cross-attention mechanism is adopted and bidirectional feature guidance and fusion is performed by exchanging the query vectors in a fusion module to obtain fused features, and then a pre-trained decoder is used to reconstruct a fused image of the luminance channel; The color restoration and output optimization module is configured to: splice the fused image of the luminance channel and the chrominance channel extracted from the visible light image and convert them to an RGB space to obtain a color fused image; construct a texture loss, a contrast loss and a color consistency loss based on the color fused image, jointly optimize the decoder and the fusion module, and obtain an optimized image enhancement fusion model; and input the infrared image and the visible light image into the optimized image enhancement fusion model to obtain a final color fused image.

Images