Infrared guided low-light image enhancement method based on global and local multi-scale fusion

Abstract

The application discloses an infrared-guided low-light image enhancement method based on global and local multi-scale fusion and belongs to the technical field of computer vision, and specifically comprises the following steps: constructing an image enhancement network; constructing a data set and training the image enhancement network; acquiring low-light images and near-infrared images under the same scene, inputting the trained image enhancement network, and obtaining an enhanced image. The infrared-guided low-light image enhancement method based on global and local multi-scale fusion solves the problems of complex calculation, single effect, no consideration of light enhancement and difficulty in realizing denoising in the prior art.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to an infrared-guided low-light image enhancement method that integrates global and local multi-scale fusion. Background Technology

[0002] Low-light scenes primarily occur when there is insufficient external light, such as nighttime, backlighting, and extremely low light conditions with no light source. The main causes of low-light images include insufficient ambient light during shooting, the position and color variations of the light source, and the exposure ratio settings of the acquisition device. These factors contribute to images that are low in brightness, high in noise, and lack detail. Image enhancement aims to improve the visual quality of low-light images, enhance the observer's perception, and thus better analyze the image content. It has significant research and application value in fields such as security monitoring, military applications, and medical imaging.

[0003] Real-world low-light image enhancement requires improving image brightness, denoising, and restoring image texture details. Most existing low-light image enhancement methods are based on single low-light images, which amplifies hidden noise while increasing brightness. The denoising process removes both texture structure and noise, resulting in the loss of crucial information. In real-world low-light nighttime scenes, near-infrared light has strong anti-interference capabilities, and acquired near-infrared images exhibit high contrast and low noise. Methods based on the fusion and enhancement of infrared and visible light images, such as the DVN method, have been proposed in recent years. Infrared image-guided low-light image enhancement methods fully utilize the high contrast and low noise characteristics of near-infrared images to solve image denoising problems. However, existing methods have the following problems: most achieve fusion under normal lighting conditions, which is not very noticeable under changes in lighting; computation is complex and the effect is limited; texture structure is lost during feature fusion; and unreasonable weighting of infrared and visible light results in a grayish enhanced image. Summary of the Invention

[0004] The purpose of this invention is to provide an infrared-guided low-light image enhancement method that integrates global and local multi-scale fusion, which solves the problems of computational complexity and limited effectiveness in existing image enhancement techniques, as well as the lack of consideration for illumination enhancement and the difficulty in achieving noise reduction.

[0005] The technical solution adopted in this invention is an infrared-guided low-light image enhancement method that fuses global and local multi-scale methods, specifically including the following steps:

[0006] S1: Construct an image enhancement network;

[0007] S2: Construct a dataset and train the image enhancement network;

[0008] S3: Acquire low-light and near-infrared images of the same scene, input them into the trained image augmentation network, and obtain the augmented image.

[0009] The invention is further characterized by:

[0010] The image enhancement network includes two 3×3 convolutional layers, each connected to an encoder. Each encoder comprises four sequentially connected modules, each containing 2, 2, 4, and 8 multi-level residual attention feature extraction modules, respectively. Global and local texture attention fusion modules are connected between corresponding modules of the two encoders. One encoder is also connected to an intermediate layer, which includes six multi-level residual attention feature extraction modules. The intermediate layer is connected to a decoder, which comprises four sequentially connected modules, each containing 2, 2, 2, and 2 multi-level residual attention feature extraction modules, respectively. The global and local texture attention fusion module between the first corresponding modules of the two decoders is connected to the fourth module of the decoder; the global and local texture attention fusion module between the second corresponding modules of the two decoders is connected to the third module of the decoder; the global and local texture attention fusion module between the third corresponding modules of the two decoders is connected to the second module of the decoder; and the global and local texture attention fusion module between the fourth corresponding modules of the two decoders is connected to the first module of the decoder. The decoder is also connected to a 3×3 convolutional layer.

[0011] The multi-level residual attention feature extraction module consists of two sequentially connected modules. The first module comprises a convolutional module consisting of a normalization layer, a 1×1 convolutional layer, and a 3×3 deconvolutional layer, a gating module, a residual channel attention module (RCAB), and a 1×1 convolutional layer, all connected sequentially. The second module comprises a normalization layer, a 1×1 convolutional layer, a gating module, and a 1×1 convolutional layer, all connected sequentially. Skip connections are introduced between the two modules. The residual channel attention module (RCAB) consists of a 1×1 convolutional layer, a Prelude activation function, a 1×1 convolutional layer, and a channel attention module (CA), all connected sequentially.

[0012] The global and local texture attention fusion module consists of two depthwise separable convolutional layers, a local feature attention module, a global feature attention module, and a sigmoid activation function. The depthwise separable convolutional layers consist of a 3×3 kernel grouped convolutional layer with C channels connected sequentially, and a 1×1 convolutional layer. The local feature attention module consists of two convolutional blocks connected by a ReLU activation function. The two convolutional blocks are each composed of a 1×1 convolutional layer and a BN layer. The global feature attention module consists of two 1×1 convolutional layers and three 3×3 3D separable convolutional layers connected together.

[0013] The dataset is constructed in S2 as follows:

[0014] Image pairs of normal lighting and daytime near-infrared images from the FMSVD dataset were selected and set to a uniform size. A pseudo-nighttime near-infrared image was generated from the daytime near-infrared image using a reconstruction method, and used as the input near-infrared image. A pseudo-nighttime low-light image was generated by adding Gaussian noise to the normal image and reducing the pixel brightness, and used as the input low-light image. The image pairs were divided into training, validation, and test sets in an 8:1:1 ratio. Simultaneously, images from the Thirdparty dataset and real nighttime scene images without reference images from the FMSVD dataset were used for testing.

[0015] When training an image augmentation network, the loss function includes a reconstruction loss function. L rec Multi-scale structural similarity loss function L ssim With color loss function L color ;

[0016] Reconstruction loss function L rec The formula is as follows:

[0017]

[0018] In the formula, N is the number of samples. To enhance the image, This is a normal lighting image. This indicates absolute value calculation;

[0019] Multi-scale structural similarity loss function L ssim The formula is as follows:

[0020]

[0021] In the formula, M Represents dimensional parameters, wj Representative at the j Weights at the level scale l j 、c j 、s j The image represents the computationally enhanced image compared to the normally lit image. j Brightness similarity, contrast similarity, and structural similarity at the level of scale;

[0022] Color loss function L color The formula is as follows:

[0023]

[0024] In the formula, It calculates the cosine similarity between the enhanced image and the normal illumination image in the R, G, and B channels. The calculation formula is as follows:

[0025]

[0026] In the formula, sqrt() This means taking the square root. a、b This represents the element for which cosine similarity calculation will be performed. K express a、b The number of elements, k Indicates the first k Each element.

[0027] S3 specifically includes the following steps:

[0028] S3.1: Acquire low-light images and near-infrared images in the same scene. Input the low-light image into a 3×3 convolutional layer connected to the encoder for preliminary feature extraction to obtain shallow low-light features. Input the near-infrared image into another 3×3 convolutional layer for preliminary feature extraction to obtain shallow near-infrared features.

[0029] S3.2: The low-light shallow features and near-infrared shallow features are respectively input into the corresponding encoders. The low-light shallow features and near-infrared shallow features pass through the four modules in the corresponding encoders in sequence. Through three downsampling operations, four low-light deep structure features and near-infrared deep structure features at different scales are obtained respectively.

[0030] S3.3: The low-light deep structure features obtained after the last downsampling are downsampled again and input into the intermediate layer, preserving the original visible light information;

[0031] S3.4: Input the low-light deep structure features and near-infrared deep structure features at the four different scales into the corresponding global and local texture attention fusion modules for fusion to obtain the dual-modal fusion features at the four different scales;

[0032] S3.5: Input the dual-modal fusion features at four different scales into the corresponding modules in the decoder, add them to the original visible light information retained in the intermediate layer, and obtain the enhanced features through upsampling;

[0033] S3.6: Input the enhanced features into a 3×3 convolutional layer to obtain an enhanced feature map. Use residual connections to add the enhanced feature map to the original low-light image to obtain the enhanced image.

[0034] Low-light shallow features and near-infrared shallow features are sequentially passed through four modules in the corresponding encoder. These four modules, connected sequentially, include 2, 2, 4, and 8 multi-level residual attention feature extraction modules, respectively. The processing procedure of the multi-level residual attention feature extraction modules for shallow features is as follows:

[0035] S3.2.1: The input shallow feature x is processed through the first normalization layer, the first 1×1 convolutional layer, and the 3×3 deconvolutional layer to obtain the local feature x1, as shown in the following formula:

[0036]

[0037] In the formula, This represents a 1×1 convolution operation. This represents a 3×3 deconvolution operation. This represents the linear normalization operation;

[0038] S3.2.2: The H×W×C feature is decomposed into two H×W×C / 2 features by the first gating module, and then multiplied pixel by pixel to obtain a feature of size H×W×C / 2. The residual channel attention module RCAB focuses on the feature of interest. After the second 1×1 convolutional layer, it is added to the original input shallow feature to obtain the structural feature y, as shown in the following formula:

[0039]

[0040] In the formula, This indicates that the gating module operation divides a feature of size H×W×C into two features X and Y of size H×W×C / 2, and performs a pixel-wise dot product between the two features, as shown in the following formula:

[0041]

[0042] In the formula, Indicates pixel-wise dot product; The residual channel attention module operation is described in detail below:

[0043]

[0044]

[0045] In the formula, x2 represents the feature obtained through the first gating module; x3 represents the feature obtained through the RCAB module.

[0046] S3.2.3: After passing the structural feature y through the second normalization layer, the third 1×1 convolutional layer, the second gating module, and the fourth 1×1 convolutional layer, it is added to the structural feature y to obtain the deep structural feature z, as shown in the following formula:

[0047] .

[0048] The corresponding low-light deep structure features and near-infrared deep structure features at different scales are respectively input into the global and local texture attention fusion modules for fusion to obtain dual-modal fusion features. The specific process is as follows:

[0049] S3.4.1: Combine the low-light deep structure features at the current scale with the near-infrared deep structure features F vis F nir The two modalities of the images are respectively input into a 1×1 deep separable convolutional layer to reduce the modal differences between the two images, and the output features are added together to obtain the texture fusion feature X. a The formula is as follows:

[0050]

[0051] In the formula, This indicates that the operation has been performed through a depthwise separable convolutional layer; This represents a convolution operation performed using a 3×3 kernel and grouping C channels into a single convolution.

[0052] S3.4.2: Texture Blending Feature X a The input is fed into the local feature attention module. The feature Xa passes through one convolutional block (ConvBlock) and one ReLU activation function, then through another convolutional block, outputting the local attention texture fusion feature Xa. L The announcement is as follows:

[0053]

[0054] In the formula, This indicates that the operation has been performed using a ConvBlock convolution block. This represents the activation function operation; Indicates BN layer operations;

[0055] S3.4.3: Texture Blending Feature X a The input is fed into the global feature attention module. The feature Xa is processed by a 3×3 depthwise convolution (dconv) to generate three tensors Q, K, and V. After passing through the global self-attention mechanism of Q, K, and V, it is passed through a 1×1 convolutional layer to obtain the global attention texture fusion feature Xa. g ;

[0056] In the global self-attention mechanism, Q... H×W×C Size remodeled to Q HW×C , will K H×W×C Size remodeled to K C×HW The self-attention map A is calculated through the interaction between tensors Q and K. C×C Self-attention map A C×CWith tensor V HW×C The interaction yields the input features (out). HW×C Tensor remodeled into out H×W×C The specific formula is as follows:

[0057]

[0058] In the formula, This represents a self-attention operation; This represents the operation of a 3×3 3D separable convolutional layer; It is a learnable scaling parameter used to control the size of the product of matrices K and Q;

[0059] S3.4.4: The global and local texture attention branch features are added together to obtain the fused feature X. lg Global and local texture attention weights W are generated using the Sigmoid activation function, and texture attention features F are applied to low-light and near-infrared images. vis With F nir Weights are assigned to obtain features based on the fusion of global and local texture attention. The formula is as follows:

[0060]

[0061] In the formula, These are learnable parameters that can be applied to the global and local texture fusion attention feature weight allocation for two different modalities.

[0062] The beneficial effects of this invention are:

[0063] This invention proposes an infrared-guided low-light image enhancement method that integrates global and local multi-scale fusion. It fully utilizes the low noise, high contrast, and complementary texture of near-infrared images in low-light scenes to enhance the image by adjusting illumination while simultaneously removing noise. The network structure consists of four blocks for both the encoder and decoder. The encoder's four blocks are composed of 2, 2, 4, and 8 Residual Attention Features (RCAMs), respectively, while the decoder's four blocks are composed of 2, 2, 2, and 2 RCAMs, respectively. Internally, the network structure comprises a multi-level residual attention feature extraction module (RCAM) and a global and local texture attention fusion module (GLAFM). The RCAM module fully extracts structural and illumination features, while the GLAFM module fuses global and local texture attention features from both modalities and then passes them on. Experimental results demonstrate that this method effectively and rationally utilizes the texture and signal-to-noise ratio information of infrared images to assist in low-light image enhancement, and solves the problems of insufficient illumination and significant noise in the low-light enhancement field. Attached Figure Description

[0064] Figure 1This is a schematic diagram of the overall process of the infrared-guided low-light image enhancement method that integrates global and local multi-scale fusion according to the present invention.

[0065] Figure 2 This is a schematic diagram of the structure of the multi-level residual attention feature extraction module (RCAM) in this invention;

[0066] Figure 3 This is a schematic diagram of the residual attention module in this invention;

[0067] Figure 4 This is a schematic diagram of the structure of the Global and Local Texture Attention Fusion (GLAFM) module in this invention;

[0068] Figure 5 This is a comparison chart of the results of this invention with other low-light image enhancement methods in the test set of the constructed dataset;

[0069] Figure 6 This is a comparison chart of the results of this invention with other low-light image enhancement methods on the nighttime dataset of FMSVD. Detailed Implementation

[0070] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0071] The infrared-guided low-light image enhancement method of the present invention, which integrates global and local multi-scale fusion, specifically includes the following steps:

[0072] S1: Construct an image enhancement network;

[0073] like Figure 1 As shown, the image enhancement network includes two 3×3 convolutional layers, each connected to an encoder. Each encoder comprises four sequentially connected modules, each containing 2, 2, 4, and 8 multi-level residual attention feature extraction modules, respectively. Global and local texture attention fusion (GLAFM) modules are connected between corresponding modules of the two encoders. One encoder is also connected to an intermediate layer, which contains six multi-level residual attention feature extraction modules. The intermediate layer is connected to a decoder, which comprises four sequentially connected modules, each containing 2, 2, 2, and 2 multi-level residual attention feature extraction modules, respectively. The GLAFM module between the first corresponding modules of the two decoders is connected to the fourth module of the decoder; the GLAFM module between the second corresponding modules of the two decoders is connected to the third module of the decoder; the GLAFM module between the third corresponding modules of the two decoders is connected to the second module of the decoder; and the GLAFM module between the fourth corresponding modules of the two decoders is connected to the first module of the decoder. The decoder is also connected to a 3×3 convolutional layer.

[0074] Multi-level residual attention feature extraction module (RCAM), such as Figure 2 As shown, it includes two sequentially connected modules. The first module consists of a convolutional module composed of a normalization layer, a 1×1 convolutional layer, and a 3×3 deconvolutional layer, a gating module, a residual channel attention module (RCAB), and a 1×1 convolutional layer, all connected in sequence. The second module consists of a normalization layer, a 1×1 convolutional layer, a gating module, and another 1×1 convolutional layer, all connected in sequence. A skip connection is introduced between the two modules. The residual channel attention module (RCAB) is as follows... Figure 3 As shown, it consists of a 1×1 convolutional layer, a Prelude activation function, a 1×1 convolutional layer, and a channel attention CA connected in sequence;

[0075] Global and local texture attention fusion module GLAFM, such as Figure 4 As shown, it comprises two depthwise separable convolutional layers, a local feature attention module, a global feature attention module, and a sigmoid activation function. The depthwise separable convolutional layers include a 3×3 kernel-sized grouped convolutional layer with C channels connected sequentially, and a 1×1 convolutional layer. The local feature attention module includes two convolutional blocks connected by a ReLU activation function, and each convolutional block is composed of a 1×1 convolutional layer and a BN layer. The global feature attention module is composed of two 1×1 convolutional layers and three 3×3 3D separable convolutional layers connected together.

[0076] S2: Construct a dataset and train the image enhancement network;

[0077] The specific process of constructing the dataset is as follows:

[0078] Image pairs of normal lighting and daytime near-infrared images from the FMSVD dataset were selected and set to a uniform size. A pseudo-nighttime near-infrared image was generated from the daytime near-infrared image using a reconstruction method, and used as the input near-infrared image. A pseudo-nighttime low-light image was generated by adding Gaussian noise to the normal image and reducing the pixel brightness, and used as the input low-light image. The image pairs were divided into training, validation, and test sets in an 8:1:1 ratio. Simultaneously, images from the Thirdparty dataset and real nighttime scene images without reference images from the FMSVD dataset were used for testing.

[0079] When training an image augmentation network, the loss function includes a reconstruction loss function. L rec Multi-scale structural similarity loss function L ssim With color loss function L color ;

[0080] Reconstruction loss function L rec The formula is as follows:

[0081]

[0082] In the formula, N is the number of samples. To enhance the image, This is a normal lighting image. This indicates absolute value calculation;

[0083] Multi-scale structural similarity loss function L ssim The formula is as follows:

[0084]

[0085] In the formula, M Represents dimensional parameters, wj Representative at the j Weights at the level scale l j 、c j 、s j The image represents the computationally enhanced image compared to the normally lit image. j Brightness similarity, contrast similarity, and structural similarity at the level of scale;

[0086] Color loss function L color The formula is as follows:

[0087]

[0088] In the formula, It calculates the cosine similarity between the enhanced image and the normal illumination image in the R, G, and B channels. The calculation formula is as follows:

[0089]

[0090] In the formula, sqrt() This means taking the square root. a、b This represents the element for which cosine similarity calculation will be performed. K express a、b The number of elements, k Indicates the first k Each element.

[0091] S3: Acquire low-light and near-infrared images of the same scene, input them into the trained image augmentation network, and obtain the augmented image. For example... Figure 1 As shown, the specific steps include the following:

[0092] S3.1: Acquire low-light images and near-infrared images in the same scene. Input the low-light image into a 3×3 convolutional layer connected to the encoder for preliminary feature extraction to obtain shallow low-light features. Input the near-infrared image into another 3×3 convolutional layer for preliminary feature extraction to obtain shallow near-infrared features.

[0093] S3.2: Low-light shallow features and near-infrared shallow features are input into their respective encoders. These features pass through four modules in the encoder sequentially, and through three downsampling operations, four different scales of low-light deep structural features and near-infrared deep structural features are obtained. Specifically:

[0094] The original input image H*W*C=128*128*3 becomes 128*128*32 after passing through a 3×3 convolutional layer. After passing through the first module of the encoder (which does not change the image size), the scale remains 128*128*32. Then, after the first D downsampling (changing the image size) and the second module of the encoder, the output scale is 64*64*64. After the second D downsampling and the third module of the encoder, the output scale is 32*32*128. After the third D downsampling and the fourth module of the encoder, the output scale is 16*16*256. The four different scale features are used as two inputs to the fusion part. The low-light image features are then downsampled a fourth time to a scale of 8*8*512.

[0095] Low-light shallow features and near-infrared shallow features are sequentially passed through four modules in the corresponding encoder. These four modules, connected sequentially, include 2, 2, 4, and 8 multi-level residual attention feature extraction modules, respectively. The processing procedure of the multi-level residual attention feature extraction modules for shallow features is as follows:

[0096] S3.2.1: The input shallow feature x is processed through the first normalization layer, the first 1×1 convolutional layer, and the 3×3 deconvolutional layer to obtain the local feature x1, as shown in the following formula:

[0097]

[0098] In the formula, This represents a 1×1 convolution operation. This represents a 3×3 deconvolution operation. This represents the linear normalization operation;

[0099] S3.2.2: The H×W×C feature is decomposed into two H×W×C / 2 features by the first gating module, and then multiplied pixel by pixel to obtain a feature of size H×W×C / 2. The residual channel attention module RCAB focuses on the feature of interest. After the second 1×1 convolutional layer, it is added to the original input shallow feature to obtain the structural feature y, as shown in the following formula:

[0100]

[0101] In the formula, This indicates that the gating module operation divides a feature of size H×W×C into two features X and Y of size H×W×C / 2, and performs a pixel-wise dot product between the two features, as shown in the following formula:

[0102]

[0103] In the formula, Indicates pixel-wise dot product; The residual channel attention module operation is described in detail below:

[0104]

[0105]

[0106] In the formula, x2 represents the feature obtained through the first gating module; x3 represents the feature obtained through the RCAB module.

[0107] S3.2.3: After passing the structural feature y through the second normalization layer, the third 1×1 convolutional layer, the second gating module, and the fourth 1×1 convolutional layer, it is added to the structural feature y to obtain the deep structural feature z, as shown in the following formula:

[0108] .

[0109] S3.3: The low-light deep structure features (scale 16*16*256) obtained after the third downsampling are downsampled again (scale 8*8*512) and input into the intermediate layer to retain the original visible light information;

[0110] S3.4: Input the low-light deep structure features and near-infrared deep structure features at the four different scales into the corresponding global and local texture attention fusion modules for fusion to obtain the dual-modal fusion features at the four different scales;

[0111] S3.4.1: Combine the low-light deep structure features at the current scale with the near-infrared deep structure features F vis F nirThe two modalities of the images are respectively input into a 1×1 deep separable convolutional layer to reduce the modal differences between the two images, and the output features are added together to obtain the texture fusion feature X. a The formula is as follows:

[0112]

[0113] In the formula, This indicates that the operation has been performed through a depthwise separable convolutional layer; This represents a convolution operation performed using a 3×3 kernel and grouping C channels into a single convolution.

[0114] S3.4.2: Texture Blending Feature X a The input is fed into the local feature attention module. The feature Xa passes through one convolutional block (ConvBlock) and one ReLU activation function, then through another convolutional block, outputting the local attention texture fusion feature Xa. L The announcement is as follows:

[0115]

[0116] In the formula, This indicates that the operation has been performed using a ConvBlock convolution block. This represents the activation function operation; Indicates BN layer operations;

[0117] S3.4.3: Texture Blending Feature X a The input is fed into the global feature attention module. The feature Xa is processed by a 3×3 depthwise convolution (dconv) to generate three tensors Q, K, and V. After passing through the global self-attention mechanism of Q, K, and V, it is passed through a 1×1 convolutional layer to obtain the global attention texture fusion feature Xa. g ;

[0118] In the global self-attention mechanism, Q... H×W×C Size remodeled to Q HW×C , will K H×W×C Size remodeled to K C×HW The self-attention map A is calculated through the interaction between tensors Q and K. C×C Self-attention map A C×C With tensor V HW×C The interaction yields the input features (out). HW×C Tensor remodeled into out H×W×C The specific formula is as follows:

[0119]

[0120] In the formula, This represents a self-attention operation; This represents the operation of a 3×3 3D separable convolutional layer; It is a learnable scaling parameter used to control the size of the product of matrices K and Q;

[0121] S3.4.4: The global and local texture attention branch features are added together to obtain the fused feature X. lg Global and local texture attention weights W are generated using the Sigmoid activation function, and texture attention features F are applied to low-light and near-infrared images. vis With F nir Weights are assigned to obtain features based on the fusion of global and local texture attention. The formula is as follows:

[0122]

[0123] In the formula, These are learnable parameters that can be applied to the global and local texture fusion attention feature weight allocation for two different modalities.

[0124] In summary, the illumination and texture information of the two modal images are fully extracted. Before entering the decoder, the features of the two modal images are pre-fused by the GLAFM module. Then, the features are obtained by global and local texture attention mechanisms and added together. The weight W obtained by the Sigmoid activation function is used as the texture feature weight of the visible light image, and 1-W is used as the texture feature weight of the infrared image. The dual-modal fusion feature is obtained by weighted fusion method.

[0125] Specifically, in low-light scenes, near-infrared images exhibit strong anti-interference capabilities, remaining unaffected by illumination and yielding more texture information. In the feature fusion module, infrared features are used as auxiliary information, while visible light image features are used as the primary information. The two modalities are reduced through deep separable convolutions to obtain complementary information, which is then fused. The fused feature is then processed through a global attention branch focusing on global texture and illumination information, and a local attention branch focusing on infrared-guided local detail denoising. The global and local texture features are then fused and added together. Attention weights are assigned using a sigmoid activation function, assigning weights to near-infrared images as auxiliary structural information, which are then added to the visible light fusion global and local texture attention features to obtain the final global and local texture attention fused feature.

[0126] S3.5: The dual-modal fusion features at four different scales are input into the corresponding modules in the decoder, and added to the original visible light information retained in the intermediate layer. Enhanced features are then obtained through upsampling. The specific process is as follows:

[0127] The first upsampling restores the scale to 16*16*256. The low-light deep structure features are added to the original visible light information after passing through the intermediate layer and then to the dual-modal features fused after the third downsampling. The input to the first module of the decoder is then upsampled for the second time to restore 32*32*128, which is added to the dual-modal features fused after the second downsampling. After passing through the second module of the decoder, the third upsampling restores 64*64*64, which is added to the dual-modal features fused after the first downsampling. After passing through the third module of the decoder, the fourth upsampling restores 128*128*32, which is added to the fused dual-modal features output from the first encoder module to obtain 128*128*32.

[0128] S3.6: Input the enhanced features into a 3×3 convolutional layer to obtain an enhanced feature map (scale restored to 128*128*3). Use residual connections to add the enhanced feature map to the original low-light image to obtain the enhanced image.

[0129] In the image enhancement network constructed in this invention, firstly, a simple gating module and a residual attention module are combined in the dual-modal encoder to improve the enhancement network's ability to extract features from two modalities of images, extracting deep features of the two types of images respectively, and making full use of the high contrast and low noise features of near-infrared images under low light conditions; secondly, the two types of image features are fused by global and local texture attention and backpropagated and added to the visible light image features to enhance the structural information in the original visible light image, and better restore the original scale of the image in the multi-scale upsampling of the decoder; finally, according to the residual strategy, the original visible light image is added to the enhanced feature map obtained by the encoder and decoder to obtain an enhanced image with normal illumination and noise removal.

[0130] Example 1

[0131] This embodiment provides an infrared-guided low-light image enhancement method that integrates global and local multi-scale fusion, specifically including the following steps:

[0132] S1: Construct an image enhancement network;

[0133] S2: Construct a dataset and train the image enhancement network;

[0134] S3: Acquire low-light and near-infrared images of the same scene, input them into the trained image augmentation network, and obtain the augmented image.

[0135] Example 2

[0136] Based on Example 1, the image enhancement network includes two 3×3 convolutional layers, each connected to an encoder. Each encoder comprises four sequentially connected modules, each including 2, 2, 4, and 8 multi-level residual attention feature extraction modules, respectively. Global and local texture attention fusion modules are connected between corresponding modules of the two encoders. One encoder is also connected to an intermediate layer, which includes six multi-level residual attention feature extraction modules. The intermediate layer is connected to a decoder, which comprises four sequentially connected modules, each including 2, 2, 2, and 2 multi-level residual attention feature extraction modules, respectively. The global and local texture attention fusion module between the first corresponding modules of the two decoders is connected to the fourth module of the decoder; the global and local texture attention fusion module between the second corresponding modules of the two decoders is connected to the third module of the decoder; the global and local texture attention fusion module between the third corresponding modules of the two decoders is connected to the second module of the decoder; and the global and local texture attention fusion module between the fourth corresponding modules of the two decoders is connected to the first module of the decoder. The decoder is also connected to a 3×3 convolutional layer.

[0137] The multi-level residual attention feature extraction module consists of two sequentially connected modules. The first module comprises a convolutional module consisting of a normalization layer, a 1×1 convolutional layer, and a 3×3 deconvolutional layer, a gating module, a residual channel attention module (RCAB), and a 1×1 convolutional layer, all connected sequentially. The second module comprises a normalization layer, a 1×1 convolutional layer, a gating module, and a 1×1 convolutional layer, all connected sequentially. Skip connections are introduced between the two modules. The residual channel attention module (RCAB) consists of a 1×1 convolutional layer, a Prelude activation function, a 1×1 convolutional layer, and a channel attention module (CA), all connected sequentially.

[0138] The global and local texture attention fusion module consists of two depthwise separable convolutional layers, a local feature attention module, a global feature attention module, and a sigmoid activation function. The depthwise separable convolutional layers consist of a 3×3 kernel grouped convolutional layer with C channels connected sequentially, and a 1×1 convolutional layer. The local feature attention module consists of two convolutional blocks connected by a ReLU activation function. The two convolutional blocks are each composed of a 1×1 convolutional layer and a BN layer. The global feature attention module consists of two 1×1 convolutional layers and three 3×3 3D separable convolutional layers connected together.

[0139] The specific process of constructing the dataset is as follows: 1000 image pairs of normal lighting and daytime near-infrared images from 26 daytime scenes in the FMSVD dataset were selected and set to a uniform size of 320×240; pseudo-nighttime near-infrared images were generated from the daytime near-infrared images using a reconstruction method and used as input near-infrared images; normal images were noise-added with Gaussian noise to reduce the pixel brightness and generate pseudo-nighttime low-light images, used as input low-light images; the image pairs were divided into training, validation, and test sets in an 8:1:1 ratio; simultaneously, images from 41 scenes in the Thirdparty dataset and 33 real nighttime scene images without reference images from the FMSVD dataset were used for testing to confirm the model's enhanced generalization ability in real nighttime low-light scenes.

[0140] When training an image augmentation network, the loss function includes a reconstruction loss function. L rec Multi-scale structural similarity loss function L ssim With color loss function L color ;

[0141] Reconstruction loss function L rec The formula is as follows:

[0142]

[0143] In the formula, N is the number of samples. To enhance the image, This is a normal lighting image. This indicates absolute value calculation;

[0144] Multi-scale structural similarity loss function L ssim The formula is as follows:

[0145]

[0146] In the formula, M Represents dimensional parameters, wj Representative at the j Weights at the level scale l j 、c j 、s j The image represents the computationally enhanced image compared to the normally lit image. j Brightness similarity, contrast similarity, and structural similarity at the level of scale;

[0147] Color loss function L colorThe formula is as follows:

[0148]

[0149] In the formula, It calculates the cosine similarity between the enhanced image and the normal illumination image in the R, G, and B channels. The calculation formula is as follows:

[0150]

[0151] In the formula, sqrt() This means taking the square root. a、b This represents the element for which cosine similarity calculation will be performed. K express a、b The number of elements, k Indicates the first k Each element.

[0152] Example 3

[0153] Building upon Example 2, further steps are taken to acquire low-light and near-infrared images of the same scene, inputting them into the trained image enhancement network to obtain the enhanced image. Specifically, the steps include the following:

[0154] S3.1: Acquire low-light images and near-infrared images in the same scene. Input the low-light image into a 3×3 convolutional layer connected to the encoder for preliminary feature extraction to obtain shallow low-light features. Input the near-infrared image into another 3×3 convolutional layer for preliminary feature extraction to obtain shallow near-infrared features.

[0155] S3.2: The low-light shallow features and near-infrared shallow features are respectively input into the corresponding encoders. The low-light shallow features and near-infrared shallow features pass through the four modules in the corresponding encoders in sequence. Through three downsampling operations, four low-light deep structure features and near-infrared deep structure features at different scales are obtained respectively.

[0156] Low-light shallow features and near-infrared shallow features are sequentially passed through four modules in the corresponding encoder. These four modules, connected sequentially, include 2, 2, 4, and 8 multi-level residual attention feature extraction modules, respectively. The processing procedure of the multi-level residual attention feature extraction modules for shallow features is as follows:

[0157] S3.2.1: The input shallow feature x is processed through the first normalization layer, the first 1×1 convolutional layer, and the 3×3 deconvolutional layer to obtain the local feature x1, as shown in the following formula:

[0158]

[0159] In the formula, This represents a 1×1 convolution operation. This represents a 3×3 deconvolution operation. This represents the linear normalization operation;

[0160] S3.2.2: The H×W×C feature is decomposed into two H×W×C / 2 features by the first gating module, and then multiplied pixel by pixel to obtain a feature of size H×W×C / 2. The residual channel attention module RCAB focuses on the feature of interest. After the second 1×1 convolutional layer, it is added to the original input shallow feature to obtain the structural feature y, as shown in the following formula:

[0161]

[0162] In the formula, This indicates that the gating module operation divides a feature of size H×W×C into two features X and Y of size H×W×C / 2, and performs a pixel-wise dot product between the two features, as shown in the following formula:

[0163]

[0164] In the formula, Indicates pixel-wise dot product; The residual channel attention module operation is described in detail below:

[0165]

[0166]

[0167] In the formula, x2 represents the feature obtained through the first gating module; x3 represents the feature obtained through the RCAB module.

[0168] S3.2.3: After passing the structural feature y through the second normalization layer, the third 1×1 convolutional layer, the second gating module, and the fourth 1×1 convolutional layer, it is added to the structural feature y to obtain the deep structural feature z, as shown in the following formula:

[0169] .

[0170] S3.3: The low-light deep structure features obtained after the last downsampling are downsampled again and input into the intermediate layer, preserving the original visible light information;

[0171] S3.4: Input the low-light deep structure features and near-infrared deep structure features at the four different scales into the corresponding global and local texture attention fusion modules for fusion to obtain the dual-modal fusion features at the four different scales;

[0172] The specific process is as follows:

[0173] S3.4.1: Combine the low-light deep structure features at the current scale with the near-infrared deep structure features Fvis F nir The two modalities of the images are respectively input into a 1×1 deep separable convolutional layer to reduce the modal differences between the two images, and the output features are added together to obtain the texture fusion feature X. a The formula is as follows:

[0174]

[0175] In the formula, This indicates that the operation has been performed through a depthwise separable convolutional layer; This represents a convolution operation performed using a 3×3 kernel and grouping C channels into a single convolution.

[0176] S3.4.2: Texture Blending Feature X a The input is fed into the local feature attention module. The feature Xa passes through one convolutional block (ConvBlock) and one ReLU activation function, then through another convolutional block, outputting the local attention texture fusion feature Xa. L The announcement is as follows:

[0177]

[0178] In the formula, This indicates that the operation has been performed using a ConvBlock convolution block. This represents the activation function operation; Indicates BN layer operations;

[0179] S3.4.3: Texture Blending Feature X a The input is fed into the global feature attention module. The feature Xa is processed by a 3×3 depthwise convolution (dconv) to generate three tensors Q, K, and V. After passing through the global self-attention mechanism of Q, K, and V, it is passed through a 1×1 convolutional layer to obtain the global attention texture fusion feature Xa. g ;

[0180] In the global self-attention mechanism, Q... H×W×C Size remodeled to Q HW×C , will K H×W×C Size remodeled to K C×HW The self-attention map A is calculated through the interaction between tensors Q and K. C×C Self-attention map A C×C With tensor V HW×C The interaction yields the input features (out). HW×C Tensor remodeled into out H×W×C The specific formula is as follows:

[0181]

[0182] In the formula, This represents a self-attention operation; This represents the operation of a 3×3 3D separable convolutional layer; It is a learnable scaling parameter used to control the size of the product of matrices K and Q;

[0183] S3.4.4: The global and local texture attention branch features are added together to obtain the fused feature X. lg Global and local texture attention weights W are generated using the Sigmoid activation function, and texture attention features F are applied to low-light and near-infrared images. vis With F nir Weights are assigned to obtain features based on the fusion of global and local texture attention. The formula is as follows:

[0184]

[0185] In the formula, These are learnable parameters that can be applied to the global and local texture fusion attention feature weight allocation for two different modalities.

[0186] S3.5: Input the dual-modal fusion features at four different scales into the corresponding modules in the decoder, add them to the original visible light information retained in the intermediate layer, and obtain the enhanced features through upsampling;

[0187] S3.6: Input the enhanced features into a 3×3 convolutional layer to obtain an enhanced feature map. Use residual connections to add the enhanced feature map to the original low-light image to obtain the enhanced image.

[0188] Simulation Experiment

[0189] Quantitative evaluations were performed on the test set of the dataset constructed in Example 2, the Thirdparty dataset, and the nighttime dataset of FMSVD. The method of this invention was compared with six cutting-edge low-light image enhancement methods and two infrared-guided fusion denoising methods. The low-light image enhancement methods included Zero-DCE, EnlightenGAN, LLFlow, SNR, LLFormer, and RetinexFormer; the infrared-guided fusion denoising methods included DVN, MN, and MNL. Peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and image similarity measure (LPIPS) were selected to evaluate the quality of the enhanced images. The comparison results on the test set of the constructed dataset and the Thirdparty dataset are shown in Table 1.

[0190] Table 1. Comparison results between existing image enhancement methods and the Thirdparty dataset.

[0191]

[0192] In the table, ↑ indicates that a larger value indicates better image quality, and ↓ indicates that a smaller value indicates better image quality. Bold text represents the best and second-best results. As shown in Table 1, in the test set of the constructed dataset, the method of this invention outperforms existing state-of-the-art enhancement and fusion denoising methods in terms of quantitative metrics PSNR, SSIM, and LPIPS (as shown in red), with the RetinexFormer method from ECCV2024 ranking second. In the Thirdparty dataset, the SSIM and LPIPS metrics reach best and second-best levels. This demonstrates that the method proposed in this invention can reduce noise while improving brightness, restoring image texture structure, and approaching the quality of the reference image.

[0193] On the test set of the constructed dataset and the real nighttime data of FMSVD, such as Figure 5 and Figure 6 As shown, the effectiveness of the method of the present invention in brightness enhancement, noise reduction, and image detail restoration is verified. Among existing single-image low-light image enhancement methods, Zero-DCE can effectively improve image brightness characteristics, but it also amplifies noise. EnlightenGAN and LLFormer methods generate a large number of pixel blocks when enhancing image brightness and removing noise, resulting in discontinuous image structure and failing to meet the visual perception of high-quality images; SNR, LLFlow, Retinexformer, and DVN methods, in... Figure 5 In the middle, the edges of the leaves are blurred and details are not fully restored; in Figure 6 In the middle, the texture of the letters on the road sign was not fully restored; Figure 6 In comparison with the input image, the first two methods can achieve noise reduction but do not improve global brightness, which affects the visual experience.

[0194] In comparison, the infrared-guided low-light image enhancement method that integrates global and local multi-scale methods can better preserve the structural information of the restored image under normal illumination, while also achieving image denoising.

[0195] As can be seen from the above, the residual attention mechanism can fully acquire the structural features of the two modalities without preprocessing. The global and local texture attention fusion method can make full use of the clear structure of near-infrared images in low-light scenes to guide the enhancement of texture structural features. Finally, the learned illumination texture difference is added to the original low-light image by the residual link to obtain an enhanced image with improved brightness and noise removal.

Claims

1. A method for infrared-guided low-light image enhancement by fusing global and local multi-scale approaches, characterized in that, Specifically, the steps include the following: S1: Construct an image enhancement network; The image enhancement network includes two 3×3 convolutional layers, each connected to an encoder. Each encoder comprises four sequentially connected modules, each including 2, 2, 4, and 8 multi-level residual attention feature extraction modules. Global and local texture attention fusion modules are connected between corresponding modules of the two encoders. One encoder is also connected to an intermediate layer, which includes six multi-level residual attention feature extraction modules. The intermediate layer is connected to a decoder, which includes four sequentially connected modules, each comprising a multi-level residual attention feature extraction module (2, 2, 2, 2 respectively). A global and local texture attention fusion module between the first corresponding modules of the two decoders is connected to the fourth module of the decoder; a global and local texture attention fusion module between the second corresponding modules of the two decoders is connected to the third module of the decoder; a global and local texture attention fusion module between the third corresponding modules of the two decoders is connected to the second module of the decoder; and a global and local texture attention fusion module between the fourth corresponding modules of the two decoders is connected to the first module of the decoder. The decoder is also connected to a 3×3 convolutional layer. The multi-level residual attention feature extraction module comprises two sequentially connected modules. The first module consists of a convolutional module composed of a normalization layer, a 1×1 convolutional layer, and a 3×3 deconvolutional layer, a gating module, a residual channel attention module RCAB, and a 1×1 convolutional layer, all connected sequentially. The second module consists of a normalization layer, a 1×1 convolutional layer, a gating module, and a 1×1 convolutional layer, all connected sequentially. Skip connections are introduced between the two modules. The residual channel attention module RCAB consists of a 1×1 convolutional layer, a PreLU activation function, a 1×1 convolutional layer, and a channel attention CA, all connected sequentially. The global and local texture attention fusion module comprises two depthwise separable convolutional layers, a local feature attention module, a global feature attention module, and a sigmoid activation function. The depthwise separable convolutional layers consist of a 3×3 kernel-sized grouped convolutional layer (grouped by C channels) and a 1×1 convolutional layer, connected sequentially. The local feature attention module comprises two convolutional blocks connected by a ReLU activation function, each block consisting of a 1×1 convolutional layer and a batch normalization (BN) layer. The global feature attention module consists of two 1×1 convolutional layers and three 3×3 3D separable convolutional layers connected together. S2: Construct a dataset and train the image enhancement network; S3: Acquire low-light and near-infrared images of the same scene, input them into the trained image enhancement network, and obtain the enhanced image; S3.1: Acquire low-light images and near-infrared images in the same scene. Input the low-light image into a 3×3 convolutional layer connected to the encoder for preliminary feature extraction to obtain shallow low-light features. Input the near-infrared image into another 3×3 convolutional layer for preliminary feature extraction to obtain shallow near-infrared features. S3.2: The low-light shallow features and near-infrared shallow features are respectively input into the corresponding encoders. The low-light shallow features and near-infrared shallow features pass through the four modules in the corresponding encoders in sequence. Through three downsampling operations, four low-light deep structure features and near-infrared deep structure features at different scales are obtained respectively. S3.3: The low-light deep structure features obtained after the last downsampling are downsampled again and input into the intermediate layer, preserving the original visible light information; S3.4: Input the low-light deep structure features and near-infrared deep structure features at the four different scales into the corresponding global and local texture attention fusion modules for fusion to obtain the dual-modal fusion features at the four different scales; The corresponding low-light deep structure features and near-infrared deep structure features at different scales are respectively input into the global and local texture attention fusion modules for fusion to obtain dual-modal fusion features. The specific process is as follows: S3.4.1: Combine the low-light deep structure features at the current scale with the near-infrared deep structure features F vis F nir The two modalities of the images are respectively input into a 1×1 deep separable convolutional layer to reduce the modal differences between the two images, and the output features are added together to obtain the texture fusion feature X. a The formula is as follows: In the formula, This indicates that the operation has been performed through a depthwise separable convolutional layer; This represents a convolution operation performed using a 3×3 kernel and grouping C channels into a single convolution. S3.4.2: Texture Blending Feature X a The input is fed into the local feature attention module. The feature Xa passes through one convolutional block (ConvBlock) and one ReLU activation function, then through another convolutional block, outputting the local attention texture fusion feature Xa. L The announcement is as follows: In the formula, This indicates that the operation has been performed using a ConvBlock convolution block. This represents the activation function operation; Indicates BN layer operations; S3.4.3: Texture Blending Feature X a The input is fed into the global feature attention module. The feature Xa is processed by a 3×3 depthwise convolution (dconv) to generate three tensors Q, K, and V. After passing through the global self-attention mechanism of Q, K, and V, it is passed through a 1×1 convolutional layer to obtain the global attention texture fusion feature Xa. g ; In the global self-attention mechanism, Q... H×W×C Size remodeled to Q HW×C , will K H×W×C Size remodeled to K C×HW The self-attention map A is calculated through the interaction between tensors Q and K. C×C Self-attention map A C×C With tensor V HW×C The interaction yields the input features (out). HW×C Tensor remodeled into out H×W×C The specific formula is as follows: In the formula, This represents a self-attention operation; This represents the operation of a 3×3 3D separable convolutional layer; It is a learnable scaling parameter used to control the size of the product of matrices K and Q; S3.4.4: The global and local texture attention branch features are added together to obtain the fused feature X. lg Global and local texture attention weights W are generated using the Sigmoid activation function, and texture attention features F are applied to low-light and near-infrared images. vis With F nir Weights are assigned to obtain features based on the fusion of global and local texture attention. The formula is as follows: In the formula, These are learnable parameters that can be applied to the global and local texture fusion attention feature weight allocation for two different modalities. S3.5: Input the dual-modal fusion features at four different scales into the corresponding modules in the decoder, add them to the original visible light information retained in the intermediate layer, and obtain the enhanced features through upsampling; S3.6: Input the enhanced features into a 3×3 convolutional layer to obtain an enhanced feature map. Use residual connections to add the enhanced feature map to the original low-light image to obtain the enhanced image.

2. The infrared-guided low-light image enhancement method based on global and local multi-scale fusion according to claim 1, characterized in that, The dataset is constructed in S2 as follows: Image pairs of normal lighting and daytime near-infrared images from the FMSVD dataset are selected and set to a uniform size. The daytime near-infrared images are used to generate pseudo-nighttime near-infrared images, which are then used as input near-infrared images. Normal images are noise-added with Gaussian noise to reduce pixel brightness and generate pseudo-nighttime low-light images, which are then used as input low-light images. Image pairs are divided into training, validation, and test sets in an 8:1:1 ratio. Simultaneously, images from the Thirdparty dataset and real nighttime scene images without reference images from the FMSVD dataset are used for testing.

3. The infrared-guided low-light image enhancement method based on global and local multi-scale fusion according to claim 1, characterized in that, When training an image augmentation network, the loss function includes a reconstruction loss function. L rec Multi-scale structural similarity loss function L ssim With color loss function L color ; The reconstruction loss function L rec The formula is as follows: In the formula, N is the number of samples. To enhance the image, This is a normal lighting image. This indicates absolute value calculation; The multi-scale structural similarity loss function L ssim The formula is as follows: In the formula, M Represents dimensional parameters, wj Representative at the j Weights at the level scale l j 、c j 、s j The image represents the computationally enhanced image compared to the normally lit image. j Brightness similarity, contrast similarity, and structural similarity at the level of scale; The color loss function L color The formula is as follows: In the formula, It calculates the cosine similarity between the enhanced image and the normal illumination image in the R, G, and B channels. The calculation formula is as follows: In the formula, sqrt() This means taking the square root. a、b This represents the element for which cosine similarity calculation will be performed. K express a、b The number of elements, k Indicates the first k Each element.

4. The infrared-guided low-light image enhancement method based on global and local multi-scale fusion according to claim 1, characterized in that, The low-light shallow features and near-infrared shallow features are sequentially passed through four modules in the corresponding encoder. These four modules, connected sequentially, include 2, 2, 4, and 8 multi-level residual attention feature extraction modules, respectively. The processing procedure of the multi-level residual attention feature extraction modules for the shallow features is as follows: S3.2.1: The input shallow feature x is processed through the first normalization layer, the first 1×1 convolutional layer, and the 3×3 deconvolutional layer to obtain the local feature x1, as shown in the following formula: In the formula, This represents a 1×1 convolution operation. This represents a 3×3 deconvolution operation. This represents the linear normalization operation; S3.2.2: The H×W×C feature is decomposed into two H×W×C / 2 features by the first gating module, and then multiplied pixel by pixel to obtain a feature of size H×W×C / 2. The residual channel attention module RCAB focuses on the feature of interest. After the second 1×1 convolutional layer, it is added to the original input shallow feature to obtain the structural feature y, as shown in the following formula: In the formula, This indicates that the gating module operation divides a feature of size H×W×C into two features X and Y of size H×W×C / 2, and performs a pixel-wise dot product between the two features, as shown in the following formula: In the formula, Indicates pixel-wise dot product; The residual channel attention module operation is described in detail below: In the formula, x2 represents the feature obtained through the first gating module; x3 represents the feature obtained through the RCAB module. S3.2.3: After passing the structural feature y through the second normalization layer, the third 1×1 convolutional layer, the second gating module, and the fourth 1×1 convolutional layer, it is added to the structural feature y to obtain the deep structural feature z, as shown in the following formula: 。

Images