Global and local multi-scale fused infrared guide low-light image enhancement method

Abstract

The invention discloses a global and local multi-scale fused infrared guide low-light image enhancement method, which belongs to the technical field of computer vision, and specifically comprises the following steps: constructing an image enhancement network; constructing a data set, and training the image enhancement network; and acquiring a low-illumination image and a near-infrared image in the same scene, and inputting the low-illumination image and the near-infrared image into the trained image enhancement network to obtain an enhanced image. According to the global and local multi-scale fused infrared guide low-light image enhancement method, the problems that in the prior art, calculation is complex, the effect is single, illumination lifting is not considered, and denoising is difficult to achieve are solved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to an infrared-guided low-light image enhancement method based on global and local multi-scale fusion. Background Art

[0002] Low-light scenes are mainly scenes with insufficient external light sources, such as nighttime, backlighting, and extremely low-light scenes without light sources. The main reasons for low-light images include the insufficient light and dim brightness of the target object during shooting, the position of the light source, color changes, and the exposure ratio setting of the acquisition equipment, which lead to the collected images having low brightness, high noise, and lack of detail information. The purpose of image enhancement is to improve the visual quality of low-light images and enhance the observer's perception ability, so as to better analyze the image content. It has important research significance and application value in security monitoring, military applications, or medical imaging.

[0003] Low-light image enhancement in real scenes requires image brightness enhancement, denoising and restoration of image texture details. Most of the existing low-light image enhancement methods are based on a single low-light image for enhancement. The brightness enhancement will amplify the hidden noise, and the denoising process will remove the texture structure and noise together, resulting in the loss of key information. In real night low-light scenes, near-infrared light has strong anti-interference ability, and the collected night near-infrared images have the characteristics of high contrast and low noise. In the past two years, infrared and visible light image fusion and enhancement methods have been gradually proposed, such as the DVN method. Infrared image-guided low-light image enhancement methods make full use of the high contrast and low noise characteristics of near-infrared images to solve the image denoising problem, but the existing methods have the following problems: most of them realize fusion under normal light, which is not very obvious to light changes; the calculation is complex and the effect is single, the texture structure is lost during feature fusion, and the unreasonable distribution of infrared and visible light weight ratios causes the enhanced image to be gray. Summary of the invention

[0004] The purpose of the present invention is to provide a global and local multi-scale fusion infrared-guided low-light image enhancement method, which solves the problems of complex calculation and single effect in existing image enhancement technology, as well as the failure to consider illumination improvement and difficulty in denoising.

[0005] The technical solution adopted by the present invention is a global and local multi-scale fusion infrared-guided low-light image enhancement method, which specifically includes the following steps: S1: Build image enhancement network; S2: Build a data set and train the image enhancement network; S3: Obtain a low-light image and a near-infrared image of the same scene, input them into the trained image enhancement network, and obtain an enhanced image.

[0006] The present invention is also characterized in that: The image enhancement network includes two 3×3 convolutional layers, the two 3×3 convolutional layers are respectively connected to an encoder, the encoder includes 4 modules connected in sequence, and the 4 modules connected in sequence include 2, 2, 4, and 8 multi-level residual attention feature extraction modules respectively; the global and local texture attention fusion modules are respectively connected between the corresponding modules of the two encoders; one of the encoders is also connected to an intermediate layer, and the intermediate layer includes 6 multi-level residual attention feature extraction modules; the intermediate layer is connected to a decoder, and the decoder includes 4 modules connected in sequence, and the 4 modules connected in sequence include 2, 2, 2, and 2 multi-level residual attention feature extraction modules respectively; the global and local texture attention fusion module between the first corresponding modules of the two decoders is connected to the fourth module of the decoder, the global and local texture attention fusion module between the second corresponding modules of the two decoders is connected to the third module of the decoder, the global and local texture attention fusion module between the third corresponding modules of the two decoders is connected to the second module of the decoder, and the global and local texture attention fusion module between the fourth corresponding modules of the two decoders is connected to the first module of the decoder; the decoder is also connected to a 3×3 convolutional layer.

[0007] The multi-level residual attention feature extraction module includes two modules connected in sequence. The first module includes a convolution module consisting of a normalization layer, a 1×1 convolution layer and a 3×3 deconvolution layer connected in sequence, a gating module, a residual channel attention module RCAB, and a 1×1 convolution layer; the second module includes a normalization layer, a 1×1 convolution layer, a gating module, and a 1×1 convolution layer connected in sequence; a jump connection is introduced between the two modules; the residual channel attention module RCAB includes a 1×1 convolution layer, a Prelu activation function, a 1×1 convolution layer and a channel attention CA connected in sequence; The global and local texture attention fusion module consists of two depthwise separable convolutional layers, a local feature attention module, a global feature attention module and a Sigmoid activation function; the depthwise separable convolutional layer includes a grouped convolution with a convolution kernel size of 3×3, grouped into channels C, and a 1×1 convolutional layer connected in sequence; the local feature attention module includes two convolutional blocks, a ReLU activation function is connected between the two convolutional blocks, and the two convolutional blocks are respectively composed of a 1×1 convolutional layer and a BN layer; the global feature attention module is composed of two 1×1 convolutional layers and three 3×3 3D separable convolutional layers.

[0008] Construct a data set in S2, specifically: Image pairs of normal lighting and daytime near-infrared images in daytime scenes in the FMSVD dataset are selected and set to a uniform size; the daytime near-infrared images are reconstructed to generate pseudo nighttime near-infrared images as the input near-infrared images; the normal images are denoised with Gaussian noise and the image pixel brightness is reduced to generate pseudo nighttime low-light images as the input low-light images; the image pairs are divided into training set, validation set and test set in a ratio of 8:1:1; at the same time, the scenes in the Thirdparty dataset and the real night scene images without reference images in the FMSVD dataset are used for testing.

[0009] When the image enhancement network is trained, the loss function includes the reconstruction loss function L rec , multi-scale structural similarity loss function L ssim With the color loss function L color ; Reconstruction loss function L rec , the formula is as follows:

[0010] In the formula, N is the number of samples, To enhance the image, is a normal lighting image, Indicates absolute value calculation; Multi-scale structural similarity loss function L ssim , the formula is as follows:

[0011] In the formula, M Represents the size parameter, wj Representative in the j The weight at the level scale, l j 、c j 、s j Represents the difference between the computed enhanced image and the normal illumination image. j Brightness similarity, contrast similarity and structural similarity at the level scale; Color loss function L color , the formula is as follows:

[0012] In the formula, It is to calculate the cosine similarity between the enhanced image and the normal illumination image in the three channels of R, G, and B. The calculation formula is as follows:

[0013] In the formula, sqrt() represents taking the square root, a, b Indicates the element for which cosine similarity calculation is to be performed. K express a, b The number of elements, k Indicates k elements.

[0014] S3 specifically includes the following steps: S3.1: Obtain a low-light image and a near-infrared image of the same scene, input the low-light image into a 3×3 convolutional layer connected to the middle layer of the encoder for preliminary feature extraction, and obtain low-light shallow features; input the near-infrared image into another 3×3 convolutional layer for preliminary feature extraction, and obtain near-infrared shallow features; S3.2: The low-light shallow features and near-infrared shallow features are input into the corresponding encoders respectively. The low-light shallow features and near-infrared shallow features are sequentially passed through the four modules in the corresponding encoders. The low-light deep structure features and near-infrared deep structure features at four different scales are obtained through three downsamplings. S3.3: The low-light deep structure features obtained after the last downsampling are input into the middle layer after downsampling again, retaining the original visible light information; S3.4: Input the corresponding low-light deep structure features and near-infrared deep structure features at four different scales into the corresponding global and local texture attention fusion modules for fusion, and obtain the bimodal fusion features at four different scales; S3.5: The bimodal fusion features at four different scales are input into the corresponding modules in the decoder respectively, and added to the original visible light information retained in the middle layer, and the enhanced features are obtained by upsampling; S3.6: The enhanced features are input into the 3×3 convolutional layer to obtain an enhanced feature map, and the enhanced feature map is added to the original low-light image using a residual connection to obtain an enhanced image.

[0015] The low-light shallow features and near-infrared shallow features pass through the four modules in the corresponding encoder in sequence. The four modules connected in sequence include 2, 2, 4, and 8 multi-level residual attention feature extraction modules respectively. The processing process of the multi-level residual attention feature extraction module for shallow features is as follows: S3.2.1: The input shallow feature x passes through the first normalization layer, the first 1×1 convolution layer and the 3×3 deconvolution layer to obtain the local feature x 1 , the formula is as follows:

[0016] In the formula, represents a 1×1 convolution operation, represents a 3×3 deconvolution operation, represents the linear normalization operation; S3.2.2: The H×W×C feature is decomposed into two H×W×C / 2 features through the first gating module, and a H×W×C / 2 size feature is obtained by pixel-by-pixel multiplication. The input residual channel attention module RCAB focuses on the feature of interest, and the second 1×1 convolutional layer is added to the original input shallow feature to obtain the structural feature y. The formula is as follows:

[0017] In the formula, It represents the operation of the gated module, which divides the feature of size H×W×C into two features X and Y of size H×W×C / 2. The two features are multiplied pixel by pixel. The formula is as follows:

[0018] In the formula, Represents pixel-by-pixel multiplication; Represents the residual channel attention module operation, the specific process is as follows:

[0019]

[0020] In the formula, x 2 represents the features obtained through the first gating module; x 3 Represents the features obtained by the RCAB module; S3.2.3: The structural feature y is added to the structural feature y after passing through the second normalization layer, the third 1×1 convolution layer, the second gating module and the fourth 1×1 convolution layer to obtain the deep structural feature z. The formula is as follows: .

[0021] The corresponding low-light deep structure features and near-infrared deep structure features at different scales are respectively input into the global and local texture attention fusion modules for fusion to obtain the bimodal fusion features. The specific process is as follows: S3.4.1: Combine the low-light deep structure features at the current scale with the near-infrared deep structure features F vis 、F nir , respectively input into the 1×1 deep separable convolution layer to reduce the modal difference of the two modal images, and the output features are added to obtain the texture fusion feature X a , the formula is as follows:

[0022] In the formula, Indicates that the depth-separable convolution layer has been operated; It represents the convolution operation after the convolution kernel size is 3×3 and the number of channels is C. S3.4.2: Texture fusion feature X a Input to the local feature attention module, feature Xa passes through 1 convolution block ConvBlock and 1 ReLU activation function, and then passes through 1 convolution block to output the local attention texture fusion feature X L , the announcement is as follows:

[0023] In the formula, Indicates that it has been operated by the convolution block ConvBlock; Represents the activation function operation; Indicates BN layer operation; S3.4.3: Texture fusion feature X a Input to the global feature attention module, feature Xa is generated by 3×3 deep convolution dconv to generate three tensors Q, K, and V. After the global self-attention mechanism of Q, K, and V, a 1×1 convolution layer is passed to obtain the global attention texture fusion feature X g ; In the global self-attention mechanism, Q H×W×C Reshape to Q HW×C , K H×W×C Reshape to K C×HW , calculate the self-attention map A through the interaction between tensors Q and K C×C , self-attention map A C×C With tensor V HW×C Interact to get the input feature out HW×C , the tensor is reshaped into out H×W×C , the specific formula is as follows:

[0024] In the formula, represents the self-attention operation; Represents a 3×3 3D separable convolutional layer operation; is a learnable scaling parameter that controls the size of the multiplication of matrices K and Q; S3.4.4: Add the global and local texture attention branch features to obtain the fused feature X lg ; Generate global and local texture attention weights W through Sigmoid activation function, and texture attention features F for low-light and near-infrared modal images vis With F nir Perform weight assignment to obtain features based on global and local texture attention fusion , the formula is as follows:

[0025] In the formula, is a learnable parameter applied to the weight allocation of global and local texture fusion attention features of two different modalities.

[0026] The beneficial effects of the present invention are: The infrared-guided low-light image enhancement method with global and local multi-scale fusion proposed in the present invention makes full use of the low noise and high contrast of near-infrared images in low-light scenes, and the texture complementarity with visible light images, so that the enhanced image adjusts the illumination and removes noise at the same time. The encoder and decoder in the network structure are composed of 4 blocks, the 4 blocks of the encoder are composed of 2, 2, 4, and 8 RCAMs respectively, and the 4 blocks of the decoder are composed of 2, 2, 2, and 2 RCAMs respectively. The network structure is composed of a multi-level residual attention feature extraction module RCAM and a global and local texture attention fusion module GLAFM. The RCAM module fully extracts the structural and illumination feature information, and the GLAFM module performs global and local texture attention feature fusion on the two modal information and transmits it backward. Experimental results show that this method can effectively and reasonably utilize the texture and signal-to-noise ratio information of infrared images to assist in the enhancement of low-light images, and solve the problems of insufficient illumination and a large amount of noise in the field of low-light enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the overall process of the infrared-guided low-light image enhancement method of the present invention with global and local multi-scale fusion; Figure 2 Schematic diagram of the structure of a multi-level residual attention feature extraction module (RCAM) in the present invention; Figure 3 Schematic diagram of the structure of the residual attention module in the present invention; Figure 4 is a schematic diagram of the structure of the global and local texture attention fusion module (GLAFM) in the present invention; Figure 5 is a comparison chart of the results of the present invention in the test set of the constructed data set with other low-light image enhancement methods; Figure 6 This is a comparison chart of the results of the present invention and other low-light image enhancement methods in the nighttime data set of FMSVD. DETAILED DESCRIPTION

[0028] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] The global and local multi-scale fusion infrared-guided low-light image enhancement method of the present invention specifically includes the following steps: S1: Build image enhancement network; like Figure 1 As shown in the figure, the image enhancement network includes two 3×3 convolutional layers, the two 3×3 convolutional layers are respectively connected with an encoder, the encoder includes 4 modules connected in sequence, and the 4 modules connected in sequence include 2, 2, 4, and 8 multi-level residual attention feature extraction modules respectively; the global and local texture attention fusion modules GLAFM are respectively connected between the corresponding modules of the two encoders; one of the encoders is also connected to an intermediate layer, and the intermediate layer includes 6 multi-level residual attention feature extraction modules; the intermediate layer is connected to a decoder, and the decoder includes 4 modules connected in sequence, and the 4 modules connected in sequence include 2, 2, 2, and 2 multi-level residual attention feature extraction modules respectively; the global and local texture attention fusion module between the first corresponding modules of the two decoders is connected to the fourth module of the decoder, the global and local texture attention fusion module between the second corresponding modules of the two decoders is connected to the third module of the decoder, the global and local texture attention fusion module between the third corresponding modules of the two decoders is connected to the second module of the decoder, and the global and local texture attention fusion module between the fourth corresponding modules of the two decoders is connected to the first module of the decoder; the decoder is also connected with a 3×3 convolutional layer.

[0030] Multi-level residual attention feature extraction module RCAM, such as Figure 2 As shown in FIG, it includes two modules connected in sequence. The first module includes a convolution module consisting of a normalization layer, a 1×1 convolution layer and a 3×3 deconvolution layer connected in sequence, a gating module, a residual channel attention module RCAB, and a 1×1 convolution layer; the second module includes a normalization layer, a 1×1 convolution layer, a gating module, and a 1×1 convolution layer connected in sequence; a skip connection is introduced between the two modules; the residual channel attention module RCAB, as shown in FIG. Figure 3 As shown, it consists of a 1×1 convolutional layer, a Prelu activation function, a 1×1 convolutional layer and a channel attention CA connected in sequence; The global and local texture attention fusion module GLAFM, such as Figure 4As shown, it includes two depth-wise separable convolutional layers, a local feature attention module, a global feature attention module and a Sigmoid activation function; the depth-wise separable convolutional layer includes a grouped convolution with a convolution kernel size of 3×3, grouped into channels C and a 1×1 convolutional layer connected in sequence; the local feature attention module includes two convolutional blocks, a ReLU activation function is connected between the two convolutional blocks, and the two convolutional blocks are respectively composed of a 1×1 convolutional layer and a BN layer; the global feature attention module is composed of two 1×1 convolutional layers and three 3×3 3D separable convolutional layers.

[0031] S2: Build a data set and train the image enhancement network; Construct a data set. The specific process is as follows: Image pairs of normal lighting and daytime near-infrared images in daytime scenes in the FMSVD dataset are selected and set to a uniform size; the daytime near-infrared images are reconstructed to generate pseudo nighttime near-infrared images as the input near-infrared images; the normal images are denoised with Gaussian noise and the image pixel brightness is reduced to generate pseudo nighttime low-light images as the input low-light images; the image pairs are divided into training set, validation set and test set in a ratio of 8:1:1; at the same time, the scenes in the Thirdparty dataset and the real night scene images without reference images in the FMSVD dataset are used for testing.

[0032] When the image enhancement network is trained, the loss function includes the reconstruction loss function L rec , multi-scale structural similarity loss function L ssim With the color loss function L color ; Reconstruction loss function L rec , the formula is as follows:

[0033] In the formula, N is the number of samples, To enhance the image, is a normal lighting image, Indicates absolute value calculation; Multi-scale structural similarity loss function L ssim , the formula is as follows:

[0034] In the formula, M Represents the size parameter, wj Representative in the j The weight at the level scale,l j 、c j 、s j Represents the difference between the computed enhanced image and the normal illumination image. j Brightness similarity, contrast similarity and structural similarity at the level scale; Color loss function L color , the formula is as follows:

[0035] In the formula, It is to calculate the cosine similarity between the enhanced image and the normal illumination image in the three channels of R, G, and B. The calculation formula is as follows:

[0036] In the formula, sqrt() represents taking the square root, a, b Indicates the element for which cosine similarity calculation is to be performed. K express a, b The number of elements, k Indicates k elements.

[0037] S3: Obtain low-light images and near-infrared images of the same scene, input them into the trained image enhancement network, and obtain enhanced images. Figure 1 As shown, the specific steps include: S3.1: Obtain a low-light image and a near-infrared image of the same scene, input the low-light image into a 3×3 convolutional layer connected to the middle layer of the encoder for preliminary feature extraction, and obtain low-light shallow features; input the near-infrared image into another 3×3 convolutional layer for preliminary feature extraction, and obtain near-infrared shallow features; S3.2: The low-light shallow features and near-infrared shallow features are input into the corresponding encoders respectively. The low-light shallow features and near-infrared shallow features are sequentially passed through the four modules in the corresponding encoders. The low-light deep structure features and near-infrared deep structure features at four different scales are obtained by three downsamplings. Specifically: The original input image H*W*C=128*128*3 becomes 128*128*32 after 3×3 convolutional layers. After the first module of the encoder (this module does not change the image size), the scale is maintained at 128*128*32. After the first D downsampling (changing the image size) and the second module of the encoder, the output scale is 64*64*64. After the second D downsampling and the third module of the encoder, the output scale is 32*32*128. After the third D downsampling and the fourth module of the encoder, the output scale is 16*16*256. The obtained 4 different scale features are used as two inputs of the fusion part. The low-light image features are downsampled for the fourth time and the scale is 8*8*512.

[0038] The low-light shallow features and near-infrared shallow features pass through the four modules in the corresponding encoder in sequence. The four modules connected in sequence include 2, 2, 4, and 8 multi-level residual attention feature extraction modules respectively. The processing process of the multi-level residual attention feature extraction module for shallow features is as follows: S3.2.1: The input shallow feature x passes through the first normalization layer, the first 1×1 convolution layer and the 3×3 deconvolution layer to obtain the local feature x 1 , the formula is as follows:

[0039] In the formula, represents a 1×1 convolution operation, represents a 3×3 deconvolution operation, represents the linear normalization operation; S3.2.2: The H×W×C feature is decomposed into two H×W×C / 2 features through the first gating module, and a H×W×C / 2 size feature is obtained by pixel-by-pixel multiplication. The input residual channel attention module RCAB focuses on the feature of interest, and the second 1×1 convolutional layer is added to the original input shallow feature to obtain the structural feature y. The formula is as follows:

[0040] In the formula, It represents the operation of the gated module, which divides the feature of size H×W×C into two features X and Y of size H×W×C / 2. The two features are multiplied pixel by pixel. The formula is as follows:

[0041] In the formula, Represents pixel-by-pixel multiplication; Represents the residual channel attention module operation, the specific process is as follows:

[0042]

[0043] In the formula, x 2 represents the features obtained through the first gating module; x 3 Represents the features obtained by the RCAB module; S3.2.3: The structural feature y is added to the structural feature y after passing through the second normalization layer, the third 1×1 convolution layer, the second gating module and the fourth 1×1 convolution layer to obtain the deep structural feature z. The formula is as follows: .

[0044] S3.3: The low-light deep structure features (scale 16*16*256) obtained by the third downsampling are input into the middle layer after downsampling again (scale 8*8*512), retaining the original visible light information; S3.4: Input the corresponding low-light deep structure features and near-infrared deep structure features at four different scales into the corresponding global and local texture attention fusion modules for fusion, and obtain the bimodal fusion features at four different scales; S3.4.1: Combine the low-light deep structure features at the current scale with the near-infrared deep structure features F vis 、F nir , respectively input into the 1×1 deep separable convolution layer to reduce the modal difference of the two modal images, and the output features are added to obtain the texture fusion feature X a , the formula is as follows:

[0045] In the formula, Indicates that the depth-separable convolution layer has been operated; It represents the convolution operation after the convolution kernel size is 3×3 and the number of channels is C. S3.4.2: Texture fusion feature X a Input to the local feature attention module, feature Xa passes through 1 convolution block ConvBlock and 1 ReLU activation function, and then passes through 1 convolution block to output the local attention texture fusion feature X L , the announcement is as follows:

[0046] In the formula, Indicates that it has been operated by the convolution block ConvBlock; Represents the activation function operation; Indicates BN layer operation; S3.4.3: Texture fusion feature X aInput to the global feature attention module, feature Xa is generated by 3×3 deep convolution dconv to generate three tensors Q, K, and V. After the global self-attention mechanism of Q, K, and V, a 1×1 convolution layer is passed to obtain the global attention texture fusion feature X g ; In the global self-attention mechanism, Q H×W×C Reshape to Q HW×C , K H×W×C Reshape to K C×HW , calculate the self-attention map A through the interaction between tensors Q and K C×C , self-attention map A C×C With tensor V HW×C Interact to get the input feature out HW×C , the tensor is reshaped into out H×W×C , the specific formula is as follows:

[0047] In the formula, represents the self-attention operation; Represents a 3×3 3D separable convolutional layer operation; is a learnable scaling parameter that controls the size of the multiplication of matrices K and Q; S3.4.4: Add the global and local texture attention branch features to obtain the fused feature X lg ; Generate global and local texture attention weights W through Sigmoid activation function, and texture attention features F for low-light and near-infrared modal images vis With F nir Perform weight assignment to obtain features based on global and local texture attention fusion , the formula is as follows:

[0048] In the formula, is a learnable parameter applied to the weight allocation of global and local texture fusion attention features of two different modalities.

[0049] In summary, the illumination, texture and other information of the two modal images are fully extracted. Before entering the decoder, the features of the two modal images are pre-fused through the GLAFM module, and then the features are added through the global and local texture attention mechanism. The weight W obtained by the Sigmoid activation function is used as the texture feature weight of the visible light image, and 1-W is used as the texture feature weight of the infrared image. The dual-modal fusion feature is obtained through the weighted fusion method.

[0050] Specifically, in low-light scenes, near-infrared images have strong anti-interference capabilities and are not affected by light. In low-light scenes, more texture information can be obtained. In the feature fusion module, infrared features are used as auxiliary information and visible light image features are used as main information. The two modal image features are reduced in modality through deep separable convolution, and complementary information is obtained and added and fused. The fused features are focused on the global texture and illumination information of the image through the global attention branch, and the local attention branch focuses on the infrared-guided image local details for denoising, and the global and local texture features are fused and added. The attention weight is assigned through the Sigmoid activation function, and the near-infrared image is given a weight as auxiliary structural information, and the fused global and local texture attention is added with the visible light to obtain the final global and local texture attention fusion feature.

[0051] S3.5: The bimodal fusion features at four different scales are input into the corresponding modules in the decoder respectively, and added to the original visible light information retained in the middle layer, and the enhanced features are obtained by upsampling; the specific process is as follows: The first upsampling restores the scale to 16*16*256. The low-light deep structure features retain the original visible light information through the intermediate layer and are added to the bimodal features fused after the third downsampling. The input decoder's first module upsamples the second time to restore 32*32*128, which is added to the bimodal features fused after the second downsampling. The second module of the decoder upsamples the third time to restore 64*64*64, which is added to the bimodal features fused after the first downsampling. The third module of the decoder upsamples the fourth time to restore 128*128*32, which is added to the fused bimodal features output by the first encoder module to obtain 128*128*32.

[0052] S3.6: The enhanced features are input into the 3×3 convolutional layer to obtain an enhanced feature map (the scale is restored to 128*128*3), and the enhanced feature map is added to the original low-light image using a residual connection to obtain an enhanced image.

[0053] In the image enhancement network constructed by the present invention, firstly, a simple gating module and a residual attention module are combined in the dual-modal encoder to improve the feature extraction capability of the enhancement network for the two modal images, respectively extract the deep features of the two types of images, and make full use of the high contrast and low noise features of the near-infrared image under low light; secondly, the two types of image features are fused through global and local texture attention and backwardly transferred to be added with the visible light image features, so as to enhance the structural information in the original visible light image, and better restore the original scale of the image in the multi-scale upsampling of the decoder; finally, according to the residual strategy, the original visible light image is added to the enhanced feature map obtained by the codec to obtain an enhanced image with normal illumination and noise removed.

[0054] Example 1 This embodiment provides a global and local multi-scale fusion infrared-guided low-light image enhancement method, which specifically includes the following steps: S1: Build image enhancement network; S2: Build a data set and train the image enhancement network; S3: Obtain a low-light image and a near-infrared image of the same scene, input them into the trained image enhancement network, and obtain an enhanced image.

[0055] Example 2 On the basis of Example 1, the image enhancement network includes two 3×3 convolutional layers, the two 3×3 convolutional layers are respectively connected to an encoder, the encoder includes 4 modules connected in sequence, and the 4 modules connected in sequence include 2, 2, 4, and 8 multi-level residual attention feature extraction modules respectively; global and local texture attention fusion modules are respectively connected between the corresponding modules of the two encoders; one of the encoders is also connected to an intermediate layer, and the intermediate layer includes 6 multi-level residual attention feature extraction modules; the intermediate layer is connected to a decoder, and the decoder includes 4 modules connected in sequence, and the 4 modules connected in sequence include 2, 2, 2, and 2 multi-level residual attention feature extraction modules respectively; the global and local texture attention fusion module between the first corresponding modules of the two decoders is connected to the fourth module of the decoder, the global and local texture attention fusion module between the second corresponding modules of the two decoders is connected to the third module of the decoder, the global and local texture attention fusion module between the third corresponding modules of the two decoders is connected to the second module of the decoder, and the global and local texture attention fusion module between the fourth corresponding modules of the two decoders is connected to the first module of the decoder; the decoder is also connected to a 3×3 convolutional layer.

[0056] The multi-level residual attention feature extraction module includes two modules connected in sequence. The first module includes a convolution module consisting of a normalization layer, a 1×1 convolution layer and a 3×3 deconvolution layer connected in sequence, a gating module, a residual channel attention module RCAB, and a 1×1 convolution layer; the second module includes a normalization layer, a 1×1 convolution layer, a gating module, and a 1×1 convolution layer connected in sequence; a jump connection is introduced between the two modules; the residual channel attention module RCAB includes a 1×1 convolution layer, a Prelu activation function, a 1×1 convolution layer and a channel attention CA connected in sequence; The global and local texture attention fusion module consists of two depthwise separable convolutional layers, a local feature attention module, a global feature attention module and a Sigmoid activation function; the depthwise separable convolutional layer includes a grouped convolution with a convolution kernel size of 3×3, grouped into channels C, and a 1×1 convolutional layer connected in sequence; the local feature attention module includes two convolutional blocks, a ReLU activation function is connected between the two convolutional blocks, and the two convolutional blocks are respectively composed of a 1×1 convolutional layer and a BN layer; the global feature attention module is composed of two 1×1 convolutional layers and three 3×3 3D separable convolutional layers.

[0057] The specific process of constructing the dataset is as follows: 1000 pairs of normal lighting and daytime near-infrared images in 26 daytime scenes in the FMSVD dataset are selected and set to a uniform size of 320×240; pseudo nighttime near-infrared images are generated from daytime near-infrared images through reconstruction methods as input near-infrared images; normal images are denoised with Gaussian noise and the image pixel brightness is reduced to generate pseudo nighttime low-light images as input low-light images; the image pairs are divided into training set, validation set and test set in a ratio of 8:1:1; at the same time, images in 41 scenes of the Thirdparty dataset and 33 real night scene images without reference images in the FMSVD dataset are used for testing to confirm the enhanced generalization ability of the model in real nighttime low-light scenes.

[0058] When the image enhancement network is trained, the loss function includes the reconstruction loss function L rec , multi-scale structural similarity loss function L ssim With the color loss function L color ; Reconstruction loss function L rec , the formula is as follows:

[0059] In the formula, N is the number of samples, To enhance the image, is a normal lighting image, Indicates absolute value calculation; Multi-scale structural similarity loss function L ssim , the formula is as follows:

[0060] In the formula, M Represents the size parameter, wj Representative in the j The weight at the level scale,l j 、c j 、s j Represents the difference between the computed enhanced image and the normal illumination image. j Brightness similarity, contrast similarity and structural similarity at the level scale; Color loss function L color , the formula is as follows:

[0061] In the formula, It is to calculate the cosine similarity between the enhanced image and the normal illumination image in the three channels of R, G, and B. The calculation formula is as follows:

[0062] In the formula, sqrt() represents taking the square root, a, b Indicates the element for which cosine similarity calculation is to be performed. K express a, b The number of elements, k Indicates k elements.

[0063] Example 3 On the basis of Example 2, further, a low-light image and a near-infrared image of the same scene are obtained, and input into a trained image enhancement network to obtain an enhanced image. Specifically, the steps include: S3.1: Obtain a low-light image and a near-infrared image of the same scene, input the low-light image into a 3×3 convolutional layer connected to the middle layer of the encoder for preliminary feature extraction, and obtain low-light shallow features; input the near-infrared image into another 3×3 convolutional layer for preliminary feature extraction, and obtain near-infrared shallow features; S3.2: The low-light shallow features and near-infrared shallow features are input into the corresponding encoders respectively. The low-light shallow features and near-infrared shallow features are sequentially passed through the four modules in the corresponding encoders. The low-light deep structure features and near-infrared deep structure features at four different scales are obtained through three downsamplings. The low-light shallow features and near-infrared shallow features pass through the four modules in the corresponding encoder in sequence. The four modules connected in sequence include 2, 2, 4, and 8 multi-level residual attention feature extraction modules respectively. The processing process of the multi-level residual attention feature extraction module for shallow features is as follows: S3.2.1: The input shallow feature x passes through the first normalization layer, the first 1×1 convolution layer and the 3×3 deconvolution layer to obtain the local feature x 1 , the formula is as follows:

[0064] In the formula, represents a 1×1 convolution operation, represents a 3×3 deconvolution operation, represents the linear normalization operation; S3.2.2: The H×W×C feature is decomposed into two H×W×C / 2 features through the first gating module, and a H×W×C / 2 size feature is obtained by pixel-by-pixel multiplication. The input residual channel attention module RCAB focuses on the feature of interest, and the second 1×1 convolutional layer is added to the original input shallow feature to obtain the structural feature y. The formula is as follows:

[0065] In the formula, It represents the operation of the gated module, which divides the feature of size H×W×C into two features X and Y of size H×W×C / 2. The two features are multiplied pixel by pixel. The formula is as follows:

[0066] In the formula, Represents pixel-by-pixel multiplication; Represents the residual channel attention module operation, the specific process is as follows:

[0067]

[0068] In the formula, x 2 represents the features obtained through the first gating module; x 3 Represents the features obtained by the RCAB module; S3.2.3: The structural feature y is added to the structural feature y after passing through the second normalization layer, the third 1×1 convolution layer, the second gating module and the fourth 1×1 convolution layer to obtain the deep structural feature z. The formula is as follows: .

[0069] S3.3: The low-light deep structure features obtained after the last downsampling are input into the middle layer after downsampling again, retaining the original visible light information; S3.4: Input the corresponding low-light deep structure features and near-infrared deep structure features at four different scales into the corresponding global and local texture attention fusion modules for fusion, and obtain the bimodal fusion features at four different scales; The specific process is as follows: S3.4.1: Combine the low-light deep structure features at the current scale with the near-infrared deep structure features F vis 、Fnir , respectively input into the 1×1 deep separable convolution layer to reduce the modal difference of the two modal images, and the output features are added to obtain the texture fusion feature X a , the formula is as follows:

[0070] In the formula, Indicates that the depth-separable convolution layer has been operated; It represents the convolution operation after the convolution kernel size is 3×3 and the number of channels is C. S3.4.2: Texture fusion feature X a Input to the local feature attention module, feature Xa passes through 1 convolution block ConvBlock and 1 ReLU activation function, and then passes through 1 convolution block to output the local attention texture fusion feature X L , the announcement is as follows:

[0071] In the formula, Indicates that it has been operated by the convolution block ConvBlock; Represents the activation function operation; Indicates BN layer operation; S3.4.3: Texture fusion feature X a Input to the global feature attention module, feature Xa is generated by 3×3 deep convolution dconv to generate three tensors Q, K, and V. After the global self-attention mechanism of Q, K, and V, a 1×1 convolution layer is passed to obtain the global attention texture fusion feature X g ; In the global self-attention mechanism, Q H×W×C Reshape to Q HW×C , K H×W×C Reshape to K C×HW , calculate the self-attention map A through the interaction between tensors Q and K C×C , self-attention map A C×C With tensor V HW×C Interact to get the input feature out HW×C , the tensor is reshaped into out H×W×C , the specific formula is as follows:

[0072] In the formula, represents the self-attention operation; Represents a 3×3 3D separable convolutional layer operation; is a learnable scaling parameter that controls the size of the multiplication of matrices K and Q; S3.4.4: Add the global and local texture attention branch features to obtain the fused feature X lg ; Generate global and local texture attention weights W through Sigmoid activation function, and texture attention features F for low-light and near-infrared modal images vis With F nir Perform weight assignment to obtain features based on global and local texture attention fusion , the formula is as follows:

[0073] In the formula, is a learnable parameter applied to the weight allocation of global and local texture fusion attention features of two different modalities.

[0074] S3.5: The bimodal fusion features at four different scales are input into the corresponding modules in the decoder respectively, and added to the original visible light information retained in the middle layer, and the enhanced features are obtained by upsampling; S3.6: The enhanced features are input into the 3×3 convolutional layer to obtain an enhanced feature map, and the enhanced feature map is added to the original low-light image using a residual connection to obtain an enhanced image.

[0075] Simulation experiment A quantitative evaluation was performed on the test set of the dataset constructed in Example 2, the Thirdparty dataset, and the nighttime dataset of FMSVD. The method of the present invention was compared with six cutting-edge low-light image enhancement methods and two infrared-guided fusion denoising methods. The low-light image enhancement methods include Zero-DCE, EnlightenGAN, LLFlow, SNR, LLFormer, and RetinexFormer, and the infrared-guided fusion denoising methods include DVN, MN, and MNL. Three evaluation indicators, peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and image similarity metric (LPIPS), were selected to evaluate the quality of the enhanced image. The comparison results on the test set of the constructed dataset and the Thirdparty dataset are shown in Table 1.

[0076] Table 1 Comparison results with existing image enhancement methods on the constructed test set and the Thirdparty dataset

[0077] In the table, ↑ means the larger the value, the better the image quality, ↓ means the smaller the value, the better the image quality, where bold fonts represent the optimal and suboptimal results. As shown in Table 1, in the test set of the constructed data set, the quantitative indicators PSNR, SSIM and LPIPS of the method of the present invention are better than the existing latest enhancement and fusion denoising methods as shown in red, and the RetinexFormer method of ECCV2024 is second. In the Thirdparty data set, the SSIM and LPIPS indicators can reach the optimal and suboptimal levels. This shows that the method proposed in the present invention can reduce noise while improving brightness, restore the image texture structure and make it close to the reference image.

[0078] On the test set of the constructed dataset and the real night data of FMSVD, such as Figure 5 and Figure 6 As shown in the figure, it is verified that the method of the present invention is effective in brightness enhancement, denoising and image detail structure restoration. Among the existing single low-light image enhancement methods, Zero-DCE can effectively improve the image brightness characteristics, but it also amplifies the noise. When the EnlightenGAN and LLFormer methods enhance the image brightness and remove noise, a large number of pixel blocks are generated, resulting in an incoherent image structure that does not meet the visual perception of high-quality images; SNR, LLFlow, Retinexformer, and DVN methods are Figure 5 In the , the edges of the leaves are blurred and the details are not fully restored; Figure 6 In the example, the letter structure texture on the road sign is not fully restored; Figure 6 In the results, compared with the input image, the first two methods can achieve denoising but fail to improve the global brightness, which affects the visual experience.

[0079] Relatively speaking, the infrared-guided low-light image enhancement method with global and local multi-scale fusion can better preserve the structural information of the restored image under normal illumination and achieve image denoising at the same time.

[0080] From the above method, it can be seen that the residual attention mechanism can fully obtain the structural features of the two modalities without preprocessing. The global and local texture attention fusion method can make full use of the clear structure of near-infrared images in low-light scenes to guide the enhancement of texture structural features. Finally, the residual link is used to add the learned illumination texture difference to the input original low-light image to obtain an enhanced image with improved brightness and removed noise.

Claims

1. An infrared-guided low-light image enhancement method based on global and local multi-scale fusion, characterized in that: The specific steps include: S1: Build image enhancement network; S2: Build a data set and train the image enhancement network; S3: Obtain a low-light image and a near-infrared image of the same scene, input them into the trained image enhancement network, and obtain an enhanced image.

2. The infrared-guided low-light image enhancement method based on global and local multi-scale fusion according to claim 1, characterized in that: The image enhancement network includes two 3×3 convolutional layers, the two 3×3 convolutional layers are respectively connected to encoders, the encoders include 4 modules connected in sequence, the 4 modules connected in sequence include 2, 2, 4, and 8 multi-level residual attention feature extraction modules respectively; the corresponding modules of the two encoders are respectively connected to global and local texture attention fusion modules; one of the encoders is also connected to an intermediate layer, and the intermediate layer includes 6 multi-level residual attention feature extraction modules; The middle layer is connected to a decoder, which includes 4 modules connected in sequence, and the 4 modules connected in sequence include 2, 2, 2, and 2 multi-level residual attention feature extraction modules respectively; the global and local texture attention fusion module between the first corresponding modules of the two decoders is connected to the fourth module of the decoder, the global and local texture attention fusion module between the second corresponding modules of the two decoders is connected to the third module of the decoder, the global and local texture attention fusion module between the third corresponding modules of the two decoders is connected to the second module of the decoder, and the global and local texture attention fusion module between the fourth corresponding modules of the two decoders is connected to the first module of the decoder; the decoder is also connected to a 3×3 convolutional layer.

3. The infrared-guided low-light image enhancement method based on global and local multi-scale fusion according to claim 2, characterized in that: The multi-level residual attention feature extraction module includes two modules connected in sequence, the first module includes a convolution module consisting of a normalization layer, a 1×1 convolution layer and a 3×3 deconvolution layer connected in sequence, a gating module, a residual channel attention module RCAB, and a 1×1 convolution layer; the second module includes a normalization layer, a 1×1 convolution layer, a gating module, and a 1×1 convolution layer connected in sequence; a jump connection is introduced between the two modules; the residual channel attention module RCAB includes a 1×1 convolution layer, a Prelu activation function, a 1×1 convolution layer and a channel attention CA connected in sequence; The global and local texture attention fusion module includes two depth-separable convolutional layers, a local feature attention module, a global feature attention module and a Sigmoid activation function; the depth-separable convolutional layer includes a grouped convolution with a convolution kernel size of 3×3, grouped into a channel number C and a 1×1 convolutional layer connected in sequence; the local feature attention module includes two convolutional blocks, a ReLU activation function is connected between the two convolutional blocks, and the two convolutional blocks are respectively composed of a 1×1 convolutional layer and a BN layer; the global feature attention module is composed of two 1×1 convolutional layers and three 3×3 3D separable convolutional layers.

4. The infrared-guided low-light image enhancement method based on global and local multi-scale fusion according to claim 1, characterized in that: Construct a data set in S2, specifically: Select the image pairs of normal illumination and daytime near-infrared images in the daytime scene in the FMSVD dataset and set them to a uniform size; generate pseudo nighttime near-infrared images from daytime near-infrared images through reconstruction methods as input near-infrared images; The normal image is denoised with Gaussian noise and the image pixel brightness is reduced to generate a pseudo night low-light image as the input low-light image; the image pairs are divided into training set, validation set and test set in a ratio of 8:1:1; at the same time, the scene images of the Thirdparty dataset and the real night scene images without reference images in the FMSVD dataset are used for testing.

5. The infrared-guided low-light image enhancement method based on global and local multi-scale fusion according to claim 1, characterized in that: When the image enhancement network is trained, the loss function includes the reconstruction loss function L rec , multi-scale structural similarity loss function L ssim With the color loss function L color ; The reconstruction loss function L rec , the formula is as follows: In the formula, N is the number of samples, To enhance the image, is a normal lighting image, Indicates absolute value calculation; The multi-scale structural similarity loss function L ssim , the formula is as follows: In the formula, M Represents the size parameter, wj Representative in the j The weight at the level scale, l j 、c j 、s j Represents the difference between the computed enhanced image and the normal illumination image. j Brightness similarity, contrast similarity and structural similarity at the level scale; The color loss function L color , the formula is as follows: In the formula, It is to calculate the cosine similarity between the enhanced image and the normal illumination image in the three channels of R, G, and B. The calculation formula is as follows: In the formula, sqrt() represents taking the square root, a, b Indicates the element for which cosine similarity calculation is to be performed. K express a, b The number of elements, k Indicates k elements.

6. The infrared-guided low-light image enhancement method based on global and local multi-scale fusion according to claim 1, characterized in that: S3 specifically includes the following steps: S3.1: Obtain a low-light image and a near-infrared image of the same scene, input the low-light image into a 3×3 convolutional layer connected to the middle layer of the encoder for preliminary feature extraction, and obtain low-light shallow features; input the near-infrared image into another 3×3 convolutional layer for preliminary feature extraction, and obtain near-infrared shallow features; S3.2: The low-light shallow features and near-infrared shallow features are input into the corresponding encoders respectively. The low-light shallow features and near-infrared shallow features are sequentially passed through the four modules in the corresponding encoders. The low-light deep structure features and near-infrared deep structure features at four different scales are obtained through three downsamplings. S3.3: The low-light deep structure features obtained after the last downsampling are input into the middle layer after downsampling again, retaining the original visible light information; S3.4: Input the corresponding low-light deep structure features and near-infrared deep structure features at four different scales into the corresponding global and local texture attention fusion modules for fusion, and obtain the bimodal fusion features at four different scales; S3.5: The bimodal fusion features at four different scales are input into the corresponding modules in the decoder respectively, and added to the original visible light information retained in the middle layer, and the enhanced features are obtained by upsampling; S3.6: The enhanced features are input into the 3×3 convolutional layer to obtain an enhanced feature map, and the enhanced feature map is added to the original low-light image using a residual connection to obtain an enhanced image.

7. The infrared-guided low-light image enhancement method based on global and local multi-scale fusion according to claim 6, characterized in that: The low-light shallow features and near-infrared shallow features are sequentially passed through the four modules in the corresponding encoder. The four modules connected in sequence include 2, 2, 4, and 8 multi-level residual attention feature extraction modules, respectively. The processing process of the multi-level residual attention feature extraction module for shallow features is as follows: S3.2.1: After the input shallow feature x passes through the first normalization layer, the first 1×1 convolution layer and the 3×3 deconvolution layer, the local feature x1 is obtained. The formula is as follows: In the formula, represents a 1×1 convolution operation, represents a 3×3 deconvolution operation, represents the linear normalization operation; S3.2.2: The H×W×C feature is decomposed into two H×W×C / 2 features through the first gating module, and a H×W×C / 2 size feature is obtained by pixel-by-pixel multiplication. The input residual channel attention module RCAB focuses on the feature of interest, and the second 1×1 convolutional layer is added to the original input shallow feature to obtain the structural feature y. The formula is as follows: In the formula, It represents the operation of the gated module, which divides the feature of size H×W×C into two features X and Y of size H×W×C / 2. The two features are multiplied pixel by pixel. The formula is as follows: In the formula, Represents pixel-by-pixel multiplication; Represents the residual channel attention module operation, the specific process is as follows: In the formula, x2 represents the feature obtained by the first gating module; x3 represents the feature obtained by the RCAB module; S3.2.3: The structural feature y is added to the structural feature y after passing through the second normalization layer, the third 1×1 convolution layer, the second gating module and the fourth 1×1 convolution layer to obtain the deep structural feature z. The formula is as follows: 。 8. The infrared-guided low-light image enhancement method based on global and local multi-scale fusion according to claim 7, characterized in that: The corresponding low-light deep structure features and near-infrared deep structure features at different scales are respectively input into the global and local texture attention fusion modules for fusion to obtain the bimodal fusion features. The specific process is as follows: S3.4.1: Combine the low-light deep structure features at the current scale with the near-infrared deep structure features F vis 、F nir , respectively input into the 1×1 deep separable convolution layer to reduce the modal difference between the two modal images, and the output features are added to obtain the texture fusion feature X a , the formula is as follows: In the formula, Indicates that the depth-separable convolution layer has been operated; It represents the convolution operation after the convolution kernel size is 3×3 and the number of channels is C. S3.4.2: Texture fusion feature X a Input to the local feature attention module, feature Xa passes through 1 convolution block ConvBlock and 1 ReLU activation function, and then passes through 1 convolution block to output the local attention texture fusion feature X L , the announcement is as follows: In the formula, Indicates that it has been operated by the convolution block ConvBlock; Represents the activation function operation; Indicates BN layer operation; S3.4.3: Texture fusion feature X a Input to the global feature attention module, feature Xa is generated by 3×3 deep convolution dconv to generate three tensors Q, K, and V. After the global self-attention mechanism of Q, K, and V, a 1×1 convolution layer is passed to obtain the global attention texture fusion feature X g ; In the global self-attention mechanism, Q H×W×C Reshape to Q HW×C , K H×W×C Reshape to K C×HW , calculate the self-attention map A through the interaction between tensors Q and K C×C , self-attention map A C×C With tensor V HW×C Interact to get the input feature out HW×C , the tensor is reshaped into out H×W×C , the specific formula is as follows: In the formula, represents the self-attention operation; Represents a 3×3 3D separable convolutional layer operation; is a learnable scaling parameter that controls the size of the multiplication of matrices K and Q; S3.4.4: Add the global and local texture attention branch features to obtain the fused feature X lg ; Generate global and local texture attention weights W through Sigmoid activation function, and texture attention features F for low-light and near-infrared modal images vis With F nir Perform weight assignment to obtain features based on global and local texture attention fusion , the formula is as follows: In the formula, is a learnable parameter applied to the weight allocation of global and local texture fusion attention features of two different modalities.

Images