Progressive low-light image enhancement method based on hybrid attention

Abstract

The application discloses a progressive low-light image enhancement method based on mixed attention, and specifically comprises the following processes: an initial feature mapping is obtained through convolution of an input low-light image, then the feature is preliminarily integrated by a reconstruction mixed attention module after a first stage subnetwork; then a second stage subnetwork and a third stage subnetwork fully utilize multi-scale features in a progressive fusion manner to realize global information expression of the features; finally, pixel loss, multi-scale perception loss and color loss are introduced to further reconstruct a higher-quality low-light image recovery result. The application solves the distortion problem after image reconstruction from coarse to fine by fully utilizing local and non-local information of multi-scale features.

Description

Technical Field

[0001] This invention belongs to the field of low-light image enhancement technology, and relates to a progressive low-light image enhancement method based on hybrid attention. Background Technology

[0002] Today, image analysis and processing provide support and decision-making basis in various fields, including medical imaging, security monitoring, and autonomous driving. However, under low-light conditions, due to insufficient light or environmental limitations, the acquired images often suffer from multiple degradation problems such as low brightness, low contrast, high noise, and color distortion, making it difficult to discern details in the image and thus affecting the accuracy and reliability of subsequent image analysis and processing. Several methods are commonly used to obtain high-quality images under low-light conditions. One method is to use external lighting equipment, but this may result in unnatural lighting effects and interfere with the subject or other objects in the scene. Another common method is to extend the exposure time or use a high ISO setting; however, this often easily leads to image blurring and the introduction of high noise, thereby reducing image quality. Therefore, low-light image enhancement (LLIE) using computer algorithms has become a research topic with high application value.

[0003] Previous methods have attempted to integrate multi-scale features to expand the model's receptive field and capture richer contextual information. For example, some studies have employed multi-scale residual blocks or Laplacian pyramid multi-scale structures to capture rich contextual information. However, these methods directly extract multi-scale information from severely degraded images, which can easily introduce unnecessary noise and degradation into the multi-scale features. Furthermore, they fail to consider that features at different scales may contain many similar or overlapping patterns, such as structures and textures, which can lead to redundancy when fusing these features. This redundancy reduces the model's sensitivity to local details, resulting in significant distortion of local details. Summary of the Invention

[0004] The purpose of this invention is to provide a progressive low-light image enhancement method based on hybrid attention, which solves the distortion problem that occurs after image reconstruction in a coarse-to-fine manner by making full use of local and non-local information of multi-scale features.

[0005] The technical solution adopted in this invention is a progressive low-light image enhancement method based on hybrid attention, which specifically includes the following process: the input low-light image is convolved to obtain an initial feature map, and then passes through a first-stage sub-network, where the features are initially integrated by the reconstruction hybrid attention module; subsequently, the second-stage and third-stage sub-networks fully utilize multi-scale features in a progressive fusion manner to achieve global information representation of the features; finally, pixel loss, multi-scale perceptual loss, and color loss are introduced to reconstruct a higher-quality low-light image restoration result.

[0006] The invention is further characterized by:

[0007] Specifically, the steps include the following:

[0008] Step 1, acquire image I containing low light. l and normal lighting image I h The image dataset is divided into a training set and a test set, where normal lighting images I... h For label images;

[0009] Step 2: Construct a progressive multi-scale feature fusion network model for low-light image I. l The progressive multi-scale feature fusion network model is reconstructed, comprising a first-stage subnetwork, a second-stage subnetwork, and a third-stage subnetwork.

[0010] Step 3: Use the training set divided in Step 1 to train the progressive multi-scale feature fusion network model constructed in Step 2, and output the trained progressive multi-scale feature fusion network model.

[0011] Step 4: Input the test set divided in Step 1 into the progressive multi-scale feature fusion network model trained in Step 3, and output the reconstructed normal optical restoration image.

[0012] In step 1, the training set images are randomly cropped and resized to 256×256 pixels to augment the training dataset. Simultaneously, the normal lighting images I in the augmented training dataset are also included. h By scaling the images by 1 / 2 and 1 / 4 respectively, label images of sizes 128×128 and 64×64 pixels were obtained. and

[0013] The specific process of step 2 is as follows:

[0014] Step 2.1, from the low-light image I l The initial feature map extracted from the image is fed into the first-stage sub-network for preliminary feature pre-enhancement, thereby outputting the first-stage sub-network enhanced feature map F1;

[0015] The enhanced feature map F1 output in steps 2.2 and 2.1 is used as the input of the second-stage sub-network. The second-stage sub-network extracts features of two scales and completes adaptive multi-scale feature fusion based on the enhanced feature map F1 of the first-stage sub-network, thus obtaining the enhanced feature map F2 of the second-stage sub-network.

[0016] The enhanced feature map F2 output from steps 2.3 and 2.2 is used as the input to the third-stage sub-network. The third-stage sub-network extracts three different scale features based on the enhanced feature map F2 of the second-stage sub-network and completes the adaptive multi-scale feature progressive fusion pairwise, thus obtaining the enhanced feature map F3 of the third-stage sub-network.

[0017] Step 2.4: Compress the third-stage sub-network enhancement feature map F3 into the RGB channel to obtain the reconstructed and enhanced image.

[0018] The specific process of step 2.1 is as follows:

[0019] Step 2.1.1, Input a low-light image After feature pre-extraction via convolution, an initial feature map is obtained. The image is then passed to the first-stage sub-network, resulting in an initial feature map with 64 channels.

[0020]

[0021] in Represents the data field;

[0022] Step 2.1.2, map the initial features. The input is processed by a hybrid attention module to filter and integrate features, resulting in a reconstructed hybrid feature map f at the same scale as the input. RM64 ;

[0023] Step 2.1.3, Feature Mapping The reconstructed hybrid feature map f is obtained after the reconstruction hybrid attention module. RM64 Then, preliminary feature processing is completed through residual connections, resulting in the first-stage sub-network enhanced feature map F1. The processing procedure is expressed as follows:

[0024]

[0025] The specific process of step 2.2 is as follows:

[0026] Step 2.2.1: Perform a 1 / 2 downsampling operation on the first-stage enhanced feature map F1, and obtain the feature map of another scale through convolution. Represented as:

[0027]

[0028] Where downsampling(·) represents a 1 / 2 downsampling operation. This indicates a scale feature where the number of channels in the second-stage subnetwork is 128;

[0029] Step 2.2.2, combine the feature maps F1 and The features are enhanced separately by a reconstruction hybrid attention module and then fused. The fused features are then further enhanced by a reconstruction hybrid attention module to filter and integrate the fused features, thus obtaining the second-stage multi-scale fused features.

[0030]

[0031] Where RMA(·) represents the feature map after reconstructing the hybrid attention module, and ↑ represents a 2× upsampling operation. It is a multi-scale fusion feature with 64 channels after the second-stage sub-network enhancement;

[0032] Step 2.2.3: Fuse features across multiple scales The second-stage sub-network enhances the feature mapping F2 through residual connections. The second-stage process is represented as follows:

[0033]

[0034] The specific process of step 2.3 is as follows:

[0035] Step 2.3.1: Perform continuous 1 / 2 downsampling operations on the second-stage enhanced feature map F2, and obtain two feature maps of different scales through convolution. and Represented as:

[0036]

[0037]

[0038] Where downsampling(·) represents a 1 / 2 downsampling operation. This indicates a scale feature where the number of channels in the third-stage subnetwork is 128. This indicates a scale feature where the number of channels in the second-stage subnetwork is 256;

[0039] Step 2.3.2, combine the features F2 at three different scales. and Adjacent features at two different scales are enhanced by a reconstruction hybrid attention module and then fused. The fused features are then filtered and integrated again by a reconstruction hybrid attention module to obtain multi-scale fused features. and

[0040]

[0041]

[0042] in, This is a multi-scale fusion feature with 64 channels after the third-stage sub-network enhancement. This is a multi-scale fusion feature with 64 channels after the second-stage sub-network enhancement; ↑ represents a 2× upsampling operation.

[0043] Step 2.3.3: Fuse features across multiple scales. and After fusion, the hybrid attention module is reconstructed to further filter and integrate the fused features, thereby gradually focusing on different scales and semantic levels to obtain the third-stage multi-scale fusion feature W:

[0044]

[0045] in, This represents the multi-scale fusion features of the third-stage subnetwork after gradual screening and fusion, while also incorporating these features. Features enhanced by reconstructed hybrid attention module and multi-scale fusion features The images are then convolved and compressed into the RGB channels respectively, resulting in reconstructed and enhanced images with resolutions of H / 2×W / 2 and H / 4×W / 4. and

[0046]

[0047]

[0048] Step 2.3.4: The multi-scale fused feature W is used to obtain the third-stage sub-network enhanced feature map F3 through residual connections. The third-stage process is represented as follows:

[0049] F3 = W + F2 (13).

[0050] In step 2.4, the enhanced image is reconstructed. The process is represented as:

[0051]

[0052] The specific process of step 3 is as follows:

[0053] Step 3.1: Reconstruct the image using the mean squared error (MSE) metric. With label image I h The pixel distance between them is used to obtain the pixel loss L. Pixel :

[0054]

[0055] Among them, FMSE It is an MSE metric used to measure the difference between an enhanced image and the corresponding ground truth.

[0056] Step 3.2, transfer the label image I h , Reconstructed and enhanced images at different resolutions Input to VGG 19 The network transforms the MSE distance into a perceptual distance between the predicted image and the real image at different scales, thus obtaining the multi-scale perceptual loss L. Multi :

[0057]

[0058] Among them, L VGG It is VGG 19 The MSE distance between features extracted by the network, with γ2 and γ4 being trade-off parameters;

[0059] Reconstruct and enhance images and label image I h Calculate the color loss L using the color loss function. Color :

[0060]

[0061] in,() p Represents a pixel, and ∠(,) is an operator for calculating the angle between two colors;

[0062] Step 3.3, the pixel loss L Pixel Multi-scale sensing loss L Multi and color loss L Color The weighted sums are then used to obtain the total target loss L:

[0063] L=λ1L Pixel +λ2L Multi +λ3L Color (18)

[0064] Where λ1, λ2, and λ3 represent the loss weights, respectively;

[0065] Step 3.4: Calculate the error gradient direction, update the target total loss L using the Adam optimizer, apply the updated network model to the next set of training samples, and repeat this process until the total loss of the network on the training set tends to stabilize, thus obtaining the optimal model parameters.

[0066] The beneficial effects of this invention are as follows:

[0067] (1) The overall network structure of this invention is a progressive multi-scale feature fusion structure, which transforms the multi-scale feature fusion process from direct extraction and fusion on low-light degraded images into a gradual enhancement and fusion process. This structure achieves efficient enhancement and fusion of features from coarse to fine, and has a significant effect on reducing the distortion of low-light enhanced images.

[0068] (2) This invention generates an adaptive refinement hybrid selection kernel by designing a reconstruction hybrid attention module, which is used to adaptively fuse spatial and channel information of features. This method fully considers the correlation between spatial and channel information, avoids redundancy of feature information, and enhances the ability of features to express local information. As a result, the reconstructed image performs excellently in the restoration of local details and adaptive enhancement of different illumination areas.

[0069] (3) The present invention consists of a lightweight end-to-end network composed of three sub-networks, each of which has strong independence. Therefore, the number of sub-networks can be flexibly selected according to the specific application scenario, which has good flexibility and portability. Attached Figure Description

[0070] Figure 1 This is a diagram of a progressive multi-scale feature fusion network structure designed in the progressive low-light image enhancement method based on hybrid attention in this invention.

[0071] Figure 2 This is a structural diagram of the reconstruction hybrid attention module in the progressive low-light image enhancement method based on hybrid attention of the present invention;

[0072] Figures 3(a) to 3(j) This is a comparison of the results of Example 2 in the progressive low-light image enhancement method based on hybrid attention proposed in this invention;

[0073] Figure 4 This is a comparison chart showing the relationship between the model parameter magnitudes and PSNR levels of each method in Embodiment 2 of the progressive low-light image enhancement method based on hybrid attention of the present invention.

[0074] Figures 5(a) to 5(f) This is a comparison chart of the results of Example 3 in the progressive low-light image enhancement method based on hybrid attention of the present invention. Detailed Implementation

[0075] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0076] This invention relates to a progressive low-light image enhancement method based on hybrid attention. First, the low-light image is input into a network model, and the output is the reconstructed image with restored normal light, i.e., the reconstructed and enhanced image. For example... Figure 1As shown, the method includes a progressive multi-scale feature fusion structure and a reconstruction hybrid attention module. The progressive multi-scale feature fusion structure is a lightweight end-to-end network composed of three sub-networks, each with strong independence. The number of sub-networks can be flexibly selected according to specific application scenarios, exhibiting good flexibility and transferability. This structure transforms the multi-scale feature fusion process from direct extraction and fusion on low-light degraded images into a progressive enhancement and fusion process, thereby achieving efficient enhancement and fusion of features from coarse to fine. The input low-light image is convolved (conv) to obtain an initial feature map with a height (H), width (W), and 64 channels. in The data domain is represented, and then passed through the first sub-network, by the reconstruction hybrid attention module (see...). Figure 2 The initial feature integration is performed. Subsequently, the second and third sub-networks progressively increase the fusion of features at scales of 128 channels (H / 2, W / 2) and 256 channels (H / 4, W / 4), fully utilizing multi-scale features in a progressive fusion manner to improve the global information representation capability of the features. The reconstruction hybrid attention module generates an adaptive refinement hybrid selection kernel to adaptively fuse spatial and channel information of the features, fully considering the correlation between spatial and channel information, avoiding feature redundancy, and enhancing the feature's ability to represent local information. Finally, pixel loss, multi-scale perceptual loss, and color loss are introduced to reconstruct higher-quality low-light image restoration results.

[0077] The low-light images referred to in this invention are images captured in low-light environments. Each image sensor manufacturer has its own standards; for example, Hikvision typically classifies low-light environments into the following categories: dark level (0.01Lux-0.1Lux), moonlight level (0.001Lux-0.01Lux), and starlight level (less than 0.001Lux). Lux is a unit used to measure light intensity, which represents the luminous flux of visible light received per unit area.

[0078] This invention relates to a progressive low-light image enhancement method based on hybrid attention, which specifically includes the following steps:

[0079] Step 1, obtain the dataset, specifically:

[0080] Step 1.1: From the MIT-Adobe FiveK (MIT) dataset, which contains 5000 pairs of manually adjusted low-light and normal-light image pairs, a training set of 97% and a test set of 3% is obtained, resulting in 4850 training pairs and 150 test pairs. The low-light images in the dataset are used as the input low-light image I of the method of this invention. l Normal lighting image as label image I h .

[0081] Step 1.2: To make the most of the limited dataset, during training, the training set image pairs partitioned in Step 1.1 are randomly cropped and resized to 256×256 pixels to augment the training dataset. Simultaneously, the normal lighting images in the augmented training dataset are scaled by 1 / 2 and 1 / 4 respectively, resulting in label images of sizes 128×128 and 64×64 pixels. and

[0082] Step 2: Construct a progressive multi-scale feature fusion network model for low-light image I. l The reconstruction is performed using a progressive multi-scale feature fusion network model, which includes a first-stage sub-network, a second-stage sub-network, and a third-stage sub-network. In this invention, the progressive multi-scale feature fusion network is implemented step-by-step through three sub-networks, starting from the low-light image I... l The initial feature map extracted from the image is fed into the first-stage sub-network for preliminary feature pre-enhancement, thereby outputting the first-stage sub-network enhanced feature map F1.

[0083] Step 2 is implemented in the following steps:

[0084] Step 2.1, first, input the low-light image. After feature pre-extraction via convolution, an initial feature map is obtained, and the image is then passed to the first sub-network layer.

[0085]

[0086] in, This represents the initial feature map with 64 channels in the first-stage sub-network. Represents the data field.

[0087] Step 2.2, map the initial features The input is processed by a Refined-Mixed Attention Network (RMAM) to filter and integrate features, thereby outputting a reconstructed mixed feature map f at the same scale as the input. RM64 The reconstruction of the hybrid attention module is carried out according to the following steps:

[0088] First, reconstruct the input feature mapping of the hybrid attention module. After sequentially applying global average pooling, channel compression convolution, PRELU nonlinear activation function, channel dilation convolution, and Sigmoid activation function, the channel feature descriptor is obtained.

[0089]

[0090] H = Sigmoid(F) cpc (3)

[0091] Among them, f cpc Represents input feature mapping The feature map is obtained through global average pooling, channel-compressed convolution, PRELU non-linear activation function, and channel-dilated convolution. Sigmoid(·) represents the Sigmoid activation function. Simultaneously, the input feature map... The parallel processing involves global average pooling and global max pooling followed by a concat operation to merge the data. Then, it is sequentially processed through convolution and a sigmoid activation function to obtain the spatial attention map.

[0092] f gcc =Conv(Concat(GAP(X) in ),GMP(X in (4)

[0093] S = Sigmoid(f gcc (5)

[0094] Among them, f gcc Represents input feature mapping Parallel processing is achieved through global average pooling and global max pooling, concat operation merging, and feature mapping after convolution.

[0095] Next, the channel descriptor H and the spatial attention map S are used to activate the input feature map, respectively. This generates the channel reshaping feature map f. h and spatial remodeling eigenmap f s :

[0096]

[0097]

[0098] Then, the channel reshaping feature map f is obtained by adding elements one by one. h and spatial remodeling eigenmap f s Initial fusion is performed, and the fused feature maps are initially compressed using global average pooling and convolution to generate a compression selection kernel.

[0099]

[0100] in, Let represent the compression selection kernel, where n is the feature compression factor. Next, the compression selection kernel... Conv is a convolutional layer that uses convolution, PRELU nonlinear activation, and two parallel channels for dilation. i After being input into the softmax activation function, an adaptive refinement hybrid selection kernel M is generated. i :

[0101]

[0102]

[0103] Where, conv i This represents the i-th channel dilated convolutional layer. Finally, the channel reshaping feature map f... h and spatial remodeling eigenmap f s By adaptively refining the hybrid selection kernel M i We perform weighted fusion and obtain the renormalized features through residual connections:

[0104]

[0105] Among them, f RM64 Representation and input feature mapping Same-scale reconstruction of hybrid feature maps.

[0106] Step 2.3, Feature Mapping The reconstructed hybrid feature map f is obtained after the reconstruction hybrid attention module. RM64 Then, preliminary feature processing is completed through residual connections, resulting in the first-stage sub-network enhanced feature map F1. The first-stage processing can be represented as follows:

[0107]

[0108] in, The first-stage sub-network is used to enhance the feature mapping through residual connections.

[0109] Step 3: The enhanced feature map F1 output in Step 2 is used as the input of the second-stage sub-network. The second-stage sub-network extracts features of two scales based on the enhanced feature map F1 and completes adaptive multi-scale feature fusion, thereby obtaining the enhanced feature map F2 of the second-stage sub-network.

[0110] Step 3 is implemented in the following steps:

[0111] Step 3.1: Perform a 1 / 2 downsampling operation on the first-stage enhanced feature map F1, and obtain the feature map of another scale through convolution. Represented as:

[0112]

[0113] Where downsampling(·) represents a 1 / 2 downsampling operation. This represents the scale feature of a second-stage subnetwork with 128 channels.

[0114] Step 3.2, feature mappings F1 and F2 at two scales. After being enhanced by the reconstruction hybrid attention module (see step 2.2), the features are fused. The fused features are then subjected to feature filtering and integration by the reconstruction hybrid attention module again, thus obtaining the second-stage multi-scale fused features.

[0115]

[0116] Where RMA(·) represents the feature map after reconstructing the hybrid attention module. ↑ represents a 2× upsampling operation. It is a multi-scale fusion feature with 64 channels after the second-stage sub-network enhancement.

[0117] Step 3.3, Multi-scale fusion features The second-stage sub-network enhances the feature mapping F2 through residual connections. The second-stage process can be represented as follows:

[0118]

[0119] in, The second-stage sub-network is used to enhance the feature mapping through residual connections.

[0120] Step 4: The enhanced feature map F2 output from Step 3 is used as the input to the third-stage sub-network. Based on the enhanced feature map F2, the third-stage sub-network extracts features of three different scales and completes the adaptive multi-scale feature progressive fusion pairwise, thus obtaining the enhanced feature map F3 of the third-stage sub-network.

[0121] Step 4 is implemented in the following steps:

[0122] Step 4.1: Perform continuous 1 / 2 downsampling operations on the second-stage enhanced feature map F2, and obtain two feature maps of different scales through convolution. and Represented as:

[0123]

[0124]

[0125] Where downsampling(·) represents a 1 / 2 downsampling operation. This indicates a scale feature where the number of channels in the third-stage subnetwork is 128. This represents the scale feature of the second-stage subnetwork with 256 channels.

[0126] Step 4.2: In the third-stage sub-network, a progressive, phased pairwise fusion method is adopted in the feature fusion stage, using F2 of features at three different scales. and The adjacent two scale features are enhanced by the reconstruction hybrid attention module (see step 2.2) and then fused. The fused features are then filtered and integrated by the reconstruction hybrid attention module again to obtain the multi-scale fused features. and

[0127]

[0128]

[0129] in, This is a multi-scale fusion feature with 64 channels after the third-stage sub-network enhancement. This is a multi-scale fusion feature with 64 channels after the second-stage sub-network enhancement, where ↑ represents a 2× upsampling operation.

[0130] Step 4.3, Multi-scale feature fusion and After further fusion, the hybrid attention module is reconstructed to finally filter and integrate the further fused features, thereby gradually focusing on different scales and semantic levels, thus obtaining the third-stage multi-scale fusion feature W:

[0131]

[0132] in, This represents the multi-scale fusion features of the third-stage subnetwork after gradual selection and fusion. Simultaneously, the features... Features enhanced by reconstructed hybrid attention module and multi-scale fusion features The images are then convolved and compressed into the RGB channels respectively, resulting in reconstructed and enhanced images with resolutions of H / 2×W / 2 and H / 4×W / 4. and

[0133]

[0134]

[0135] Step 4.4: The multi-scale fused feature W is used to obtain the third-stage sub-network enhanced feature map F3 through residual connections. The third-stage process can be represented as:

[0136] F3 = W + F2(23)

[0137] in, The feature mapping is enhanced by the third-stage sub-network obtained through residual connections.

[0138] Step 5: Compress the third-stage sub-network enhancement feature map F3 into the RGB channel to obtain the reconstructed and enhanced image. Step 5 is implemented in the following steps:

[0139] Step 5.1: In the third stage, the enhanced feature map F3 of the sub-network is compressed into the RGB channel through convolution and residual connections to obtain a reconstructed and enhanced image with a resolution of H×W. Therefore, the reconstruction process can be represented as:

[0140]

[0141] in, It is a reconstructed and enhanced image obtained through residual connections.

[0142] Step 6: Reconstruct and enhance the images at different resolutions obtained from all the above steps. and label image I h , The error gradient is input into the joint loss function for backpropagation and then updated using the Adam optimizer. The updated network model is then applied to the next set of training samples, and this process is repeated to obtain the optimal model parameters.

[0143] Step 6 is implemented in the following steps:

[0144] Step 6.1: To ensure the output is as close as possible to the real image, this invention reconstructs the image using the mean square error (MSE) metric. With label image I h The pixel distance between them is used to obtain the pixel loss L. Pixel :

[0145]

[0146] Among them, F MSE It is an MSE index used to measure the difference between an enhanced image and the corresponding ground truth.

[0147] Label image I h , Reconstructed and enhanced images at different resolutions Input to VGG 19To account for both low-level and high-level features at different scales, we transform the MSE distance into a perceptual distance between the predicted and ground truth images at different scales, thus obtaining the multi-scale perceptual loss L. Multi :

[0148]

[0149] Among them, L VGG It is VGG 19 The MSE distance between features extracted by the network was calculated with trade-off parameters γ2 and γ4 of 0.8 and 0.6, respectively.

[0150] In order to recover rich and realistic images and better solve the problem of color distortion in reconstructed images, this invention will reconstruct and enhance the image. and label image I h Calculate the color loss L using the color loss function. Color :

[0151]

[0152] in,() p Represents a pixel, and ∠(,) is an operator for calculating the angle between two colors.

[0153] Step 6.2: Reduce pixel loss L Pixel Multi-scale sensing loss L Multi and color loss L Color The weighted sums are then used to obtain the total target loss L:

[0154] L=λ1L Pixel +λ2L Multi +λ3L Color (28)

[0155] In the experiment, the weights λ1, λ2 and λ3 of these losses were empirically set to 1.5, 1 and 0.5, respectively.

[0156] Step 6.3: Calculate the error gradient direction and update the parameters of the target total loss L using the Adam optimizer. Apply the updated network model to the next set of training samples, repeating this process until the total loss of the network on the training set becomes stable, thus obtaining the optimal model parameters. In the experiments of this method, after training all training samples for 1000 rounds, the total loss of the training set basically no longer changed, at which point the training can be terminated.

[0157] Step 7: During the testing phase, the optimal model parameters trained in the training phase are loaded into the model of this method. After inputting the low-light images in the test set, the reconstructed normal light restoration image can be output. The feature propagation process is similar to the reconstruction process in the training phase.

[0158] Example 1

[0159] Table 1 shows the values ​​of three mainstream image quality evaluation metrics, PSNR, SSIM, and LPIPS, obtained using existing technologies such as Enl-GAN (EnlightenGAN), MBLLEN, DeepUPE, RetinexDIP, DeepLPF, LL-Fomer, MIRNet, UHDF, LLFlow, MBPNet, and the method of this invention, when using the LOL dataset, SYS dataset, and MIT-Adobe FiveK (MIT) to construct three different training and testing datasets, respectively. Higher PSNR and SSIM values ​​indicate that the enhanced image quality is closer to the label image, while lower LPIPS values ​​indicate that the enhanced image quality is closer to the label image. Specific experimental metrics are shown in the table below.

[0160] Table 1: Comparison of performance indicators between the method of this invention and existing low-light image enhancement methods

[0161]

[0162]

[0163] In comparison, the method of this invention is generally superior to several other mainstream comparison methods in terms of objective evaluation indicators.

[0164] Example 2

[0165] To further verify the visual restoration effect of the method of the present invention, we used the method of the present invention and seven other latest benchmark methods (see...). Figures 3(a) to 3(j) Figure 3(a) shows the input low-light image, and Figure 3(b) shows the image after processing Figure 3(a) using the Enl-GAN (EnlightenGAN) method.

[0166] Figure 3(c) is the image of Figure 3(a) processed using the MBLLEN method; Figure 3(d) is the image of Figure 3(a) processed using the LL-Fomer method; Figure 3(e) is the image of Figure 3(a) processed using the MIRNet method; Figure 3(f) is the image of Figure 3(a) processed using the UHDF method; Figure 3(g) is the image of Figure 3(a) processed using the LLFlow method; Figure 3(h) is the image of Figure 3(a) processed using the MBPNet method; Figure 3(i) is the image of Figure 3(a) processed using the method of this invention; and Figure 3(j) is the label image.

[0167] Through the Figures 3(a) to 3(j)A comparison reveals that the other seven benchmark methods all result in varying degrees of overexposure when enhancing the LED sign in the image, leading to low recognizability of the enhanced text. However, the method of this invention efficiently and adaptively fuses features at all levels of the low-light image, possessing adaptive enhancement weights in different lighting regions. This results in high recognizability of the enhanced LED sign text in Figure 3(i) and a visual effect most similar to the label image in Figure 3(j).

[0168] To demonstrate the effective balance between lightweight design and performance, the relationship between model parameters and PSNR performance metrics was statistically analyzed for eight advanced methods: Enl-GAN (EnlightenGAN), MBLLEN, DeepUPE, RetinexDIP, DeepLPF, MIRNet, LLFlow, MBPNet, and the proposed method using only the first-stage sub-network enhancement model Ours-1, the first and second-stage sub-network enhancement model Ours-1+2, and all three-stage sub-network enhancement models Ours-1+2+3. Figure 4 As shown, the horizontal axis represents the network parameters (MB) in logarithmic coordinates, and the vertical axis represents the average PSNR value across the three databases.

[0169] from Figure 4 The simulation results show that the present invention leads other methods in terms of performance enhancement while maintaining a moderate computing cost.

[0170] Example 3

[0171] Enhancing blind-input low-light images in real-world scenarios is a significant challenge. To evaluate the robustness of the method of this invention in such challenging scenarios, the method of this invention was used along with four other high-performance benchmark methods: Figure 5(a) shows the input low-light image, Figure 5(b) shows the image of Figure 5(a) processed using the MIRNet method, Figure 5(c) shows the image of Figure 5(a) processed using the UHDF method, Figure 5(d) shows the image of Figure 5(a) processed using the LL-Former method, Figure 5(e) shows the image of Figure 5(a) processed using the LLFlow method, and Figure 5(f) shows the image of Figure 5(a) processed using the method of this invention.

[0172] Figures 5(a) to 5(f) The invention presents a comparative demonstration of the recovery results of low-light images captured from various real-world scenes using a smartphone with night mode turned off. The comparison shows that the method of the present invention exhibits remarkable robustness in recovering the brightness, color, and low signal-to-noise ratio dark details of blind-input low-light images.

[0173] Table 2 uses three non-reference quality assessment metrics, namely NIQE, BRISQUE, and RankIQA, to compare and evaluate different methods. The lower the value of each metric, the higher the image restoration quality.

[0174] Table 2: Comparison of metrics for real-world scene images

[0175]

[0176] The experimental results of Examples 1-3 show that, compared with existing low-light image enhancement methods, the present invention has a stronger enhancement and recovery capability for areas with low signal-to-noise ratio in low-light images, can obtain more image details, and the recovered image has a better visual perception effect, without obvious distortion and artifacts, and is closer to the real image.

Claims

1. A progressive low-light image enhancement method based on hybrid attention, characterized in that: Specifically, the steps include the following: Step 1, acquire images containing low light. and normal lighting image The image dataset was divided into a training set and a test set, containing images with normal lighting. For label images; Step 2: Construct a progressive multi-scale feature fusion network model for low-light images. The progressive multi-scale feature fusion network model is reconstructed, comprising a first-stage subnetwork, a second-stage subnetwork, and a third-stage subnetwork. Map the initial features The input is processed by a hybrid attention module to filter and integrate features, resulting in a reconstructed hybrid feature map at the same scale as the input. The reconstruction of the hybrid attention module is carried out according to the following steps: First, reconstruct the input feature mapping of the hybrid attention module. After sequentially applying global average pooling, channel compression convolution, PRELU nonlinear activation function, channel dilation convolution, and Sigmoid activation function, the channel feature descriptor is obtained. : （1） （2） in, Represents input feature mapping Feature mappings obtained through global average pooling, channel-compressed convolution, PRELU non-linear activation function, and channel-dilated convolution. This represents the Sigmoid activation function; it also serves as the input feature map. The parallel processing involves global average pooling and global max pooling followed by a concat operation to merge the data. Then, it is sequentially processed through convolution and a sigmoid activation function to obtain the spatial attention map. : （3） （4） in, Represents input feature mapping Parallel processing is achieved through global average pooling and global max pooling, concat operation merging, and feature mapping after convolution; Next, we will use the channel description operator. Spatial attention map To stimulate input feature mapping This generates channel reshaping feature maps. and spatial remodeling feature mapping : （5） （6） Then, the channel remodeling feature map is remapped by adding elements one by one. and spatial remodeling feature mapping Initial fusion is performed, and the fused feature maps are initially compressed using global average pooling and convolution to generate a compression selection kernel. : （7） in, This represents the compression selection kernel, where n is the feature compression factor; next, the compression selection kernel... The convolutional layer is expanded using convolution, PRELU nonlinear activation, and two parallel channels. After being input into the softmax activation function, an adaptive refined hybrid selection kernel is generated. : （8） （9） in, Representing the One channel dilated convolutional layer; finally, channel reshaping feature maps. and spatial remodeling feature mapping Through adaptive refinement of hybrid selection kernel We perform weighted fusion and obtain the renormalized features through residual connections: （10） in, Representation and input feature mapping Reconstructing hybrid feature maps at the same scale; The third-stage subnetwork employs a progressive, phased pairwise fusion method in the feature fusion stage, using features at three different scales. , and Adjacent features at two different scales are enhanced by a reconstruction hybrid attention module and then fused. The fused features are then filtered and integrated again by a reconstruction hybrid attention module to obtain multi-scale fused features. and : （11） （12） in, This is a multi-scale fusion feature with 64 channels after the third-stage sub-network enhancement. This is a multi-scale fusion feature with 64 channels after the second-stage sub-network enhancement. represent Upsampling operation; Step 3: Use the training set divided in Step 1 to train the progressive multi-scale feature fusion network model constructed in Step 2, and output the trained progressive multi-scale feature fusion network model. Step 4: Input the test set divided in Step 1 into the progressive multi-scale feature fusion network model trained in Step 3, and output the reconstructed normal optical restoration image.

2. The progressive low-light image enhancement method based on hybrid attention according to claim 1, characterized in that: In step 1, the training set images are randomly cropped and resized to 256×256 pixels to augment the training dataset. Simultaneously, normal lighting images from the augmented training dataset are also included. Perform separately and By scaling proportionally, label images with sizes of 128×128 and 64×64 pixels were obtained. and .

3. The progressive low-light image enhancement method based on hybrid attention according to claim 2, characterized in that: The specific process of step 2 is as follows: Step 2.1, from the low-light image The initial feature map extracted from the image is fed into the first-stage sub-network for preliminary feature pre-enhancement, thereby outputting the enhanced feature map of the first-stage sub-network. ; The enhanced feature maps output from steps 2.2 and 2.1 As input to the second-stage sub-network, the second-stage sub-network enhances the feature mapping of the first-stage sub-network. Based on this, features at two scales are extracted and adaptive multi-scale feature fusion is performed to obtain the second-stage sub-network enhanced feature mapping. ; Enhanced feature maps output from steps 2.3 and 2.2 As input to the third-stage sub-network, the third-stage sub-network enhances the feature mapping of the second-stage sub-network. Based on this, three different scale features are extracted and adaptive multi-scale feature pairwise progressive fusion is performed to obtain the third-stage sub-network enhanced feature mapping. ; Step 2.4: Enhance the feature mapping of the third-stage sub-network. The image is compressed into the RGB channel to obtain the reconstructed and enhanced image. .

4. The progressive low-light image enhancement method based on hybrid attention according to claim 3, characterized in that: The specific process of step 2.1 is as follows: Step 2.1.1, Input a low-light image After feature pre-extraction via convolution, an initial feature map is obtained. The image is then passed to the first-stage sub-network, resulting in an initial feature map with 64 channels. : （13） in, H represents the data field, where H represents the image height and W represents the image width. Step 2.1.2, map the initial features. The input is processed by a hybrid attention module to filter and integrate features, resulting in a reconstructed hybrid feature map at the same scale as the input. ; Step 2.1.3, Feature Mapping The reconstructed hybrid feature map is obtained after the reconstruction hybrid attention module. Subsequently, preliminary feature processing is completed through residual connections, thereby obtaining the first-stage sub-network enhanced feature mapping. The processing procedure is represented as follows: （14）。 5. The progressive low-light image enhancement method based on hybrid attention according to claim 4, characterized in that: The specific process of step 2.2 is as follows: Step 2.2.1, enhance the feature mapping in the first stage. conduct Downsampling operation, and then convolution to obtain feature maps at another scale. , represented as: （15） in, express Downsampling operation, This indicates a scale feature where the number of channels in the second-stage subnetwork is 128; Step 2.2.2, feature mapping and The features are enhanced separately by a reconstruction hybrid attention module and then fused. The fused features are then further enhanced by a reconstruction hybrid attention module to filter and integrate the fused features, thus obtaining the second-stage multi-scale fused features. : （16） in, This represents the feature map after the hybrid attention module has been reconstructed. represent Upsampling operation, It is a multi-scale fusion feature with 64 channels after the second-stage sub-network enhancement; Step 2.2.3: Fuse features across multiple scales The second-stage subnetwork is enhanced with feature maps obtained through residual connections. The second stage process is represented as follows: （17）。 6. The progressive low-light image enhancement method based on hybrid attention according to claim 5, characterized in that: The specific process of step 2.3 is as follows: Step 2.3.1, enhance the feature mapping in the second stage. Perform continuously Downsampling operation, and convolution to obtain feature maps of two different scales. and , represented as: （18） （19） in, express Downsampling operation, This indicates a scale feature where the number of channels in the third-stage subnetwork is 128. This indicates a scale feature where the number of channels in the second-stage subnetwork is 256; Step 2.3.2: Fuse features across multiple scales and After fusion, the hybrid attention module is reconstructed to further filter and integrate the fused features, thereby gradually focusing on different scales and semantic levels to obtain the third-stage multi-scale fusion features. : （20） in, This represents the multi-scale fusion features of the third-stage subnetwork after gradual screening and fusion, while also incorporating these features. Features enhanced by reconstructed hybrid attention module and multi-scale fusion features Each channel is then convolved and compressed into the RGB channels to obtain a resolution of [resolution value missing]. and Reconstructed and enhanced images and : （21） （22） Step 2.3.4, Multi-scale feature fusion The third-stage subnetwork is enhanced with feature maps obtained through residual connections. The third stage process is represented as follows: （23）。 7. The progressive low-light image enhancement method based on hybrid attention according to claim 6, characterized in that: In step 2.4, the enhanced image is reconstructed. The process can be represented as: （24）。 8. The progressive low-light image enhancement method based on hybrid attention according to claim 7, characterized in that: The specific process of step 3 is as follows: Step 3.1: Reconstruct the image using the mean squared error (MSE) metric. With label image The pixel distance between them is used to obtain pixel loss. : （25） in, It is an MSE metric used to measure the difference between an enhanced image and the corresponding ground truth. Step 3.2, transfer the label image , , Reconstructed and enhanced images at different resolutions , , Input to The network transforms the MSE distance into a perceptual distance between the predicted image and the real image at different scales, thus obtaining the multi-scale perceptual loss. : （26） in, It is MSE distance between features extracted by the network and To weigh parameters; Reconstruct and enhance images and label images Calculate color loss using the color loss function : （27） in, Represents one pixel. It is an operator for calculating the angle between two colors; Step 3.3, pixel loss Multi-scale sensing loss and color loss The weighted sums are then used to obtain the total target loss. : （28） in, , and These represent the loss weights; Step 3.4, calculate the error gradient direction, and calculate the total target loss. The parameters are updated using the Adam optimizer, and the updated network model is applied to the next set of training samples. This process is repeated until the total loss of the network on the training set tends to stabilize, thus obtaining the optimal model parameters.

Images