Low-light image enhancement method based on Retinex multi-scale fusion

By constructing the Retinex multi-scale adaptive fusion low-light image enhancement network RMAF-Net, the problems of unnatural enhancement effects and texture details being buried by noise in the existing technology are solved, achieving more natural brightness enhancement and color restoration, and improving image quality.

CN117974524BActive Publication Date: 2026-11-10XIDIAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410209728.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-02-26
Publication Date
2026-11-10
Estimated Expiration
2044-02-26

AI Technical Summary

Technical Problem

Existing low-light image enhancement methods are insufficient in terms of texture detail preservation and noise suppression, resulting in unnatural enhancement effects, and detail information is easily buried when multi-scale feature fusion is performed.

Method used

We construct a low-light image enhancement network RMAF-Net based on Retinex multi-scale adaptive fusion. It is trained by combining smoothing loss and reconstruction loss through a Retinex semi-decoupled network and an adaptive cross-fusion module to achieve accurate estimation of illumination images and preservation of detail information.

Benefits of technology

It achieves more natural brightness enhancement and color restoration, improves the overall image quality, and outperforms existing technologies in objective metrics such as PSNR, SSIM, and LPIPS, with more complete information recovery in darker areas.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117974524B_ABST
    Figure CN117974524B_ABST
Patent Text Reader

Abstract

This invention discloses a low-light image enhancement method based on Retinex multi-scale fusion, mainly addressing the problems of unnatural image effects and noise-induced textural detail obscuring the image after processing by existing methods. The method includes: 1) preprocessing the initial image to construct a training set; 2) constructing a low-light image enhancement network comprising a Retinex semi-decoupled network and an adaptive cross-fusion module, extracting features through parallel processing of three branches, then estimating components based on a bilateral grid, obtaining multi-level features through parallel decoupling at different scales and fusing them; 3) constructing a joint loss formula consisting of semi-decoupled smoothing loss and reconstruction loss; 4) training the network using the training set until convergence; 5) inputting the image to be processed into the trained network to achieve image enhancement. This invention effectively improves the low-light image effect, resulting in more natural brightness enhancement and color restoration, while also increasing the completeness of information recovery in darker areas.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and further relates to low-light image processing methods, specifically a low-light image enhancement method based on Retinex multi-scale fusion. It can be used to enhance low-light images, making their brightness improvement and color restoration more natural, while also restoring information in darker areas more completely. Background Technology

[0002] Low-light environments are unavoidable during image capture, such as dusk and night, unlit indoor environments, and backlighting. These low-light images suffer from low contrast and unclear details, negatively impacting the subjective viewing experience and severely affecting visually-dependent target detection and recognition. Therefore, illumination enhancement for low-light images has significant practical value.

[0003] Existing low-light algorithms based on Retinex theory can be broadly categorized into two types: traditional methods and deep learning methods. For example, the Low-light Image Enhancement (LIME) algorithm proposed by Guo et al. initializes the illumination map of a low-light image by selecting the maximum value in each pixel channel, then adds a structural prior to refine the initial illumination map, and uses reflection as the final enhancement result. See Xiao Jie Guo, Yu Li, and Haibin Ling. Lime: Low-light image enhancement via illumination map estimation. IEEE Transactions on Image Processing, 26(2):982–993, 2017. With the development of deep learning, numerous experiments have shown that deep learning methods have stronger generalization ability and enhancement effect than traditional methods. For example, Wei et al. designed a deep network called RetinexNet, which uses a decomposition network (Decom-Net) for decomposition and an enhancement network (Enhance-Net) for illumination adjustment based on convolutional neural networks to obtain the final enhancement result. See C. Wei, W. Wang, W. Yang, and J. Liu, “Deep retinex decomposition for low-light enhancement,” in British Machine Vision Conference, 2018. Lv et al. designed a multi-branch enhancement network MBLLEN-Net, which uses a feature extraction module and an enhancement module based on a fully convolutional neural network to extract rich multi-level features from low-light images. The final fusion module uses a 1×1 convolutional kernel to merge the multi-branch features to obtain the final output. See LV Feifan, LU Feng, WU Jianhua, et al. MBLLEN: Lowlight image / video enhancement using CNNs[C]. British Machine Vision Conference, 2018, 13: 133-145.

[0004] While the aforementioned existing technologies have improved the low-light image quality to some extent, they still have the following shortcomings: 1) The decomposition of the illumination component in traditional algorithms often requires manually designed constraints and parameters, which limits the generalization of such methods in various scenarios; in deep learning-based methods, the decomposition network simultaneously decomposes the illumination map and reflectance map, often resulting in unnatural enhancement effects. 2) Existing methods do not fully utilize multi-scale features, and the texture details of the image are easily obscured by noise during multi-scale information fusion. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of the aforementioned technologies by proposing a low-light image enhancement method based on Retinex multi-scale fusion. This method solves the problems of unnatural image effects after enhancement and the loss of texture details due to noise, resulting in more natural enhancement results in terms of brightness improvement and color restoration, while also providing more complete information recovery in darker areas.

[0006] To achieve the above objectives, the technical solution of the present invention includes the following:

[0007] (1) Obtain the training image set:

[0008] (1a) Normalize the initial image to limit its pixel values ​​to the range [0,1] to obtain a normalized image;

[0009] (1b) The normalized image is downsampled by multiples of 1, 1 / 2 and 1 / 4 in sequence to obtain the sampled low-resolution image, which forms the training image set;

[0010] (2) Constructing a low-light image enhancement network RMAF-Net based on Retinex multi-scale adaptive fusion within the PyTorch framework:

[0011] (2a) Construct a low-resolution lighting prediction module:

[0012] (2a1) Construct a channel self-attention module and six convolutional units. The first to fourth convolutional units are all convolutional layers, LeakyReLU activation layers and normalization layers connected in series, with padding values ​​of 0, kernel size of 3*3 and stride of 2. The fifth and sixth convolutional units are both convolutional layers and LeakyReLU activation layers connected in series, with padding values ​​of 1, kernel size of 3*3 and stride of 1.

[0013] (2a2) Concatenate the first to fourth convolutional units, and then concatenate the channel self-attention module and the fifth convolutional unit as two parallel branches with the concatenated output through the channel dimension splicing operation to obtain the sixth convolutional unit, thus obtaining the low-resolution illumination prediction module.

[0014] (2b) Constructing the Retinex semi-decoupled network:

[0015] (2b1) The content feature enhancement module is composed of two residual units and a channel self-attention module cascaded together;

[0016] (2b2) ​​After the content feature enhancement module and the low-resolution illumination prediction module, add bilateral grid upsampling, pixel-by-pixel division and pixel-by-pixel addition operations to obtain the Retinex semi-decoupled network; the input of this network is two images of different resolutions, which are fed into the content feature enhancement module and the low-resolution illumination prediction module respectively.

[0017] (2c) Constructing the low-light image enhancement network RMAF-Net based on Retinex multi-scale adaptive fusion:

[0018] (2c1) An adaptive cross-fusion module is constructed using nine convolutional units, pixel-wise addition, spatial dimension average pooling, channel dimension max pooling, and pixel-wise multiplication. Each of the nine convolutional units is a concatenated structure of convolutional layers and LeakyReLU activation layers, with a kernel size of 3*3 and a stride of 1. Except for the fourth convolutional unit, which has a padding value of 0, all other convolutional units have a padding value of 1. The first and second convolutional units are used to extract features from two different scales of image information input to the module, and then concatenated with pixel-wise addition and the third convolutional unit. The concatenated result is processed in parallel through two branches in the spatial and channel dimensions, and then subjected to matrix multiplication. Softmax operation is used to achieve branch interaction. Each branch undergoes parallel pixel-wise multiplication, and the parallel result is concatenated with the pixel-wise addition operation as the output of the adaptive cross-fusion module.

[0019] (2c2) The low-light image enhancement network RMAF-Net, which is composed of two Retinex semi-decoupled networks that process in parallel at different scales and an adaptive cross-fusion module, consists of two Retinex semi-decoupled networks that process in parallel at different scales and an adaptive cross-fusion module.

[0020] (3) Construct a semi-decoupled smooth loss L smooth and reconstruction losses L r The formula for the combined loss is:

[0021] L total =ω s L smooth (I,L)+ω r L r (output,gt)

[0022] Where I is the illumination map output by the Retinex semi-decoupled network, L is the input low-light map; output is the enhanced image output by the network, gt is the normal illumination map referenced in the dataset, and ω s ω r These are the semi-decoupling smoothing weight factor and the reconstruction weight factor;

[0023] (4) Training the low-light image enhancement network RMAF-Net:

[0024] The training set is input into RMAF-Net, and the network parameters are iteratively updated using the Adam optimization algorithm until the loss function converges, thus obtaining the trained low-light image enhancement network.

[0025] (5) Enhance realistic low-light images:

[0026] Real low-light images are input into the trained network to perform low-light image enhancement, and the enhanced image is output.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] First, because this invention constructs a Retinex semi-decoupled network based on illumination prediction, it obtains a better decoupling effect through accurate estimation of the illumination image, thereby achieving a more natural enhancement effect.

[0029] Secondly, because the present invention constructs an adaptive fusion module, it can better preserve and enhance detailed information during multi-scale fusion, thereby achieving the goal of improving the enhancement effect;

[0030] Third, the weighted loss training network designed in this invention comprehensively considers decoupling effect, local texture and color factors, so that the edge texture information of the enhanced image is better preserved and the image color reproduction is more accurate.

[0031] Simulation results show that the present invention achieves more natural brightness enhancement and color restoration, while restoring information in darker areas more completely. Compared with other existing algorithms, the processed results have more natural colors, the overall quality of the enhanced image is higher, and the objective indicators in Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and Learning Perceptual Image Patch Similarity (LPIPS) are all superior to existing technologies. Attached Figure Description

[0032] Figure 1 This is a flowchart illustrating the overall implementation of the present invention;

[0033] Figure 2 This is a schematic diagram of the low-resolution lighting prediction module constructed in this invention;

[0034] Figure 3This is the Retinex semi-decoupled network constructed in this invention; where (a) is a schematic diagram of the overall framework of the Retinex semi-decoupled network, and (b) is a schematic diagram of the content feature enhancement module structure in the Retinex semi-decoupled network.

[0035] Figure 4 This is a schematic diagram of the adaptive cross-fusion module structure constructed in this invention;

[0036] Figure 5 This is a schematic diagram of the RMAF-Net framework, a low-light image enhancement network based on Retinex multi-scale adaptive fusion, constructed in this invention.

[0037] Figure 6 The images show a comparison of the processing effects of this invention and existing low-light image enhancement methods on real low-light images; where: a represents four real low-light images; b represents the enhancement result of a using the LIME algorithm; c represents the enhancement result of a using the RRM algorithm; d represents the enhancement result of a using the RetinexNet algorithm; e represents the enhancement result of a using the MBLLEN-Net algorithm; f represents the enhancement result of a using the SCI-Net algorithm; g represents the enhancement result of a using the Bread algorithm; h represents the enhancement result of a using the LANet algorithm; and i represents the enhancement result of a using the method of this invention. Detailed Implementation

[0038] The specific embodiments and effects of the present invention will be further described below with reference to the accompanying drawings.

[0039] Example 1: Refer to Figure 1 The low-light image enhancement method based on Retinex multi-scale fusion proposed in this invention has the following specific implementation steps:

[0040] Step 1. Obtain the training image set:

[0041] (1a) Normalize the initial image to limit its pixel values ​​to the range [0,1] to obtain a normalized image;

[0042] (1b) The normalized image is downsampled by multiples of 1, 1 / 2 and 1 / 4 in sequence to obtain the sampled low-resolution image, which forms the training image set;

[0043] Step 2. Construct the low-light image enhancement network RMAF-Net based on Retinex multi-scale adaptive fusion in the PyTorch framework:

[0044] (2a) Construct a low-resolution lighting prediction module:

[0045] (2a1) Construct a channel self-attention module and six convolutional units. The first to fourth convolutional units are all convolutional layers, LeakyReLU activation layers and normalization layers connected in series, with padding values ​​of 0, kernel size of 3*3 and stride of 2. The fifth and sixth convolutional units are both convolutional layers and LeakyReLU activation layers connected in series, with padding values ​​of 1, kernel size of 3*3 and stride of 1.

[0046] The aforementioned channel self-attention module consists of three parallel branches: the first branch is composed of convolutional units, reshape operations, and matrix transpose operations in series; the second branch consists of convolutional units and reshape operations; and the third branch contains only convolutional units. Each branch's convolutional unit comprises a convolutional layer and a LeakyReLU activation layer, with a kernel size of 1*1 and a stride of 1. The module multiplies the output R1 from the first branch with the output R2 from the second branch to obtain matrix E(X), then multiplies the output R3 from the third branch with E(X) to obtain D(X), followed by a reshape operation to obtain Y(X), and finally outputs Y(X).

[0047] (2a2) The first to fourth convolutional units are concatenated, and the channel self-attention module and the fifth convolutional unit are treated as two parallel branches. After the concatenated output is concatenated with the channel-dimensional concatenation operation, they are concatenated with the sixth convolutional unit to obtain the low-resolution illumination prediction module. In this embodiment, the first to fourth convolutional units are concatenated to obtain R0, and the two parallel branches of the channel self-attention module and the fifth convolutional unit obtain R1 and R2 respectively. The three parallel results of R0, R1, and R2 are concatenated with the sixth convolutional unit after the channel-dimensional concatenation operation, and the output of this concatenation is used as the module output.

[0048] (2b) Constructing the Retinex semi-decoupled network:

[0049] (2b1) The content feature enhancement module is composed of two residual units and a channel self-attention module cascaded together. The two residual units have the same structure. Each residual unit consists of three identical convolutional units, which are concatenated convolutional layers and LeakyReLU activation layers with padding of 1, a kernel size of 3*3, and a stride of 1. In this embodiment, the residual unit is input with feature I, which is cascaded through the first to third convolutional units to obtain R. I and R are added to obtain R', which is then concatenated with the LeakyReLU activation layer to obtain the output of the residual unit, which is the output result of the residual unit.

[0050] (2b2) ​​After the content feature enhancement module and the low-resolution illumination prediction module, a double-sided grid upsampling, pixel-by-pixel division, and pixel-by-pixel addition operation is added to obtain the Retinex semi-decoupled network. The input of this network is two images of different resolutions, which are fed into the content feature enhancement module and the low-resolution illumination prediction module, respectively. In this embodiment, the two images I1 and I2 of different resolutions are used as the input of the Retinex semi-decoupled network. I1 is processed by the feature enhancement module to obtain I1', and I2 is processed by the low-resolution illumination prediction module to obtain I2'. Then, after the double-sided grid upsampling, pixel-by-pixel division, and pixel-by-pixel addition operation, the output result of the Retinex semi-decoupled network is obtained.

[0051] (2c) Constructing the low-light image enhancement network RMAF-Net based on Retinex multi-scale adaptive fusion:

[0052] (2c1) An adaptive cross-fusion module is constructed using nine convolutional units, pixel-wise addition, spatial dimension average pooling, channel dimension max pooling, and pixel-wise multiplication. The nine convolutional units are all convolutional layers and LeakyReLU activation layers in series, with a kernel size of 3*3 and a stride of 1. Except for the fourth convolutional unit, which has a padding value of 0, the padding values ​​of the other convolutional units are all 1. The first and second convolutional units are used to extract features from the two different scale image information input to the module, and then concatenate them with the pixel-wise addition and the third convolutional unit. The concatenated result is processed in parallel through two branches in the spatial dimension and the channel dimension, and then matrix multiplication is performed. The branches interact through the Softmax operation. Each branch is processed in parallel through the pixel-wise multiplication operation, and the parallel result is concatenated with the pixel-wise addition operation as the output of the adaptive cross-fusion module. The two branches mentioned above are as follows: the first branch is a spatial dimension average pooling operation → the fourth convolutional unit → the fifth and sixth convolutional units in parallel; the second branch is a channel dimension maximum pooling operation → the seventh convolutional unit → a sigmoid activation operation → the eighth and ninth convolutional units in parallel. The branch interaction is achieved by performing matrix multiplication on the outputs of the fifth and eighth convolutional units and the outputs of the sixth and ninth convolutional units, and then calibrating them using a softmax operation.

[0053] (2c2) The low-light image enhancement network RMAF-Net, based on Retinex multi-scale adaptive fusion, consists of two Retinex semi-decoupled networks that process in parallel at different scales and an adaptive cross-fusion module. In this embodiment, three images I0, I1, and I2 of different resolutions are input to the RMAF-Net network. Images I0 and I1 are processed by the first Retinex semi-decoupled network to obtain S1, and images I1 and I2 are processed by the second Retinex semi-decoupled network to obtain S2. The parallel results S1 and S2 are processed by the adaptive cross-fusion module to obtain the output result, which is the final result of the low-light image enhancement network RMAF-Net.

[0054] Step 3. Construct a semi-decoupled smoothing loss L smooth and reconstruction losses L r The formula for the combined loss is:

[0055] L total =ω s L smooth (I,L)+ω r L r (output,gt)

[0056] Where I is the illumination map output by the Retinex semi-decoupled network, L is the input low-light map; output is the enhanced image output by the network, gt is the normal illumination map referenced in the dataset, and ω s ω r These are the semi-decoupled smoothing weight factor and the reconstruction weight factor.

[0057] The semi-decoupling smoothing loss L smooth and reconstruction losses L r The expression is as follows:

[0058]

[0059] L r =‖gt-output‖ 2

[0060] Among them, I p This represents the p-th pixel in the illumination map output by the Retinex semi-decoupled network, where c represents all channels of each pixel. δ x δ y Let L represent the partial derivatives in the horizontal and vertical directions in the image space, respectively. p Let be the p-th pixel in the low-light image input, ∈ be the smoothing coefficient to prevent the denominator from being zero, and θ be the sensitivity coefficient.

[0061] Step 4. Train the low-light image enhancement network RMAF-Net:

[0062] The training set is input into RMAF-Net, and the network parameters are iteratively updated using the Adam optimization algorithm until the loss function converges, thus obtaining the trained low-light image enhancement network.

[0063] Step 5. Enhance the realistic low-light image:

[0064] Real low-light images are input into the trained network to perform low-light image enhancement, and the enhanced image is output.

[0065] Example 2: Refer to Figure 2-5 The low-light image enhancement method proposed in this embodiment is implemented in the same way as in Embodiment 1. Specific parameter settings are given below, and the implementation process of this invention is further described through examples:

[0066] Step A. Construct a training image set for a low-light image enhancement network based on Retinex multi-scale adaptive fusion.

[0067] 1.1) Select the LOL dataset from publicly available low-light dehazing image datasets. This dataset consists of 500 image pairs. Use 485 image pairs as the training set and the remaining 15 image pairs as the test set. Normalize the initial images to limit the pixel values ​​to the range [0,1] to obtain the training image set;

[0068] 1.2) Perform downsampling operations of 1, 1 / 2 and 1 / 4 on the original dataset to obtain low-resolution images.

[0069] Step B. Construct a low-resolution illumination prediction module consisting of convolutional units and channel self-attention modules, such as... Figure 2 As shown; this step is implemented as follows:

[0070] 2.1) Construct six convolutional units consisting of convolutional layers, activation layers, and normalization layers.

[0071] Among them, the first convolutional unit, the second convolutional unit to the fourth convolutional unit are all structures of convolutional layer, LeakyReLU activation layer and normalization layer connected in series, with padding value of 0, convolutional kernel size of 3*3 and convolution stride of 2.

[0072] Both the fifth and sixth convolutional units have a structure of a concatenated convolutional layer and a LeakyReLU activation layer, with a padding value of 1, a kernel size of 3*3, and a stride of 1.

[0073] 2.2) Construct a channel self-attention module consisting of three parallel branches, where:

[0074] The first branch has a structure consisting of a convolutional unit, a reshape operation, and a matrix transpose operation connected in series, wherein the convolutional kernel size is 1*1 and the stride is 1;

[0075] The second branch has a structure of convolutional units and reshape operations, where the convolutional kernel size is 1*1 and the stride is 1.

[0076] The third branch has a structure containing only convolutional units, wherein the convolutional kernel size is 1*1 and the stride is 1;

[0077] Multiply the output R1 from the first branch by the output R2 from the second branch to obtain matrix E(X); multiply the output R3 from the third branch by E(X) to obtain D(X); then obtain the module output Y(X) through the Reshape operation.

[0078] 2.3) Construct a low-resolution illumination prediction module consisting of a first convolutional unit, convolutional units two through six, and a channel self-attention module. First, the first, second, and fourth convolutional units are cascaded to obtain R0. Then, through two branches, the first branch is channel self-attention, yielding R1; the second branch is the fifth convolutional unit, yielding R2. These two branches run in parallel, and their parallel outputs R1, R2, and R0 are sequentially concatenated with the channel dimension and then cascaded with the sixth convolutional unit. The output of this cascade is the final output of the low-resolution illumination prediction module.

[0079] Step C. Construct a Retinex semi-decoupled network, such as... Figure 3 As shown; this step is implemented as follows:

[0080] 3.1) Construct a content feature enhancement module consisting of two residual units and a channel self-attention module, such as... Figure 3 As shown in (b) above, where:

[0081] The input feature I in the residual unit is cascaded through the first, second and third convolutional units to obtain R. I and R are added together to obtain R', which is then concatenated with the LeakyRelu activation layer to obtain the output of the residual unit. The padding value of the convolutional layer is 1, the kernel size is 3*3, and the stride is 1.

[0082] The content feature enhancement module consists of a first residual unit, a second residual unit, and a channel self-attention module cascaded together.

[0083] 3.2) Construct a Retinex semi-decoupled network consisting of a content feature enhancement module, a low-resolution illumination prediction module, a bilateral grid upsampling operation, pixel-by-pixel division, and pixel-by-pixel addition, such as... Figure 3As shown in (a) above, where:

[0084] The Retinex semi-decoupled network takes two images, I1 and I2, at different resolutions as input. I1 is processed by a content feature enhancement module to obtain I1', and I2 is processed by a low-resolution illumination prediction module to obtain I2'. Then, a bilateral grid upsampling operation is performed to obtain the final result of the Retinex semi-decoupled network.

[0085] Step D: Refer to Figure 4 and Figure 5 In the PyTorch framework, a low-light image enhancement network RMAF-Net based on Retinex multi-scale adaptive fusion is constructed. This step is implemented as follows:

[0086] 4.1) Construct an adaptive cross-fusion module consisting of two parallel processing branches, such as... Figure 4 As shown, where:

[0087] The fourth convolutional unit has padding values ​​of 0, a kernel size of 1*1, and a stride of 1.

[0088] The first, second, third, fifth, sixth, seventh, eighth, and ninth convolutional units all have a padding value of 1, a kernel size of 3*3, and a stride of 1.

[0089] The adaptive cross-fusion module takes two images, I1 and I2, at different resolutions as input. I1 is processed by the first convolutional unit to obtain I1', and I2 is processed by the second convolutional unit to obtain I2'. Then, I1' and I2' are sequentially cascaded with pixel-wise addition operations and the third convolutional unit to obtain I3. Next, they pass through two branches. The first branch is a spatial average pooling operation cascaded with the fourth convolutional unit, followed by the fifth and sixth convolutional units to obtain IS1 and IS2, respectively. The second branch is a channel-dimensional max pooling operation cascaded with the seventh convolutional unit and the sigmoid activation operation, followed by the eighth and ninth convolutional units to obtain IC1 and IC2, respectively.

[0090] Next, matrix multiplication and softmax operations are performed on (IC1, IS1) and (IC2, IS2) respectively to obtain ISatt and ICatt; pixel-wise multiplication is performed on (ISatt, I1`) and (ICatt, I2`) respectively, and the results are added pixel-wise to obtain the output result, which is the final result of the adaptive cross-fusion module.

[0091] 4.2) Construct a low-light image enhancement network RMAF-Net based on Retinex multi-scale adaptive fusion, consisting of two Retinex semi-decoupled modules that process in parallel at different scales and an adaptive cross-fusion module, as shown below. Figure 5 As shown, where:

[0092] The RMAF-Net takes three images of different resolutions, I0, I1, and I2, as input from the image set described in step A. The first and second Retinex semi-decoupled networks have the same structure as described in step C, and the adaptive cross-fusion module is the same as described in step 4.1.

[0093] Input images I0 and I1 are processed through a first Retinex semi-decoupled network to obtain S1; images I1 and I1 are processed through a second Retinex semi-decoupled network to obtain S2; S1 and S2 are processed through an adaptive cross-fusion module to obtain the output result, which is the final result of the low-light image enhancement network RMAF-Net.

[0094] Step E: Construct a joint loss formula consisting of semi-decoupled smoothing loss and reconstruction loss.

[0095] L total =ω s L smooth (I,L)+ω r L r (output,gt)

[0096] Among them, L smooth For the smoothing loss formula, I is the illumination map output by the Retinex semi-decoupled network, and L is the input low-light map; L r For the reconstruction loss formula L r =‖gt-output‖ 2 , where output is the enhanced image output by the network, and gt is the normal illumination map referenced in the dataset. ω s ω r For parameter weighting factors.

[0097]

[0098] Where p represents each pixel in the image, and c represents all channels of each pixel. δ x δ y It is the partial derivative in the horizontal and vertical directions in the image space. ∈ is set to 0.0001 to prevent the denominator from being zero, and θ is set to 1.2 based on experience.

[0099] Step F: Train the low-light image enhancement network RMAF-Net.

[0100] 5.1) Set J of normal illumination images t 'and low-light image set I t Divide the images into multiple paired groups according to their batch size, and then... t The first image group is input into the RMAF-Net network to obtain the initial weights W for each convolution operation of the network. n and bias value B n The weights D of the deconvolution operation n and the network-estimated normal illumination image group

[0101] 5.2) Will be with I t 'Paired J t 'First image group and network-estimated normal illumination image group' Substitute the values ​​into the joint total loss formula to calculate the total loss value Loss for each image in the training set;

[0102] 5.3) The gradient direction is updated using the network parameters of the Adam algorithm. The minimum joint total loss is used as the objective to update the parameters of the low-light image enhancement network, resulting in the neural network RMAF-Net after one parameter update.

[0103] 5.4) Input the second set of images into the neural network RMAF-Net after the first parameter update, and repeat steps 5.1) to 5.3) to obtain the neural network RMAF-Net after the second parameter update; and so on, until the last set of images is input into the neural network RMAF-Net after the previous update to obtain the neural network RMAF-Net after one training.

[0104] 5.5) Then J t '、I t All image groups are sequentially input into the neural network RMAF-Net after one training iteration. Steps 5.1) to 5.4) are repeated to obtain the neural network RMAF-Net after a second training iteration. This process is repeated until all image groups have been input 1000 times to complete the training of the low-light image enhancement network RMAF-Net.

[0105] Step G: Enhance the real low-light image.

[0106] A real low-light image I c The input is fed into the RMAF-Net low-light image augmentation network, which has completed all training, and the network outputs the augmented image J. c This completes the enhancement processing of low-light images.

[0107] The effects of this invention are further illustrated by the following simulations:

[0108] I. Test Conditions and Methods

[0109] 1. Test images: 15 real low-light images were selected;

[0110] 2. Testing methods: Eight methods were used, including the existing LIME (2016), RRM (2018), RetinexNet (2018), MBLLEN-Net (2018), SCI-Net (2022), Bread (2023), LANet (2023), and the present invention.

[0111] 3. Simulation test content:

[0112] Simulation Test 1: The above method was used to enhance four real low-light images respectively, and the results are as follows. Figure 6 As shown in the figure; the analysis of the results shown in the figure is as follows:

[0113] While existing traditional algorithms like LIME and RRM can improve the visibility of the original input, they cannot restore severely degraded colors to vividness. RRM, on the other hand, improves the overall naturalness of scene lighting and effectively suppresses imaging noise. However, it exhibits strong edge artifacts in darker areas, causing some blurring of image details.

[0114] The results using the existing Retinex-net have strong pseudo-colors, presenting unrealistic colors, and the BM3D algorithm is needed for further denoising.

[0115] Using existing SCI-Net results improves the overall scene lighting and preserves more information, but it also lacks effective noise suppression.

[0116] The results using the existing MBLLEN-Net and Bread algorithms effectively suppressed imaging noise and produced sharp edges, but the surrounding area had a hazy appearance and poor information recovery in darker areas.

[0117] Images processed using the existing LANet also suffer from blurred edges and loss of detail in darker areas.

[0118] It is evident that the image restored using the method of this invention outperforms other algorithms. The enhanced image exhibits natural colors, high saturation, clear local details, and excellent restoration of details in darker areas.

[0119] Simulation Test 2: The above method was used to process 15 sets of real low-light images, and the Peak Signal-to-Noise Ratio (PNSR), Structural Similarity Index (SSIM), and Learning Perceptual Image Patch Similarity Index (LPIPS) were calculated. Higher PNSR and SSIM values ​​indicate greater similarity between the two images; lower LPIPS values ​​indicate greater similarity, and vice versa. The results are shown in Table 1.

[0120] Table 1

[0121] index LIME RRM RetinexNet MBLLEN-Net SCI-Net Bread LANet This invention PSNR 15.24 13.88 16.77 17.858 14.78 22.96 21.71 22.948 SSIM 0.419 0.66 0.56 0.7270 0.525 0.838 0.81 0.842 LPIPS 0.2203 0.169 0.108 0.1526 0.339 0.151 0.101 0.074

[0122] As shown in Table 1, the PSNR, SSIM, and LPIPS values ​​of the method of the present invention are superior to those of other algorithms, consistent with subjective results, effectively improving image brightness and restoring image color.

[0123] This invention constructs a low-light image enhancement network based on Retinex multi-scale adaptive fusion. Illumination features are extracted in parallel by three branches on a low-resolution image. Then, an illumination map is estimated based on a bilateral grid, and reflectance components are further obtained, achieving decoupling of the low-light image in a fast and lightweight manner. Next, multi-level features are obtained through parallel decoupling at different scales, and further adaptively fused to preserve richer texture details and information in the enhancement result. This illumination estimation and multi-scale feature extraction fusion method makes the low-light image enhancement result more natural in brightness improvement and color restoration, while also restoring information in darker areas more completely. The enhancement result of this invention outperforms existing technologies in three objective metrics: peak signal-to-noise ratio, structural similarity, and learned perceptual image patch similarity, and can be used for low-light image enhancement processing.

[0124] The above simulation analysis proves the correctness and effectiveness of the method proposed in this invention.

[0125] The parts of this invention not described in detail are common knowledge to those skilled in the art.

[0126] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Obviously, those skilled in the art, after understanding the content and principle of the present invention, may make various modifications and changes in form and detail without departing from the principle and structure of the present invention. However, these modifications and changes based on the concept of the present invention are still within the scope of protection of the claims of the present invention.

Claims

1. A low-light image enhancement method based on Retinex multi-scale fusion, characterized in that, Includes the following steps: (1) Obtain the training image set: (1a) Normalize the initial image to limit its pixel values ​​to the range [0,1] to obtain a normalized image; (1b) The normalized image is downsampled by multiples of 1, 1 / 2 and 1 / 4 in sequence to obtain the sampled low-resolution image, which forms the training image set; (2) Constructing a low-light image enhancement network RMAF-Net based on Retinex multi-scale adaptive fusion within the PyTorch framework: (2a) Construct a low-resolution lighting prediction module: (2a1) Construct a channel self-attention module and six convolutional units. The first to fourth convolutional units are all convolutional layers, LeakyReLU activation layers and normalization layers connected in series, with padding values ​​of 0, kernel size of 3*3 and stride of 2. The fifth and sixth convolutional units are both convolutional layers and LeakyReLU activation layers connected in series, with padding values ​​of 1, kernel size of 3*3 and stride of 1. (2a2) Concatenate the first to fourth convolutional units, and then concatenate the channel self-attention module and the fifth convolutional unit as two parallel branches with the concatenated output through the channel dimension splicing operation to obtain the sixth convolutional unit, thus obtaining the low-resolution illumination prediction module. (2b) Constructing the Retinex semi-decoupled network: (2b1) The content feature enhancement module is composed of two residual units and a channel self-attention module cascaded together; (2b2) ​​After the content feature enhancement module and the low-resolution illumination prediction module, add bilateral grid upsampling, pixel-by-pixel division and pixel-by-pixel addition operations to obtain the Retinex semi-decoupled network; the input of this network is two images of different resolutions, which are fed into the content feature enhancement module and the low-resolution illumination prediction module respectively. (2c) Constructing the low-light image enhancement network RMAF-Net based on Retinex multi-scale adaptive fusion: (2c1) An adaptive cross-fusion module is constructed using nine convolutional units, pixel-wise addition, spatial dimension average pooling, channel dimension max pooling, and pixel-wise multiplication. Each of the nine convolutional units is a concatenated structure of convolutional layers and LeakyReLU activation layers, with a kernel size of 3*3 and a stride of 1. Except for the fourth convolutional unit, which has a padding value of 0, all other convolutional units have a padding value of 1. The first and second convolutional units are used to extract features from two different scales of image information input to the module, and then concatenated with pixel-wise addition and the third convolutional unit. The concatenated result is processed in parallel through two branches in the spatial and channel dimensions, and then subjected to matrix multiplication. Softmax operation is used to achieve branch interaction. Each branch undergoes parallel pixel-wise multiplication, and the parallel result is concatenated with the pixel-wise addition operation as the output of the adaptive cross-fusion module. (2c2) The low-light image enhancement network RMAF-Net, which is composed of two Retinex semi-decoupled networks that process in parallel at different scales and an adaptive cross-fusion module, consists of two Retinex semi-decoupled networks that process in parallel at different scales and an adaptive cross-fusion module. (3) Construct a semi-decoupled smooth loss L smooth and reconstruction losses L r The formula for the combined loss is: L total =ω s L smooth (I,L)+ω r L r (output,g t ) Where I is the illumination map output by the Retinex semi-decoupled network, L is the input low-light map; output is the enhanced image output by the network, and g t For the normal illumination map referenced in the dataset, ω s ω r These are the semi-decoupling smoothing weight factor and the reconstruction weight factor; (4) Training the low-light image enhancement network RMAF-Net: The training set is input into RMAF-Net, and the network parameters are iteratively updated using the Adam optimization algorithm until the loss function converges, thus obtaining the trained low-light image enhancement network. (5) Enhance realistic low-light images: Real low-light images are input into the trained network to perform low-light image enhancement, and the enhanced image is output.

2. The method according to claim 1, characterized in that: The channel self-attention module described in step (2a1) consists of three parallel branches: the first branch is composed of convolutional units, reshape operations, and matrix transpose operations in series; the second branch is composed of convolutional units and reshape operations; and the third branch contains only convolutional units. The convolutional units in each branch are composed of convolutional layers and LeakyReLU activation layers, and the kernel size is 1*1 and the stride is 1.

3. The method according to claim 2, characterized in that: The channel self-attention module multiplies the output R1 from the first branch with the output R2 from the second branch to obtain matrix E(X), then multiplies the output R3 from the third branch with E(X) to obtain D(X), then obtains Y(X) through a reshape operation, and finally uses Y(X) as the module output.

4. The method according to claim 1, characterized in that: In step (2a2), the low-resolution illumination prediction module is described, wherein the first to fourth convolutional units are concatenated to obtain R0, and the two parallel branches of the channel self-attention module and the fifth convolutional unit obtain R1 and R2 respectively. The three parallel results of R0, R1 and R2 are concatenated with the sixth convolutional unit through the channel dimension concatenation operation, and the output of the concatenation is used as the module output.

5. The method according to claim 1, characterized in that: The two residual units in step (2b1) are the same. Each residual unit consists of three identical convolutional units with a structure of a convolutional layer and a LeakyRelu activation layer connected in series, a padding value of 1, a kernel size of 3*3, and a stride of 1.

6. The method according to claim 5, characterized in that: The residual unit takes its input feature I and passes it through the first to third convolutional units to obtain R. Then, it adds I and R to obtain R'. Finally, it is connected in series with the LeakyRelu activation layer to obtain the output of the residual unit, which is the output result of the residual unit.

7. The method according to claim 1, characterized in that: In step (2b2), the Retinex semi-decoupled network takes two images, I1 and I2, with different resolutions as input. I1 is processed by the feature enhancement module to obtain I1', and I2 is processed by the low-resolution illumination prediction module to obtain I2'. Then, after bilateral grid upsampling, pixel-by-pixel division and pixel-by-pixel addition operations, the output result of the Retinex semi-decoupled network is obtained.

8. The method according to claim 1, characterized in that: The two branches mentioned in step (2c1) are as follows: the first branch is a spatial dimension average pooling operation → fourth convolutional unit → parallel fifth and sixth convolutional units; the second branch is a channel dimension maximum pooling operation → seventh convolutional unit → sigmoid activation operation → parallel eighth and ninth convolutional units; the branch interaction is achieved by performing matrix multiplication on the outputs of the fifth and eighth convolutional units and the outputs of the sixth and ninth convolutional units respectively, followed by a Softmax operation for calibration to obtain their respective results.

9. The method according to claim 1, characterized in that: The low-light image enhancement network RMAF-Net based on Retinex multi-scale adaptive fusion described in step (2c2) takes three images I0, I1 and I2 of different resolutions as input. Images I0 and I1 are processed by a first Retinex semi-decoupled network to obtain S1, and images I1 and I2 are processed by a second Retinex semi-decoupled network to obtain S2. The parallel results S1 and S2 are processed by an adaptive cross-fusion module to obtain the output result, which is the final result of the low-light image enhancement network RMAF-Net.

10. The method according to claim 1, characterized in that: The semi-decoupling smoothing loss L mentioned in step (3) smooth and reconstruction losses L r The expression is as follows: L r =||gt-output|| 2 Among them, I p This represents the p-th pixel in the illumination map output by the Retinex semi-decoupled network, where c represents all channels of each pixel. δ x δ y Let L represent the partial derivatives in the horizontal and vertical directions in the image space, respectively. p Let ∈ be the p-th pixel in the low-light image, where ∈ is the smoothing coefficient and θ is the sensitivity coefficient.