Low-light image enhancement method, system and device and storage medium
Through the Retinex decomposition network and multimodal fusion network combined with multi-scale illumination estimation, the problem of structural information loss and inaccurate brightness recovery in low-light enhancement technology is solved, and the image structure details are restored and visual effects are improved.
Patent Information
- Application Number
- CN202510425647.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-07
- Publication Date
- 2025-07-25
AI Technical Summary
The problems of loss of structure information and inaccurate brightness recovery during image recovery in existing low-light enhancement technologies.
Retinex decomposition network, adaptive multimodal fusion network, multi-scale illumination estimation network, feature selection network and reflection component enhancement network are used to enhance image structure and visual perception through multi-modal images (infrared and depth maps), and combine multi-scale illumination estimation strategy.
Effectively restore image structural details, provide better visual effects, and improve image brightness while reducing the problems of structural information loss and inaccurate brightness recovery.
Smart Images
Figure CN120374475A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and particularly to a low-light image enhancement method, system, device, and storage medium. Background Art
[0002] With the rapid development and application of computer science and technology, images have become an important carrier for obtaining information. However, during the image acquisition process, images are often affected by insufficient lighting conditions, resulting in degradation problems such as low contrast, color distortion, and noise, leading to a decrease in image quality and an inability to display clear scene and structural details of the image. This not only reduces the visual effect of the image but also greatly affects computer vision tasks such as object detection and scene understanding.
[0003] To solve these problems, traditional low-light image enhancement techniques mainly use two methods: histogram equalization and the Retinex (retina and cortex) theory. However, most traditional methods rely on manually designed priors, which can bring other problems such as noise amplification and color distortion while enhancing the image brightness.
[0004] With the development of deep learning, deep learning-based methods have been applied to computer vision tasks. Currently, most methods are combined with the Retinex theory and focus on using a denoising network to restore the structural clarity of the image in the reflection component containing structural information. However, the problem of structural loss during the denoising process is ignored. Moreover, the amount of information in a single low-light image is small, which undoubtedly increases the difficulty of image structure restoration. Some methods have introduced multi-modal images into the image enhancement process, such as by introducing edge maps, semantic segmentation maps, infrared maps, etc. to guide image restoration. However, these methods usually rely only on single-modal images for guidance. Single-modal images can only provide information from a limited perspective, ignoring the complementarity and richer information that multi-modal information can provide. In addition, the lighting problems in images are usually not evenly distributed. Some methods estimate the lighting from a single scale, and in areas with uneven lighting, local features may be ignored, resulting in inaccurate lighting estimation. Therefore, by estimating the lighting at multiple scales, these different-scale lighting problems can be effectively captured, and at the same time, the error in lighting estimation caused by problems such as noise can be reduced.
[0005] In summary, in order to better enhance the structural information of the image, give full play to the role of multi-modal images in image restoration, and address the accuracy problem of uneven lighting estimation, a method for enhancing the structure and visual perception of the image using multi-modal images (infrared maps and depth maps) and a multi-scale lighting estimation strategy are proposed. Summary of the Invention
[0006] The present invention solves the problems of loss of structural information and inaccurate brightness restoration in the image restoration process of existing low-light enhancement technologies.
[0007] A low-light image enhancement method according to the present invention includes the following steps:
[0008] Step S1: Obtain a training set and a test set respectively. Both the training set and the test set include low-light images and corresponding normal-light images.
[0009] Step S2: Use the multi-modal pre-training model for the low-light images corresponding to the training set to generate a low-light multi-modal data set corresponding to the low-light images in the training set. The low-light multi-modal data set includes infrared images and depth images.
[0010] Step S3: Use the adaptive multi-modal fusion network for the low-light images, infrared images, and depth images corresponding to the training set to obtain adaptive multi-modal fusion features.
[0011] Step S4: Input the low-light images in the training set into the Retinex decomposition network to obtain the illumination component and the reflection component corresponding to the low-light images in the training set respectively. Input the adaptive multi-modal fusion features into the Retinex decomposition network for preliminary screening.
[0012] Step S5: Enhance the illumination component corresponding to the low-light images in the training set through the multi-scale illumination estimation network model to obtain the enhanced illumination component corresponding to the low-light images in the training set.
[0013] Step S6: Input the reflection component corresponding to the low-light images in the training set and the adaptively multi-modal fusion features after preliminary screening into the feature selection network for selection and adjustment in sequence to obtain the reflection component features and the multi-modal features respectively.
[0014] Step S7: Enhance the respectively obtained reflection component features and multi-modal features through the reflection component enhancement network model to obtain the enhanced reflection component corresponding to the low-light images in the training set.
[0015] Step S8: Multiply the enhanced illumination component corresponding to the low-light images in the training set and the enhanced reflection component corresponding to the low-light images in the training set element by element to obtain the enhanced image.
[0016] Step S9: Use the loss function to train the Retinex decomposition network, the adaptive multi-modal fusion network, the multi-scale illumination estimation network model, the feature selection network, and the reflection component enhancement network model respectively.
[0017] Step S10: Perform the operations of steps S2 to S8 on the low-light images in the test set and their corresponding normal-light images to obtain the enhanced low-light images.
[0018] Further, in an embodiment of the present invention, in step S4 or step S9, the Retinex decomposition network is specifically:
[0019] Multiple convolutional layers are connected in series, activation functions are interspersed between the multiple convolutional layers, residual connections are used between each convolutional layer to obtain 4-channel features, 1-channel and 3-channel features are divided along the channels, and the illumination component and reflection component corresponding to the low-light images in the training set are obtained using activation functions.
[0020] Further, in an embodiment of the present invention, in step S3, the adaptive multi-modal fusion network is specifically:
[0021] After the low-light image and the infrared image are respectively subjected to feature extraction through convolutional layers, the feature-extracted low-light image and the feature-extracted infrared image are concatenated along the channels. The concatenated feature-extracted low-light image and the feature-extracted infrared image are successively subjected to the operations of convolutional layers and activation functions to obtain a guidance map. The guidance map is multiplied element-wise with the low-light image to modulate the low-light features, and then, in a manner of using convolutional layers and element-wise addition, the modulated low-light features and the feature-extracted infrared image are feature-fused;
[0022] The low-light image and the depth image are subjected to the same operations, and finally the two fused features obtained are added together.
[0023] Further, in an embodiment of the present invention, in step S5, the multi-scale illumination estimation network model is specifically:
[0024] The illumination component L corresponding to the low-light images in the training set is used to obtain an initial illumination estimation map through 1 - L. The initial illumination estimation map is successively subjected to two interpolations, convolutional layers, SS2D, adaptive pooling, and activation operations to obtain brightness attention weights containing different scale information. At the same time, the illumination component L corresponding to the low-light images in the training set is successively subjected to two interpolations and convolutional layer operations to output the feature-extracted illumination component L. The feature-extracted illumination component L is multiplied element-wise with the brightness attention weights containing different scale information to obtain multiple adjusted brightness maps. After the multiple adjusted brightness maps and the feature-extracted illumination component L are added and fused together, the enhanced illumination component corresponding to the low-light images in the training set is obtained.
[0025] Further, in an embodiment of the present invention, in step S6, the feature selection network is specifically:
[0026] The reflection component corresponding to the low-light image in the training set and the adaptively multi-modal fusion features after preliminary screening are respectively operated on by a convolutional layer, and the reflection component feature map after preliminary feature extraction and the multi-modal feature map after preliminary feature extraction are respectively output. After splicing the reflection component feature map after preliminary feature extraction and the multi-modal feature map after preliminary feature extraction, operations of deformable convolution and activation function are sequentially performed, and the reflection component feature after feature extraction and the multi-modal feature after feature extraction are respectively output. The reflection component feature after feature extraction and the reflection component corresponding to the low-light image in the training set are sequentially multiplied and added to obtain the reflection component feature, and the multi-modal feature after feature extraction and the adaptively multi-modal fusion features after preliminary screening are sequentially multiplied and added to obtain the multi-modal feature.
[0027] Further, in an embodiment of the present invention, in step S7, the reflection component enhancement network model is specifically:
[0028] The reflection component feature and the multi-modal feature are two parallel branches, which are respectively sequentially operated on by a normalization layer, a convolutional layer, channel attention, and SS2D, and the reflection component feature after global feature extraction and the multi-modal feature after global feature extraction are respectively output. After performing operations of feature exchange and splicing on the reflection component feature after global feature extraction and the multi-modal feature after global feature extraction, depth feature extraction is performed by depthwise separable convolution. Finally, operations of splicing, depthwise separable convolution, and activation function are performed to obtain the enhanced reflection component corresponding to the low-light image in the training set.
[0029] Further, in an embodiment of the present invention, in step S9, the loss function is specifically:
[0030] L = λ1L decom + λ2L R + λ3L light + λ4L recon + λ5L ssim + λ6L vgg ;
[0031] where L is the loss function, λ1, λ2, λ3, λ4, λ5, and λ6 are all loss weights, and L decom is the Retinex decomposition loss between the current low-light image and the normal-light image, L R is the l1 loss between the enhanced reflection map and the reflection map and multi-modal information of the normal light, L light is the l1 loss between the enhanced illumination map and the illumination map of the normal light, L recon is the l1 loss between the finally output map and the normal-light image, L ssim is the structural similarity loss, L vggIt is the VGG loss.
[0032] A low-light image enhancement system according to the present invention includes the following modules:
[0033] Module S1, which respectively obtains a training set and a test set. Both the training set and the test set include low-light images and their corresponding normal-light images;
[0034] Module S2, using the low-light images corresponding to the training set to generate a low-light multimodal dataset corresponding to the low-light images in the training set by means of a multimodal pre-trained model. The low-light multimodal dataset includes infrared images and depth maps;
[0035] Module S3, using the low-light images, infrared images, and depth maps corresponding to the training set to obtain adaptive multimodal fusion features by means of an adaptive multimodal fusion network;
[0036] Module S4, inputting the low-light images in the training set into a Retinex decomposition network to respectively obtain an illumination component and a reflection component corresponding to the low-light images in the training set, and inputting the adaptive multimodal fusion features into the Retinex decomposition network for preliminary screening;
[0037] Module S5, enhancing the illumination component corresponding to the low-light images in the training set through a multi-scale illumination estimation network model to obtain an enhanced illumination component corresponding to the low-light images in the training set;
[0038] Module S6, inputting the reflection component corresponding to the low-light images in the training set and the adaptively multimodal fusion features after preliminary screening into a feature selection network for selection and adjustment in sequence to respectively obtain reflection component features and multimodal features;
[0039] Module S7, enhancing the respectively obtained reflection component features and multimodal features through a reflection component enhancement network model to obtain an enhanced reflection component corresponding to the low-light images in the training set;
[0040] Module S8, multiplying the enhanced illumination component corresponding to the low-light images in the training set and the enhanced reflection component corresponding to the low-light images in the training set element by element to obtain an enhanced image;
[0041] Module S9, using a loss function to train the Retinex decomposition network, the adaptive multimodal fusion network, the multi-scale illumination estimation network model, the feature selection network, and the reflection component enhancement network model respectively;
[0042] Module S10, performing the operations of Module S2 to Module S8 on the low-light images in the test set and their corresponding normal-light images to obtain enhanced low-light images.
[0043] An electronic device according to the present invention includes a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus;
[0044] The memory is used to store computer programs;
[0045] The processor is used to implement the method steps described in any of the above methods when executing the programs stored on the memory.
[0046] A computer-readable storage medium according to the present invention stores a computer program therein, and when the computer program is executed by a processor, it implements the method steps described in any of the above methods.
[0047] The present invention solves the problems of loss of structural information and inaccurate brightness restoration in the image restoration process of existing low-light enhancement technologies. The specific beneficial effects include:
[0048] 1. A low-light image enhancement method according to the present invention, which solves the problems of loss of structural information and inaccurate brightness restoration in the image restoration process of existing low-light enhancement technologies. To solve the above technical problems, the present invention designs a Retinex decomposition network model, an adaptive multi-modal fusion network model, a multi-scale illumination adjustment network model, a feature selection network model, and a reflection component enhancement network model. The Retinex decomposition network model decomposes a low-light image into a reflection map and an illumination map. The adaptive multi-modal fusion network model can adaptively adjust the image features of different modalities, fuse multi-modal information together, and obtain multi-modal features for helping the restoration of the reflection component. The multi-scale illumination adjustment network model estimates the illumination weight according to multi-scale information, and it can accurately enhance the brightness of the image. The feature selection network model is used to select more important multi-modal features and reflection component features. The reflection component enhancement network model uses the structural information from the infrared map and the depth information from the depth map in the multi-modal features to help the reflection component achieve better detail restoration and visual effects. Multiply the enhanced illumination image and the reflection image element by element to obtain the enhanced image, and this method effectively solves the problems existing in the existing low-light image enhancement technologies;
[0049] A low-light image enhancement method according to the present invention performs experiments on an image data set under low-light conditions and conducts comparative analysis with the prior art. The experimental data confirm that the present invention can effectively restore the structural details of the image while enhancing the image brightness, providing a better visual effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] The above and / or additional aspects and advantages of the present invention will become apparent and be readily understood from the following description of embodiments in conjunction with the accompanying drawings, where:
[0051] Figure 1 is the overall framework diagram of the low-light image enhancement method described in Embodiment 1;
[0052] Figure 2 is the adaptive multi-modal fusion network diagram described in Embodiment 1;
[0053] Figure 3 is the Retinex decomposition network diagram described in Embodiment 1;
[0054] Figure 4 is the multi-scale illumination adjustment network structure diagram described in Embodiment 1;
[0055] Figure 5 is the feature selection network structure diagram described in Embodiment 1;
[0056] Figure 6 is the reflection component enhancement network structure diagram described in Embodiment 1. Detailed Embodiments
[0057] The following will clearly and completely describe various embodiments of the present invention in conjunction with the accompanying drawings. The embodiments described by referring to the accompanying drawings are exemplary and are intended to explain the present invention, and should not be construed as limiting the present invention.
[0058] Embodiment 1. A low-light image enhancement method described in this embodiment includes the following steps:
[0059] Step S1, respectively obtain a training set and a test set, and both the training set and the test set include low-light images and corresponding normal-light images;
[0060] Step S2, use the multi-modal pre-training model for the low-light images corresponding to the training set to generate a low-light multi-modal data set corresponding to the low-light images in the training set, and the low-light multi-modal data set includes infrared images and depth images;
[0061] Step S3, use the adaptive multi-modal fusion network for the low-light images, infrared images, and depth images corresponding to the training set to obtain adaptive multi-modal fusion features;
[0062] Step S4, input the low-light images in the training set into the Retinex decomposition network to respectively obtain the illumination component and the reflection component corresponding to the low-light images in the training set, and input the adaptive multi-modal fusion features into the Retinex decomposition network for preliminary screening;
[0063] Step S5: The illumination component corresponding to the low-light image in the training set is enhanced through the multi-scale illumination estimation network model to obtain the enhanced illumination component corresponding to the low-light image in the training set.
[0064] Step S6: The reflection component corresponding to the low-light image in the training set and the adaptively multi-modal fusion features after preliminary screening are input into the feature selection network for selection and adjustment in sequence to obtain the reflection component features and multi-modal features respectively.
[0065] Step S7: The respectively obtained reflection component features and multi-modal features are enhanced through the reflection component enhancement network model to obtain the enhanced reflection component corresponding to the low-light image in the training set.
[0066] Step S8: The enhanced illumination component corresponding to the low-light image in the training set and the enhanced reflection component corresponding to the low-light image in the training set are multiplied element by element to obtain the enhanced image.
[0067] Step S9: The Retinex decomposition network, adaptively multi-modal fusion network, multi-scale illumination estimation network model, feature selection network, and reflection component enhancement network model are trained respectively using the loss function.
[0068] Step S10: The low-light image in the test set and the corresponding normal-light image perform the operations of Step S2 to Step S8 to obtain the enhanced low-light image.
[0069] In this embodiment, in Step S4 or Step S9, the Retinex decomposition network is specifically:
[0070] Multiple convolutional layers are connected in series, activation functions are interspersed between multiple convolutional layers, residual connections are used between each convolutional layer to obtain 4-channel features, 1-channel and 3-channel features are divided along the channels, and the illumination component and reflection component corresponding to the low-light image in the training set are obtained using the activation function.
[0071] In this embodiment, in Step S3, the adaptively multi-modal fusion network is specifically:
[0072] After the low-light image and the infrared image are respectively subjected to feature extraction through convolutional layers, the feature-extracted low-light image and the feature-extracted infrared image are concatenated along the channels. The concatenated feature-extracted low-light image and the feature-extracted infrared image are sequentially passed through the operations of convolutional layers and activation functions to obtain a guidance map. The guidance map is multiplied element by element with the low-light image to modulate the low-light features, and then the modulated low-light features and the feature-extracted infrared image are feature-fused using the convolutional layer and element-by-element addition method.
[0073] The low-light image and the depth map undergo the same operations, and finally, the two fused features obtained are added together.
[0074] In this embodiment, in step S5, the multi-scale illumination estimation network model is specifically as follows:
[0075] The illumination component L corresponding to the low-light image in the training set is used to obtain an initial illumination estimation map through 1 - L. The initial illumination estimation map undergoes operations of two interpolations, a convolutional layer, SS2D, adaptive pooling, and activation in sequence to obtain luminance attention weights containing different scale information. At the same time, the illumination component L corresponding to the low-light image in the training set undergoes operations of two interpolations and a convolutional layer in sequence, and the illumination component L after feature extraction is output. The illumination component L after feature extraction is multiplied element-wise with the luminance attention weights containing different scale information to obtain multiple adjusted luminance maps. After adding and fusing the multiple adjusted luminance maps with the illumination component L after feature extraction, the enhanced illumination component corresponding to the low-light image in the training set is obtained.
[0076] In this embodiment, in step S6, the feature selection network is specifically as follows:
[0077] The reflection component corresponding to the low-light image in the training set and the preliminarily screened adaptive multi-modal fusion feature respectively undergo operations of a convolutional layer, and the reflection component feature map after preliminary feature extraction and the multi-modal feature map after preliminary feature extraction are respectively output. After splicing the reflection component feature map after preliminary feature extraction and the multi-modal feature map after preliminary feature extraction, operations of deformable convolution and an activation function are performed in sequence, and the reflection component feature after feature extraction and the multi-modal feature after feature extraction are respectively output. The reflection component feature after feature extraction and the reflection component corresponding to the low-light image in the training set undergo multiplication and addition operations in sequence to obtain the reflection component feature. The multi-modal feature after feature extraction and the preliminarily screened adaptive multi-modal fusion feature undergo multiplication and addition operations in sequence to obtain the multi-modal feature.
[0078] In this embodiment, in step S7, the reflection component enhancement network model is specifically as follows:
[0079] The reflection component feature and the multi-modal feature are two parallel branches, which respectively undergo operations of a normalization layer, a convolutional layer, channel attention, and SS2D in sequence, and the reflection component feature after global feature extraction and the multi-modal feature after global feature extraction are respectively output. After performing operations of feature exchange and splicing on the reflection component feature after global feature extraction and the multi-modal feature after global feature extraction, depth feature extraction is performed through a depthwise separable convolution. Finally, operations of splicing, depthwise separable convolution, and an activation function are performed to obtain the enhanced reflection component corresponding to the low-light image in the training set.
[0080] In this embodiment, in step S9, the loss function is specifically as follows:
[0081] L = λ1L decom + λ2L R + λ3L light + λ4L recon + λ5L ssim + λ6L vgg ;
[0082] where L is the loss function, λ1, λ2, λ3, λ4, λ5, and λ6 are all loss weights, L decom is the Retinex decomposition loss between the current low-light image and the normal-light image, L R is the l1 loss between the enhanced reflection map and the reflection map and multi-modal information of the normal light, L light is the l1 loss between the enhanced illumination map and the illumination map of the normal light, L recon is the l1 loss between the finally output image and the normal-light image, L ssim is the structural similarity loss, L vgg is the VGG loss.
[0083] In the prior art, the low-light image enhancement technology combines with the Retinex theory, uses convolution to decompose the image into a reflection component and an illumination component, and restores them separately. However, in the process of restoring the reflection component, there is a problem that the denoising network causes the loss of the structural information of the image, and the global estimation of the illumination is not accurate enough when restoring the brightness.
[0084] To solve the above technical problems, as Figure 1 shown, this embodiment designs a low-light image enhancement method based on the Retinex theory, using multi-modal images and multi-scale strategies, including the following steps:
[0085] Step S1, obtain a paired data set, the data set includes a training set and a test set, the training set includes a low-light image I low and a normal-light image I low paired with the low-light image I high , the test set includes a low-light image I' low and a normal-light image I' low paired with the low-light image I' high ;
[0086] Step S2, for the low-light image I in the training set low use the multi-modal pre-training model to generate a low-light multi-modal data set corresponding to the low-light image I in the training set low The low-light multi-modal data set includes an infrared image Ifr and depth map I depth ;
[0087] As Figure 1 shown, the multi-modal pre-training network model is an infrared pre-training network and a depth map pre-training network, which are directly obtained from existing advanced methods for infrared map generation and depth map generation.
[0088] Step S3, after the infrared map I fr , depth map I depth and the low-light image I in the training set low are fused through the adaptive multi-modal fusion network, the adaptive multi-modal fusion feature F m is obtained;
[0089] As Figure 2 shown, the infrared map I fr , depth map I depth and the low-light image I in the training set low are respectively subjected to feature extraction through ConvB (convolution block) (consisting of three stacked Conv, BatchNorm, and ReLU convolutional layers). After the features of the extracted infrared map and depth map are respectively concatenated with the low-light image feature Concat, they are fused with weights using Conv and Sigmoid. The low-light map is used as a global adjustment factor to guide the fusion process, and F′ fr , F′ depth are respectively obtained. Finally, the two fusion features are combined by addition to obtain the output adaptive multi-modal fusion feature F m .
[0090] Step S4, the low-light image I low is passed through the Retinex-based decomposition network to respectively obtain the illumination component L low and the reflection component R low corresponding to the low-light image in the training set. At the same time, the adaptive multi-modal fusion feature F m is subjected to preliminary feature screening through the same decomposition network with shared weights, and the part F r for restoring the reflection component of the low-light image is extracted;
[0091] As Figure 3 shown, the decomposition network contains a Retinex decomposition network model including multiple 3×3 convolutional layers in series, with ReLU activation functions interspersed between the convolutions. Residual connections are used between each convolutional layer. The last convolution outputs 4-channel features, which are divided into two features of 1 channel and 3 channels along the channels. Finally, the reflection component and the illumination component are obtained using the Sigmoid activation function. At the same time, according to the Retinex decomposition theory, the normal light image I corresponding to the low-light image highThrough the decomposition network with the same parameter sharing, the illumination component L is obtained. high and the reflection component R high . Therefore, the corresponding decomposition loss function L can be calculated decom , and the decomposition loss function is defined as:
[0092]
[0093] where λ is the weight, is the gradient, and the last term is used to constrain the illumination smoothness. The purpose of this loss function is to ensure that the decomposition network can correctly decompose the reflection map and the illumination map.
[0094] Step S5: Construct a multi-scale illumination estimation network model, and input the illumination image L decomposed from the low-light image low into the multi-scale illumination adjustment network model for brightness adjustment to obtain the enhanced illumination image L out ;
[0095] As Figure 4 shown, first, the low-light illumination image L low is passed through 1 - L low to obtain the initial illumination estimation map, and then this map is interpolated twice to obtain illumination estimation maps of different scales. Features are extracted from different scales using convolutional blocks and SS2D to obtain L att1 , L att2 , L att3 , and then through adaptive average pooling and Sigmoid to obtain more accurate brightness attention weights w1, w2, w3. The low-light illumination image L low is also interpolated twice, and after extracting features using convolutional blocks, the features containing different scale information are made the same size. The three features L1, L2, L3 obtained are multiplied element-wise with the brightness weights w1, w2, w3 and residually connected to obtain L en1 , L en2 , L en3 . The L en1 , L en2 , L en3 estimated through different scale features are finally fused by sum to finally obtain the illumination image L of the enhanced low-light image out ;
[0096] Based on the illumination image L of the enhanced low-light image out and the illumination image L of the normal-light image high the brightness loss L light (L1 loss function) is calculated, and it is defined as:
[0097] L light = λ||L out-L high ||1。
[0098] Step S6, the reflection component R of the low-light image low and the preliminarily screened multi-modal information F obtained in step S4 r are further selected and adjusted through the feature selection network to obtain more critical reflection component features R aj and multi-modal features F aj ;
[0099] As Figure 5 shown, R low , F r are denoted as R′ low , F′ r after downsampling. R′ low , F′ r are adjusted in number of channels through 1×1 convolution, and then R low , F r are Concat spliced and fed into the deformable convolution DeformConv to further extract depth features. The weights are calculated using Sigmoid to adjust R′ low , F′ r . The enhanced results R low , F′ r of the adjusted R aj , F aj are obtained;
[0100] Step S7, the reflection component features R aj and multi-modal features F aj obtained in step S6 are enhanced through the reflection component enhancement network model to obtain the enhanced reflection component R of the low-light image out ;
[0101] As Figure 6 shown, the reflection component enhancement module normalizes the adjusted R aj , F aj through the normalization layer LayerNorm respectively, then adjusts the number of channels through Conv, inputs to the channel attention (CA) to focus on key information at the channel level, uses SS2D to extract features, improves efficiency while realizing in-depth extraction of global features. Then, the two branch features are Concat complemented to each other. After the two features are further extracted for depth features through the separable convolution DConv, they are Concat again and fused into one branch through the depthwise separable convolution DConv. Then, it is divided into two branches along the channel, and one branch is multiplied by the other branch after passing through the gelu non-linear activation function to non-linearly enhance the features and improve the expression ability of the features. Finally, as Figure 1As shown, the enhanced reflection component R is obtained after upsampling and restoring to the original image size. out .
[0102] The reflected image R of the low-light image enhanced by the reflected component enhancement network model out is combined with the reflected image R of the normal-light image high and the multi-modal fusion image to calculate the reflection reconstruction loss L through the L1 loss function (mean absolute error, MAE), R defined as:
[0103] L R = λ h ||R out - R high ||1 + λ m ‖R out - F m ‖1.
[0104] Step S8: Multiply the illumination component L of the enhanced low-light image out and the reflection component R of the low-light image element by element to obtain the enhanced image I out ; out ;
[0105] Step S9: Design a loss function based on the enhanced image I out and the normal-light image I paired with the low-light image high and use the loss function to update the parameters of the Retinex decomposition network model, the adaptive multi-modal fusion network model, the multi-scale illumination adjustment network model, the feature selection network model, and the reflected component enhancement network model through the backpropagation algorithm;
[0106] To ensure pixel-level accuracy, the L1 loss function (mean absolute error, MAE) and the structural similarity loss are used to make the structural consistency between the network-predicted image and the real normal-light image, denoted as L recon , L ssim . In addition, a perceptual loss based on the VGG19 feature representation is introduced to ensure high-level semantic details and visual fidelity, denoted as L vgg .
[0107] Therefore, the total loss L is expressed as:
[0108] L = λ1L decom + λ2L R + λ3L light + λ4L recon + λ5L ssim + λ6L vgg ;
[0109] Among them, λ1, λ2, λ3, λ4, λ5, and λ6 are all weights.
[0110] Step S10: Set the number of training iterations, save the parameters of the network during training. When the specified number of network iterations is reached, stop the network training, and save the parameters of the trained and optimized network model in a file.
[0111] Step S11: Load the trained model, and input the low-light image I′ low in the test set and its corresponding normal-light image I′ high , as well as the multi-modal images I′ fr , I′ depth obtained through multi-modal and training network, into the entire trained network to output the final enhanced image.
[0112] To prove the effectiveness of the proposed method, the experimental results of this embodiment are compared with existing excellent low-light image enhancement methods. As shown in Table 1, the performances on three datasets, namely LOL-v2-Real, LOL-v2-Syn, and LSRW-Huawei, are compared respectively. Among them, on the LOL-v2-Real dataset, the PSNR (the higher the value, the better), SSIM (the higher the value, the better), and LPIPS (the lower the value, the better) metrics all achieved the best results; on the LOL-v2-Syn dataset, PSNR achieved the second-best result, and SSIM and LPIPS achieved the best results; on the LSRW-Huawei dataset, PSNR and SSIM achieved the best results, and LPIPS achieved the second-best result.
[0113] Table 1
[0114]
[0115] In summary, in this embodiment, the low-light image is decomposed into a reflection map and an illumination map through the Retinex decomposition network. The multi-modal images (infrared map, depth map, low-light map) are fused through the adaptive multi-modal fusion network to help the reflection component restore clear structure and visual effects. Through the multi-scale illumination adjustment network model, which estimates the weights containing global and local information, different levels of features are combined to obtain more accurate brightness attention weights to help restore the image brightness. Through the feature selection network model, more important reflection component features and multi-modal features are focused on and selected. Through the reflection component enhancement network, the obtained multi-modal fusion features and the reflection component are mutually enhanced and fused to help the reflection component restore clear structure and visual effects. Thus, the problems of structure loss during the restoration of the reflection component and inaccurate complex illumination adjustment in the existing low-light image enhancement technology based on the Retinex theory are solved.
[0116] Embodiment 2. A low-light image enhancement system described in this embodiment includes the following modules:
[0117] Module S1, which obtains a training set and a test set respectively. Both the training set and the test set include low-light images and their corresponding normal-light images;
[0118] Module S2, using the low-light images corresponding to the training set to generate a low-light multi-modal dataset corresponding to the low-light images in the training set by a multi-modal pre-training model. The low-light multi-modal dataset includes infrared images and depth maps;
[0119] Module S3, using the low-light images, infrared images, and depth maps corresponding to the training set to obtain adaptive multi-modal fusion features by an adaptive multi-modal fusion network;
[0120] Module S4, inputting the low-light images in the training set into a Retinex decomposition network to obtain the illumination component and the reflection component corresponding to the low-light images in the training set respectively, and inputting the adaptive multi-modal fusion features into the Retinex decomposition network for preliminary screening;
[0121] Module S5, enhancing the illumination component corresponding to the low-light images in the training set through a multi-scale illumination estimation network model to obtain the enhanced illumination component corresponding to the low-light images in the training set;
[0122] Module S6, inputting the reflection component corresponding to the low-light images in the training set and the adaptively multi-modal fusion features after preliminary screening into a feature selection network for selection and adjustment in sequence to obtain the reflection component features and multi-modal features respectively;
[0123] Module S7, enhancing the respectively obtained reflection component features and multi-modal features through a reflection component enhancement network model to obtain the enhanced reflection component corresponding to the low-light images in the training set;
[0124] Module S8, multiplying the enhanced illumination component corresponding to the low-light images in the training set and the enhanced reflection component corresponding to the low-light images in the training set element by element to obtain the enhanced image;
[0125] Module S9, training the Retinex decomposition network, the adaptive multi-modal fusion network, the multi-scale illumination estimation network model, the feature selection network, and the reflection component enhancement network model respectively using a loss function;
[0126] Module S10, performing the operations of Module S2 to Module S8 on the low-light images in the test set and their corresponding normal-light images to obtain the enhanced low-light images.
[0127] Embodiment 3. An electronic device described in this embodiment includes a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus;
[0128] The memory is used to store a computer program;
[0129] The processor is configured to implement the method steps described in Embodiment 1 when executing the program stored on the memory.
[0130] Embodiment 4. A computer-readable storage medium described in this embodiment stores a computer program therein, and when the computer program is executed by a processor, the method steps described in Embodiment 1 are implemented.
[0131] The above has introduced in detail a low-light image enhancement method, system, device, and storage medium proposed by the present invention. Specific examples are used in this article to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.
Claims
1. A low-light image enhancement method, characterized in that Including the following steps: Step S1: Obtain a training set and a test set respectively. Both the training set and the test set include low-light images and their corresponding normal-light images. Step S2: Use the multi-modal pre-trained model to generate a low-light multi-modal data set corresponding to the low-light images in the training set from the low-light images in the training set. The low-light multi-modal data set includes infrared images and depth maps. Step S3: Use the adaptive multi-modal fusion network for the low-light images, infrared images, and depth maps in the training set to obtain adaptive multi-modal fusion features. Step S4: Input the low-light images in the training set into the Retinex decomposition network to obtain the illumination component and the reflection component corresponding to the low-light images in the training set respectively. Input the adaptive multi-modal fusion features into the Retinex decomposition network for preliminary screening. Step S5: Enhance the illumination component corresponding to the low-light images in the training set through the multi-scale illumination estimation network model to obtain the enhanced illumination component corresponding to the low-light images in the training set. Step S6: Input the reflection component corresponding to the low-light images in the training set and the adaptively multi-modal fusion features after preliminary screening into the feature selection network for selection and adjustment in sequence to obtain the reflection component features and multi-modal features respectively. Step S7: Enhance the respectively obtained reflection component features and multi-modal features through the reflection component enhancement network model to obtain the enhanced reflection component corresponding to the low-light images in the training set. Step S8: Multiply the enhanced illumination component corresponding to the low-light images in the training set and the enhanced reflection component corresponding to the low-light images in the training set element by element to obtain the enhanced image. Step S9: Use the loss function to train the Retinex decomposition network, the adaptive multi-modal fusion network, the multi-scale illumination estimation network model, the feature selection network, and the reflection component enhancement network model respectively. Step S10: Perform the operations of steps S2 to S8 on the low-light images in the test set and their corresponding normal-light images to obtain the enhanced low-light images.
2. The low-light image enhancement method according to claim 1, wherein In step S4 or step S9, the Retinex decomposition network is specifically: Multiple convolutional layers are connected in series, activation functions are interspersed between multiple convolutional layers, residual connections are used between each convolutional layer to obtain 4-channel features, the 4-channel features are divided into 1-channel and 3-channel features along the channels, and the illumination component and the reflection component corresponding to the low-light images in the training set are obtained using activation functions.
3. A low-light image enhancement method according to claim 1, characterized in that, In step S3, the adaptive multi-modal fusion network is specifically: After the low-light image and the infrared image are respectively subjected to feature extraction through convolutional layers, the feature-extracted low-light image and the feature-extracted infrared image are concatenated along the channels. The concatenated feature-extracted low-light image and the feature-extracted infrared image are sequentially passed through convolutional layers and activation function operations to obtain a guidance map. The guidance map is multiplied element by element with the low-light image to modulate the low-light features, and then the modulated low-light features and the feature-extracted infrared image are subjected to feature fusion using convolutional layers and element-by-element addition. The low-light image and the depth map are processed by the same operations, and finally the two fused features obtained are added together.
4. A low-light image enhancement method according to claim 1, characterized in that In the step S5, the multi-scale illumination estimation network model is specifically as follows: The illumination component L corresponding to the low-light image in the training set is used to obtain the initial illumination estimation map through 1 - L. After the initial illumination estimation map is successively subjected to two interpolation operations, a convolutional layer, SS2D, adaptive pooling, and activation operations, a luminance attention weight containing different scale information is obtained. At the same time, the illumination component L corresponding to the low-light image in the training set is successively subjected to two interpolation and convolutional layer operations, and the illumination component L after feature extraction is output. The illumination component L after feature extraction is multiplied element-wise with the luminance attention weight containing different scale information to obtain multiple adjusted luminance maps. After the multiple adjusted luminance maps are added and fused with the illumination component L after feature extraction, the enhanced illumination component corresponding to the low-light image in the training set is obtained.
5. A low-light image enhancement method according to claim 1, characterized in that, In the step S6, the feature selection network is specifically as follows: The reflection component corresponding to the low-light image in the training set and the adaptively multi-modal fused feature after preliminary screening are respectively subjected to convolutional layer operations, and the feature maps of the reflection component after preliminary feature extraction and the multi-modal feature after preliminary feature extraction are respectively output. After the feature map of the reflection component after preliminary feature extraction and the multi-modal feature after preliminary feature extraction are concatenated, deformable convolution and activation function operations are successively performed to respectively output the reflection component feature after feature extraction and the multi-modal feature after feature extraction. The reflection component feature after feature extraction and the reflection component corresponding to the low-light image in the training set are successively subjected to multiplication and addition operations to obtain the reflection component feature. The multi-modal feature after feature extraction and the adaptively multi-modal fused feature after preliminary screening are successively subjected to multiplication and addition operations to obtain the multi-modal feature.
6. The low-light image enhancement method according to claim 1, characterized in that In the step S7, the reflection component enhancement network model is specifically as follows: The reflection component feature and the multi-modal feature are two parallel branches, which are respectively successively subjected to normalization layer, convolutional layer, channel attention, and SS2D operations, and the reflection component feature after global feature extraction and the multi-modal feature after global feature extraction are respectively output. After the reflection component feature after global feature extraction and the multi-modal feature after global feature extraction are subjected to feature exchange and concatenation operations, depthwise separable convolution is performed for depth feature extraction. Finally, concatenation, depthwise separable convolution, and activation function operations are performed to obtain the enhanced reflection component corresponding to the low-light image in the training set.
7. A low-light image enhancement method according to claim 1, characterized in that, In the step S9, the loss function is specifically as follows: L = λ1L decom + λ2L R + λ3L light + λ4L recon + λ5L ssim + λ6L vgg ; Among them, L is the loss function, and λ1, λ2, λ3, λ4, λ5, and λ6 are all loss weights. L decom is the Retinex decomposition loss between the current low-light image and the normal-light image. L R is the l1 loss between the enhanced reflectance map and the reflectance map and multi-modal information of the normal light. L light is the l1 loss between the enhanced illumination map and the illumination map of the normal light. L recon is the l1 loss between the finally output image and the normal-light image. L ssim is the structural similarity loss. L vgg is the VGG loss.
8. A low-light image enhancement system, characterized in that, It includes the following modules: Module S1, the training set and the test set are respectively obtained. Both the training set and the test set include low-light images and the normal light images corresponding thereto; Module S2, the low-light image corresponding to the training set uses the multi-modal pre-trained model to generate a low-light multi-modal data set corresponding to the low-light image in the training set. The low-light multi-modal data set includes infrared images and depth maps; Module S3, which uses an adaptive multi-modal fusion network to obtain adaptive multi-modal fusion features from the low-light images, infrared images, and depth maps corresponding to the training set; Module S4, which inputs the low-light images in the training set into the Retinex decomposition network to respectively obtain the illumination component and the reflection component corresponding to the low-light images in the training set, and inputs the adaptive multi-modal fusion features into the Retinex decomposition network for preliminary screening; Module S5, where the illumination component corresponding to the low-light images in the training set is enhanced through a multi-scale illumination estimation network model to obtain the enhanced illumination component corresponding to the low-light images in the training set; Module S6, which inputs the reflection component corresponding to the low-light images in the training set and the adaptively multi-modal fusion features after preliminary screening into the feature selection network for selection and adjustment in sequence to respectively obtain the reflection component features and the multi-modal features; Module S7, which enhances the respectively obtained reflection component features and multi-modal features through a reflection component enhancement network model to obtain the enhanced reflection component corresponding to the low-light images in the training set; Module S8, which multiplies the enhanced illumination component corresponding to the low-light images in the training set and the enhanced reflection component corresponding to the low-light images in the training set element by element to obtain the enhanced image; Module S9, which uses a loss function to train the Retinex decomposition network, the adaptive multi-modal fusion network, the multi-scale illumination estimation network model, the feature selection network, and the reflection component enhancement network model respectively; Module S10, which performs the operations of Module S2 to Module S8 on the low-light images in the test set and the corresponding normal-light images to obtain the enhanced low-light images.
9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete mutual communication through the communication bus; The memory is used to store computer programs; The processor, when executing the programs stored on the memory, implements the method steps described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the method steps described in any one of claims 1-7.