Unsupervised low-light image enhancement method and device based on MRBNet
Through the dual-branch structure and blind-spot network of MRBNet, the reflection components in low-noise light images are extracted and denoised, and the problems of low-enhanced enhancement performance and poor noise suppression are solved, achieving high-quality low-intensity image enhancement and Good generalization.
Patent Information
- Application Number
- CN202410982952.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-22
AI Technical Summary
The existing unsupervised low-light image enhancement method has low performance, poor noise suppression after enhancement, and requires a large number of synthetic images for training, which has poor generalization.
Using a dual-branch structure based on MRBNet, the reflected components are extracted through image detail enhancement branches and encoder-decoder enhancement branches, and the local detail perception module and blind spot network are used to refer to each other and suppress noise to remove noise information.
It improves the performance and noise suppression effect of low-light images, enhances the extraction of texture details and color information of the image, and has good generalization. It only requires low-light images for training, without paired low-light and normal light pictures.
Smart Images

Figure CN118918053A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of low-light image enhancement, and in particular to a low-light image enhancement method and device based on MRBNet (Mutual referencing and blind denoising network, referred to as MRBNet) mutual referencing low-light enhancement and blind spot denoising. Background Art
[0002] Low-light image enhancement is an important research direction in the field of computer vision, which aims to improve the quality and visualization of images acquired under low-light conditions. In an uncontrolled environment, the imaging quality of the shooting equipment is often difficult to guarantee, especially in dim or extremely dark lighting conditions, the appearance of low-quality images is inevitable. Such low-light images not only damage the aesthetic quality of the image, but also have a serious impact on the performance of advanced computer vision tasks that rely on images as information input, such as target tracking, object recognition, and target detection. The development of low-light image enhancement technology provides an effective solution for improving image quality. This not only improves the human visual experience, but also significantly improves the performance of computer vision tasks that rely on image processing and analysis. Through the application of these technologies, high-quality images can be obtained in a variety of complex environments, thereby promoting the development and application of related fields.
[0003] The main purpose of low-light image enhancement is to make the originally unclear or low-brightness image clear or increase its brightness by highlighting the key information in the image and eliminating the interference information as much as possible. With the continuous advancement of technology, deep learning has shown strong application potential in various fields, and has been increasingly applied to the field of low-light image enhancement, and has made important progress. However, there are still some important problems in the field of low-light image enhancement that need to be solved: supervised low-light image enhancement methods have strong enhancement effects on dataset images but low generalization, poor performance in processing scenes different from the training dataset, and low-light and normal-light image pairs are difficult to capture in reality, requiring a large number of synthetic pictures for training. Unsupervised low-light image enhancement methods have low performance in low-light image enhancement due to the lack of supervised information. In addition, both supervised and unsupervised methods face the problem of noise. The degree of noise suppression after enhancement directly determines the image quality. The present invention adopts a deep learning method, uses dual branches to extract reflection components and form mutual references, uses a local detail perception module to retain detail information, and then uses a blind spot network to denoise the enhanced image to complete low-light image enhancement. Summary of the invention
[0004] Aiming at the problems of low image enhancement performance and noise suppression of enhanced images in unsupervised image enhancement methods, the present invention proposes an unsupervised low-light image enhancement method and device based on MRBNet.
[0005] The present invention uses two image enhancement branches to extract the reflection component, and the results are used as references to each other. This method improves the overall performance of the image enhancement stage, and the two branches use different network structures, aiming to retain more texture detail information, color information, and extract deeper semantic information. The method uses a blind spot network to suppress the noise of the enhanced image. The use of the blind spot network greatly suppresses the noise of the enhanced image. This is also a bold application of deep learning denoising technology in low-light image enhancement technology, and has achieved good results. This method improves the network's performance for image enhancement and suppresses the noise of the enhanced image, and has good generalization.
[0006] The present invention proposes an unsupervised low-light image enhancement method based on MRBNet, comprising:
[0007] S1. Constructing the dataset: The dataset only contains low-light images;
[0008] S2. randomly crop the input low-light image to be enhanced into an image of 128×128 pixels;
[0009] S3. Input the low-light image into the preset MRBNet; the preset MRBNet includes two stages: an image enhancement stage and a denoising stage; the image enhancement stage includes: an image detail enhancement branch and an encoder-decoder enhancement branch; the denoising stage includes: a blind spot network (BSN) denoising module;
[0010] S4. Input the low-light image into the image detail enhancement branch, separate the illumination component through convolution operation; calculate the reflection component according to the Retinex theory; input the reflection component into the local detail perception module to further extract the image detail information;
[0011] S5. Input the low-light image into the encoder-decoder enhancement branch to separate the illumination component, and calculate the reflection component according to the Retinex theory;
[0012] S6. During training, the two branches of the image enhancement stage simultaneously decompose the same input to obtain the illumination component, and then calculate the reflection component and form a reference with each other; during testing, only the image detail enhancement branch is used to obtain the reflection component;
[0013] S7. The reflection component obtained from the image detail enhancement branch is input as the input image to the blind spot network for denoising, and finally the low-light image enhancement result is obtained.
[0014] Among them, the image enhancement stage described in S3 includes two image enhancement branches: an image detail enhancement branch and an encoder-decoder enhancement branch; the image detail enhancement branch only uses a simple five-layer convolutional layer to extract the feature information of the image, and then uses a local detail perception module to refine the texture details and color information; compared with the image detail enhancement branch, the encoder-decoder enhancement branch aims to extract the multi-scale feature information of the image, and add jump connections to the network, while retaining low-level edge information and high-level semantic information; both networks ultimately calculate the reflection component based on the Retinex theory, and according to the consistency of the reflection component, the reflection components obtained by the two branches form a reference to each other, remove the noise information therein and complement the missing feature information of each other.
[0015] Furthermore, the image detail enhancement branch uses a simple five-layer convolutional layer to separate the illumination component, and the convolution kernel size used is 3×3; reflection padding is used before each convolutional layer, and emission padding can effectively reduce the generation of edge artifacts while preserving the spatial structure of the image;
[0016] The output layer uses the Sigmoid activation function, and the other layers use the ReLU activation function. The reflection component is calculated based on the Retinex theory using the illumination component. The corresponding calculation formula is:
[0017]
[0018] Where R is the reflection component, T is the illumination component, I is the input low-light image, and the operator is element-by-element division;
[0019] The image detail enhancement branch uses a local detail perception module to refine the texture details and color information of the reflection component; the local detail perception module creates a local receptive field by stacking 3×3 convolutional layers; in order to refine the color information, a channel attention mechanism is also added to the network; the reflection component passes through the local perception network to obtain the enhanced result of the image enhancement stage.
[0020] Furthermore, the encoder-decoder enhancement branch consists of an encoder, a decoder, and a skip connection structure;
[0021] The encoder structure has five layers of convolution blocks, each of which consists of a 3×3 convolution layer, a batch normalization layer, and a ReLU activation function. The convolution operation with a convolution step of 2 is used for downsampling, allowing the network to learn richer features while downsampling.
[0022] The decoder is also a five-layer convolutional block. Each convolutional block contains a transposed convolutional layer to achieve upsampling. The feature map of the encoder is spliced to the corresponding layer of the decoder using skip connections. The encoder-decoder network structure can extract multi-scale features of the input image, from low-level edge information to high-level semantic information. The features of each layer of the encoder are directly transferred to the corresponding decoder layer using skip connections. This connection method can retain more detailed information.
[0023] The illumination component is separated through the encoder-decoder network, and then the reflection component is calculated according to the Retinex theory.
[0024] Among them, a hybrid loss function is used to train the network for the image detail enhancement branch and the encoder-decoder enhancement branch in the image enhancement stage; including illumination separation loss, reflection consistency loss, Retinex loss and local detail perception loss;
[0025] Among them, the illumination separation loss can effectively retain more detail information to avoid losing too much information when separating the illumination components, which will cause the subsequent local detail perception module to fail to work; its expression is as follows:
[0026]
[0027] Where I is the input low-light image, T1 is the illumination component separated by the image detail enhancement branch, Expressed as squared L2 norm;
[0028] According to the Retinex theory, the reflection component of an image does not change due to external factors, and the reflection components obtained by the two branches should be consistent; the reason why the reflection components obtained by the two branches are different is that there is a lot of noise in the low-light image and the performance of the two branches is different. Using reflection consistency loss can make the two reflection components retain the same information and remove different information, which is mostly noise; its expression is:
[0029]
[0030] Among them, R1 is the reflection component calculated by the image detail enhancement branch, and R2 is the reflection component calculated by the encoder-decoder enhancement branch;
[0031] Since the Retinex theory is used to enhance low-light images, the Retinex loss is introduced to form a constraint, and its expression is:
[0032]
[0033] Where I0 is calculated by the maximum value of R, G, and B channels, and the expression is:
[0034]
[0035] The expression of local detail perception loss is:
[0036] L lan =||R1-R e ||1 (6)
[0037] Where R e is the output result of the local detail perception module, ||·||1 represents the L1 norm;
[0038] The loss function L used in the final image enhancement stage training is L t , L c , L r , L lan The weighted sum of t +λ2L c +λ3L r +λ4L lan ; Among them, λ1, λ2, λ3, and λ4 are loss weight coefficients.
[0039] The denoising stage described in step S3 uses the output of the image detail enhancement branch in the image enhancement stage as input and uses a blind spot network for denoising; the denoising using a blind spot network is characterized in that only a single noisy image can be used to train the network for denoising;
[0040] The blind spot network makes assumptions about the noisy image: the signals at different pixel positions are correlated with each other, the noises at different pixel positions are uncorrelated with each other, and the mean value of the noise is 0;
[0041] Based on this assumption, the blind spot network creates blind spot pixels for the input image pixels, and predicts the blind spot pixels based on the surrounding pixels of the blind spot pixels. According to the assumption, during the training process, since the blind spot position pixel and the surrounding pixel signals are correlated, the network can predict the pixel signal of the blind spot position. However, since the noise information contained in the blind spot position pixel and the noise information contained in the surrounding pixels are unrelated, the network cannot predict the noise information of the blind spot position pixel, thereby achieving the denoising effect.
[0042] The superiority of the blind spot network denoising has been proven by scholars in the denoising field. However, in reality, noisy images are difficult to meet the blind spot network's assumptions about noisy images. Therefore, an asymmetric PD (Pixel-shuffle) downsampling operation is used to minimize the spatial correlation of adjacent noisy pixels, so that the noisy image meets the blind spot network's assumptions about noisy images. Therefore, in the denoising stage, the image is first processed using asymmetric PD downsampling, then the blind spot network is used to denoise the image, and finally the inverse operation of asymmetric PD downsampling is used to restore the original size of the image to complete the denoising.
[0043] Furthermore, the loss function expression used in the blind spot network training in the denoising stage is:
[0044] L BSN =||PD -1 (BSN(PD(R1)))-R1||1 (7)
[0045] Where PD is the PD downsampling operation, PD -1 is the inverse operation of asymmetric PD downsampling, BSN is the blind spot network denoising operation, and R1 is the output of the image enhancement stage.
[0046] The second aspect of the present invention relates to an unsupervised low-light image enhancement device based on MRBNet, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the unsupervised low-light image enhancement method based on MRBNet of the present invention.
[0047] A third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the unsupervised low-light image enhancement method based on MRBNet of the present invention.
[0048] The present invention no longer requires paired low-light and normal-light images, and only uses low-light images for training, and can better solve the problem of insufficient image enhancement in an unsupervised training method and the problem of large image noise after enhancement affecting image quality.
[0049] Compared with the prior art, the present invention has the following advantages and technical effects:
[0050] The method proposed in the present invention uses only low-light images for training and adopts an unsupervised way to train the network; a method using two branches to extract reflection components and refer to each other is proposed to enhance the performance of image enhancement; one of the two branches uses a simple stacked convolutional layer to extract image feature information, and a local detail perception module in the branch enhances the extraction of texture detail information and color information; the other branch uses an encoder-decoder network structure and adds jump connections, and the network can simultaneously retain the low-level edge information and high-level semantic information of the image; the reflection components extracted by the two branches form a mutual reference, and the structure improves the performance of low-light enhancement and achieves satisfactory results; the method proposes to use a blind spot network to denoise the low-light image enhancement results, and the blind spot network has a very outstanding effect on noise suppression of enhanced pictures, effectively solving the problem of noise affecting image quality; the method proposed in the present invention improves the performance of low-light image enhancement and has good generalization when only low-light images are used for training. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 is a flow chart of the method of the present invention;
[0052] Figure 2 A network structure diagram of the method of the present invention;
[0053] Figure 3 This is a LAN module structure diagram in the network structure diagram of the present invention. DETAILED DESCRIPTION
[0054] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings:
[0055] Example 1
[0056] like Figure 1 As shown, the present invention proposes an unsupervised low-light image enhancement method based on MRBNet, comprising the following steps:
[0057] Step 1: Preprocess the image and randomly crop the input image into an image of 128×128 pixels;
[0058] Step 2: Input the image into the image detail enhancement branch, extract features through convolution operation and LAN module, separate the illumination component and then calculate the reflection component according to Retinex theory;
[0059] Step 3: Input the image into the encoder-decoder branch, separate the illumination component through the encoder-decoder network, and then calculate the reflection component according to the Retinex theory;
[0060] Step 4: The reflection components obtained by the two branches are referenced to each other to obtain the final output result of the image enhancement stage;
[0061] Step 5: Input the output result of the image enhancement stage into the blind spot network for image denoising;
[0062] Step 6: Output the final low-light image enhancement result.
[0063] Below, each step and network structure are described in detail.
[0064] like Figure 2 As shown in the figure, MRBNet uses two branches to separate two illumination components during the training phase, and calculates each illumination component based on the Retinex theory. During testing, only the image detail enhancement branch is used to complete the low-light image enhancement task.
[0065] During the training phase, the initial learning rate is set to 1×10 -4 , the Adam optimizer is used to optimize the network; the number of training times is set to 200. The low-light images in the dataset are from the LOL dataset.
[0066] After random cropping preprocessing, the image is input into the image detail enhancement branch and the encoder-decoder enhancement branch;
[0067] The image detail enhancement branch aims to extract more image detail information and color information from the image. In order to simplify the network structure, 5 convolutional layers are used to extract feature information. The convolution kernel size used is 3×3. Reflection padding is used before each convolutional layer. Reflection padding effectively retains the features at the edge by adding a circle of reflected pixel values at the edge of the image, reducing the loss of edge information during the convolution process.
[0068] The output layer uses the Sigmoid activation function, and the other layers use the ReLU activation function. The reflection component is calculated based on the Retinex theory using the illumination component. The corresponding calculation formula is:
[0069]
[0070] Where R is the reflection component, T is the illumination component, I is the input low-light image, and the operator is element-by-element division;
[0071] The image detail branch uses a local detail perception module to refine the texture details and color information of the reflection component; the local detail perception module is as follows Figure 3As shown in the figure, the local detail perception module consists of a convolutional layer and multiple residual blocks (RB). The residual structure is used in the module and multiple residual blocks are nested. The purpose of this is to enhance the model's expressiveness and learning ability, thereby improving the effect of image enhancement tasks. By nesting multiple residual blocks, the depth of the network can be increased without significantly increasing the computational complexity. Deeper networks usually have stronger expressiveness and can better capture and process detailed information in images, especially in tasks such as low-light image enhancement that require high-precision processing.
[0072] A channel attention layer (CA) is added to the RB structure. The channel attention layer can adaptively assign weights to enhance the overall attention of the module to key features. During feature extraction, some feature channels may contain more important brightness or texture detail information. The channel attention mechanism can dynamically increase the weights of these important channels, so that the network pays more attention to these key information when processing images, thereby improving the enhancement effect. In summary, the local detail perception module described above can enable the network to extract more texture details, color, and brightness information, enhancing the final low-light image enhancement effect.
[0073] After the reflection component passes through the local detail module, the output reflection component R1 of the image detail branch is finally obtained.
[0074] like Figure 2 As shown, the encoder-decoder enhancement branch consists of an encoder-decoder network and a skip connection structure.
[0075] The encoder structure has five layers of convolution blocks, each of which consists of a 3×3 convolution layer, a batch normalization layer, and a ReLU activation function. The convolution operation with a convolution step of 2 is used for downsampling, allowing the network to learn richer features while downsampling.
[0076] The decoder is also a five-layer convolutional block. Each convolutional block contains a transposed convolutional layer to achieve upsampling. The feature map of the encoder is spliced to the corresponding layer of the decoder using skip connections. The encoder-decoder network structure can extract multi-scale features of the input image, from low-level edge information to high-level semantic information. The features of each layer of the encoder are directly transferred to the corresponding decoder layer using skip connections. This connection method can retain more detailed information.
[0077] The advantage of using the above description structure to extract features is that it can extract features at different scales. The encoder part gradually downsamples to extract the global information of the image, while the decoder part gradually upsamples to restore the details of the image. Efficient fusion of features of different scales is achieved. This fusion helps to improve the effect of image enhancement, allowing the network to better integrate information at different levels and enhance details and contrast when processing low-light images.
[0078] The illumination component is separated by the encoder-decoder network, and then the reflection component R2 is calculated according to the Retinex theory.
[0079] The two reflection components R1 and R2 are obtained through the image detail enhancement branch and the encoder-decoder enhancement branch respectively. The two reflection components form a reference to each other. The purpose of this is to obtain more information for the two branch networks during training and better train the enhancement network. The theoretical basis for the two reflection components to form a reference to each other is that the Retinex theory believes that the perception of an image is the result of the separation of illumination and reflection. Illumination represents the light conditions in the scene, while reflection represents the inherent color and texture of the surface of an object. Therefore, the reflection components obtained by the two branches should be consistent; the reason why the reflection components obtained by the two branches are different is that there is a lot of noise in the low-light image and the performance of the two branches is different; therefore, the reflection components extracted by the two branches can form a reference to each other.
[0080] A hybrid loss function is used to train the network for the image detail enhancement branch and the encoder-decoder enhancement branch in the image enhancement stage; this includes illumination separation loss, reflection consistency loss, Retinex loss, and local detail perception loss.
[0081] The illumination separation loss can effectively retain more detail information to avoid losing too much information when separating the illumination components, which would cause the subsequent local detail perception module to fail to work; its expression is as follows:
[0082]
[0083] Where I is the input low-light image, T1 is the illumination component separated by the image detail enhancement branch, Expressed as the squared L2 norm.
[0084] Using reflection consistency loss can make the two reflection components retain the same information and remove different information, which is mostly noise. Its expression is:
[0085]
[0086] Among them, R1 is the reflection component calculated by the image detail enhancement branch, and R2 is the reflection component calculated by the encoder-decoder enhancement branch.
[0087] Since the Retinex theory is used to enhance low-light images, the Retinex loss is introduced to form a constraint, and its expression is:
[0088]
[0089] Where I0 is calculated by the maximum value of R, G, and B channels, and the expression is:
[0090]
[0091] The expression of local detail perception loss is:
[0092] L lan =||R1-R e ||1 (6)
[0093] Where R e is the output result of the local detail perception module, ||·||1 represents the L1 norm;
[0094] The loss function L used in the final image enhancement stage training is L t , L c , L r , L lan The weighted sum of t +λ2L c +λ3L r +λ4L lan ; Among them, λ1, λ2, λ3, and λ4 are loss weight coefficients.
[0095] Finally, R1 is used as the final output result in the image enhancement stage and as the input image in the denoising stage.
[0096] like Figure 2 As shown, the blind spot network is used for denoising; the blind spot network denoising also only requires one noisy picture for denoising. The input image after the image enhancement stage has a brightness similar to that of the normal light image, but contains a lot of noise and can be directly used as the input image of the blind spot network.
[0097] The blind spot network makes assumptions about noisy images: the signals at different pixel positions are correlated with each other, the noises at different pixel positions are uncorrelated with each other, and the mean of the noise is 0.
[0098] Based on this assumption, the blind spot network creates blind spot pixels for the input image pixels, and predicts the blind spot pixels based on the surrounding pixels of the blind spot pixels; according to the assumption, during the training process, since the blind spot position pixels and the surrounding pixel signals are correlated with each other, the network can predict the pixel signals at the blind spot position; however, since the noise information contained in the blind spot position pixels is unrelated to the noise information contained in the surrounding pixels, the network cannot predict the noise information of the blind spot position pixels, thereby achieving the denoising effect.
[0099] The excellence of the blind spot network denoising has been proven by scholars in the denoising field. However, in reality, noisy images are difficult to meet the blind spot network's assumptions about noisy images. Therefore, an asymmetric PD (Pixel-shuffle) downsampling operation is used to minimize the spatial correlation of adjacent noisy pixels, so that the noisy image meets the blind spot network's assumptions about noisy images. Therefore, in the denoising stage, the image is first processed using asymmetric PD downsampling, then the blind spot network is used to denoise the image, and finally the inverse operation of asymmetric PD downsampling is used to restore the original size of the image to complete the denoising.
[0100] The loss function expression used in the blind spot network training in the denoising stage is:
[0101] L BSN =||PD -1 (BSN(PD(R1)))-R1||1 (7)
[0102] Where PD is the PD downsampling operation, PD -1 is the inverse operation of asymmetric PD downsampling, BSN is the blind spot network denoising operation, and R1 is the output of the image enhancement stage.
[0103] The method proposed in this example of extracting reflection components by dual branches to form a mutual reference and using a blind spot network to denoise the enhanced image has significantly improved the performance of low-light image enhancement and achieved a satisfactory low-light enhancement effect.
[0104] Example 2
[0105] The present embodiment relates to an unsupervised low-light image enhancement device based on MRBNet, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the unsupervised low-light image enhancement method based on MRBNet described in Example 1.
[0106] Example 3
[0107] This embodiment relates to a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the unsupervised low-light image enhancement method based on MRBNet of Example 1 is implemented.
[0108] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.
Claims
1. An unsupervised low-light image enhancement method based on MRBNet, characterized in that: The following steps are involved: S1. Constructing the dataset: The dataset only contains low-light images; S2. randomly crop the input low-light image to be enhanced into an image of 128×128 pixels; S3. Input the low-light image into the preset MRBNet; The preset MRBNet consists of two stages: image enhancement stage and denoising stage; The image enhancement stage includes: image detail enhancement branch and encoder-decoder enhancement branch; the denoising stage includes: blind spot network (BSN) denoising module; S4. Input the low-light image into the image detail enhancement branch, separate the illumination component through convolution operation; calculate the reflection component according to the Retinex theory; input the reflection component into the local detail perception module to further extract the image detail information; S5. Input the low-light image into the encoder-decoder enhancement branch to separate the illumination component, and calculate the reflection component according to the Retinex theory; S6. During training, the two branches of the image enhancement stage simultaneously decompose the same input to obtain the illumination component, and then calculate the reflection component and form a reference with each other; during testing, only the image detail enhancement branch is used to obtain the reflection component; S7. The reflection component obtained from the image detail enhancement branch is input as the input image to the blind spot network for denoising, and finally the low-light image enhancement result is obtained.
2. The unsupervised low-light image enhancement method based on MRBNet according to claim 1, characterized in that: The image enhancement stage described in step S3 includes two image enhancement branches: an image detail enhancement branch and an encoder-decoder enhancement branch; the image detail branch only uses a simple five-layer convolutional layer to extract the feature information of the image, and then uses a local detail perception module to refine the texture details and color information; compared with the image detail branch, the encoder-decoder enhancement branch aims to extract the multi-scale feature information of the image, and add jump connections to the network, while retaining low-level edge information and high-level semantic information; both networks ultimately calculate the reflection component based on the Retinex theory, and according to the consistency of the reflection component, the reflection components obtained by the two branches form a reference to each other, remove the noise information therein and complement the feature information missing from each other.
3. The unsupervised low-light image enhancement method based on MRBNet according to claim 2, characterized in that: The image detail enhancement branch uses a simple five-layer convolutional layer to separate the illumination component, and the convolution kernel size used is 3×3. Reflection padding is used before each convolutional layer. Emission padding can effectively reduce the generation of edge artifacts while preserving the spatial structure of the image. The output layer uses Sigmoid activation function, and other layers use ReLU activation function; Use the illumination component to calculate the reflection component based on the Retinex theory. The corresponding calculation formula is: Where R is the reflection component, T is the illumination component, I is the input low-light image, and the operator is element-by-element division; The image detail branch uses a local detail perception module to refine the texture details and color information of the reflection component; the local detail perception module creates a local receptive field by stacking 3×3 convolutional layers; in order to refine the color information, a channel attention mechanism is also added to the network; the reflection component passes through the local perception network to obtain the enhanced result of the image enhancement stage.
4. The unsupervised low-light image enhancement method based on MRBNet according to claim 2, characterized in that: The encoder-decoder enhancement branch consists of an encoder, a decoder, and a skip connection structure; The encoder structure has five layers of convolution blocks, each of which consists of a 3×3 convolution layer, a batch normalization layer, and a ReLU activation function. The convolution operation with a convolution step of 2 is used for downsampling, allowing the network to learn richer features while downsampling. The decoder is also a five-layer convolutional block. Each convolutional block contains a transposed convolutional layer to achieve upsampling. The feature map of the encoder is spliced to the corresponding layer of the decoder using skip connections. The encoder-decoder network structure can extract multi-scale features of the input image, from low-level edge information to high-level semantic information. The features of each layer of the encoder are directly transferred to the corresponding decoder layer using skip connections. This connection method can retain more detailed information. The illumination component is separated through the encoder-decoder network, and then the reflection component is calculated according to the Retinex theory.
5. The unsupervised low-light image enhancement method based on MRBNet according to claim 1, characterized in that: The image detail enhancement branch and the encoder-decoder enhancement branch of the image enhancement stage described in step S3 use a hybrid loss function to train the network; These include illumination separation loss, reflection consistency loss, Retinex loss, and local detail perception loss.
6. The unsupervised low-light image enhancement method based on MRBNet according to claim 5, characterized in that: The illumination separation loss can effectively retain more detail information to avoid losing too much information when separating the illumination components, which would cause the subsequent local detail perception module to fail to work; its expression is as follows: Where I is the input low-light image, T1 is the illumination component separated by the image detail enhancement branch, Expressed as squared L2 norm; According to the Retinex theory, the reflection component of an image does not change due to external factors, and the reflection components obtained by the two branches should be consistent; the reason why the reflection components obtained by the two branches are different is that there is a lot of noise in the low-light image and the performance of the two branches is different. Using reflection consistency loss can make the two reflection components retain the same information and remove different information, which is mostly noise; its expression is: Among them, R1 is the reflection component calculated by the image detail enhancement branch, and R2 is the reflection component calculated by the encoder-decoder enhancement branch; Since the Retinex theory is used to enhance low-light images, the Retinex loss is introduced to form a constraint, and its expression is: Where I0 is calculated by the maximum value of R, G, and B channels, and the expression is: The expression of local detail perception loss is: L lan =||R1-R e ||1 (6) Where R e is the output result of the local detail perception module, ||·||1 represents the L1 norm; The loss function L used in the final image enhancement stage training is L t , L c , L r , L lan The weighted sum of t +λ2L c +λ3L r +λ4L lan ; Among them, λ1, λ2, λ3, and λ4 are loss weight coefficients.
7. The unsupervised low-light image enhancement method based on MRBNet according to claim 1, characterized in that: The denoising stage described in step S3 uses the output of the image detail enhancement branch in the image enhancement stage as input and uses a blind spot network for denoising; the use of a blind spot network for denoising is characterized in that only a single noisy image can be used to train the network for denoising; The blind spot network makes assumptions about the noisy image: the signals at different pixel positions are correlated with each other, the noises at different pixel positions are uncorrelated with each other, and the mean value of the noise is 0; Based on this assumption, the blind spot network creates blind spot pixels for the input image pixels, and predicts the blind spot pixels based on the surrounding pixels of the blind spot pixels. According to the assumption, during the training process, since the blind spot position pixel and the surrounding pixel signals are correlated, the network can predict the pixel signal of the blind spot position. However, since the noise information contained in the blind spot position pixel and the noise information contained in the surrounding pixels are unrelated, the network cannot predict the noise information of the blind spot position pixel, thereby achieving the denoising effect. The excellence of the blind spot network denoising has been proven by scholars in the denoising field. However, in reality, noisy images are difficult to meet the blind spot network's assumptions about noisy images. Therefore, an asymmetric PD (Pixel-shuffle) downsampling operation is used to minimize the spatial correlation of adjacent noisy pixels, so that the noisy image meets the blind spot network's assumptions about noisy images. Therefore, in the denoising stage, the image is first processed using asymmetric PD downsampling, then the blind spot network is used to denoise the image, and finally the inverse operation of asymmetric PD downsampling is used to restore the original size of the image to complete the denoising.
8. The unsupervised low-light image enhancement method based on MRBNet according to claim 7, characterized in that: The loss function expression used in the blind spot network training in the denoising stage is: L BSN =||PD -1 (BSN(PD(R1)))-R1||1 (7) Where PD is the PD downsampling operation, PD -1 is the inverse operation of asymmetric PD downsampling, BSN is the blind spot network denoising operation, and R1 is the output of the image enhancement stage.
9. An unsupervised low-light image enhancement device based on MRBNet, characterized in that: It comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the unsupervised low-light image enhancement method based on MRBNet described in any one of claims 1-8.
10. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the unsupervised low-light image enhancement method based on MRBNet described in any one of claims 1 to 8 is implemented.
Citation Information
Patent Citations
Low-illumination image enhancement method and system based on image inpainting technology
CN113450290A
Image enhancement method, device and equipment and readable storage medium
CN114648457A
Low-illumination image enhancement method based on multi-scale and context learning network
CN114998145A
Unsupervised single image enhancement method and system based on Retinex fractional order variation network
CN115797205A
Low-light image enhancement method and device fusing high and low frequency feature information
CN116152120A
Cited By
Self-supervised image denoising method and system for multi-angle sequence
CN120543415A
Infrared-guided unsupervised low-light image denoising and enhancing method and system, and medium
CN121563822A
