Low-light image decoupling enhancement method based on Retinex model

Abstract

The present invention discloses a low-light image decoupling enhancement method based on a Retinex model. First, the input low-light image is Retinex-decomposed in the RGB space; the lighting color, the illumination component and the reflection component are obtained; the obtained illumination component is adjusted through the NR formula for multiple cycles to obtain the adjusted illumination component, and an enhanced image is obtained based on the illumination component. Finally, the enhanced noise image is denoised to obtain the final enhancement result. The present invention makes up for the limitations of the current supervised learning method in dealing with illumination changes, and improves the visual quality and usability of images under non-ideal illumination conditions.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital image processing and computer vision, and in particular relates to a low-light image decoupling enhancement method based on a Retinex model. Background Art

[0002] Low-light image enhancement is an important research direction in the field of image processing, mainly for optimizing images taken in low-light environments. Such images usually have problems such as high noise, low contrast, and unclear details. Traditional methods for enhancing such images include histogram equalization and contrast stretching. These methods only analyze from a single perspective such as contrast or dynamic range to achieve brightness and contrast improvement. However, algorithms based on deep learning can improve the quality of visual perception from multiple perspectives such as color, content, contrast, and lighting at the same time, with significant performance improvements.

[0003] Existing low-light image enhancement methods based on deep learning mostly adopt a supervised approach. It is difficult for such methods to learn a stable model with good generalization performance based on a given small data set. This is because the image has complex mixed degradation problems, and the illumination distribution is very complex and difficult to learn. To this end, some methods have begun to decouple multiple mixed degradations, converting a supervised problem into multiple supervised sub-problems through the Retinex model and Fourier transform. The present invention fully analyzes the characteristics of various degradation problems in low-light images, and uses the Retinex model to decouple the low-light image enhancement problem into two sub-problems: illumination adjustment and denoising. Different from previous decoupling methods, the illumination adjustment in this method is implemented using the unsupervised Naka-Rushton (NR) formula, while the denoising module is implemented using supervised deep learning. Summary of the invention

[0004] The purpose of the present invention is to provide a low-light image decoupling enhancement method based on the Retinex model to make up for the limitations of current supervised learning methods in dealing with illumination changes and to improve the visual quality and usability of images under non-ideal illumination conditions.

[0005] The technical solution adopted by the present invention is a low-light image decoupling enhancement method based on the Retinex model, which is specifically implemented according to the following steps:

[0006] Step 1: Perform Retinex decomposition on the input low-light image in RGB space to obtain the lighting color c, the illumination component L and the reflection component R;

[0007] Step 2: The illumination component L obtained in step 1 is adjusted through the NR formula for multiple cycles to obtain the adjusted illumination component L. out , based on L out Get the enhanced image Inoise =c·L out ⊙R;

[0008] Step 3: Enhanced noise image I noise Denoising is performed to obtain the final enhanced result I en .

[0009] The present invention is also characterized in that:

[0010] In step 1, the input low-light image is retinex-decomposed in RGB space as follows:

[0011] I=c·L⊙R (1)

[0012] Where I represents the low-light image, L∈R m×n is the illumination component, m represents the width of the low-light image, n represents the height of the image, R∈R m×n×3 is the reflection component, c=[c r , c g , c b ] represents the lighting color, where c r , c g , c b Respectively represent the lighting values ​​corresponding to the RGB color channels.

[0013] Step 1 is as follows:

[0014] Step 1.1, the illumination color c is regarded as atmospheric light, and the dark channel prior principle is used to obtain the illumination color c; that is, first estimate the dark channel map of the low-light image, then select the pixels with the top 10% brightness values ​​in the dark channel map, and use the average brightness of these pixels on the RGB color channel as c r , c g , c b The value of .

[0015] Step 1.2: The process of solving the illumination component L is as follows:

[0016] First, the V component of the low-light image I in the HSV color space is downsampled to 1 / 4 of its original size and input into the traditional Retinex decomposition algorithm JieP to obtain the low-resolution illumination component L lr , low-resolution illumination component L lr It can effectively capture the basic features of the explicit prior constraints imposed on the illumination component in the JieP decomposition method;

[0017] First, L lr Upsample to the original image size Then build a linear fusion model, the specific calculation is as follows:

[0018]

[0019]

[0020] Among them, f g is the guided filter, V represents the V component of image I in the HSV color space, Indicates is the result of the guided filtering operation on V by the guided graph, V filter is the result of two consecutive guided filtering, α is a weight value;

[0021] Step 1.3, after obtaining the illumination component L and the illumination color c, solve the reflection component R by the following formula:

[0022]

[0023] It means that the corresponding elements in the matrix are divided, and ∈ is a constant used to prevent the denominator from being zero.

[0024] The specific calculation method for each lighting adjustment in step 2 is as follows:

[0025] Let the illumination component to be adjusted be L in , the adjusted illumination component is L out , the calculation process is as follows:

[0026]

[0027]

[0028] Among them, σ(x, y) represents the adaptive adjustment factor at the pixel point (x, y), and σ(x, y) is adaptively fused with the local adjustment factor σ l (x, y) and a global adjustment factor σ g Find,ω l (x, y) and ω g (x, y) represent the weights of the local adjustment factor and the global adjustment factor at the pixel point (x, y); σ g Characterizes the brightness level of the input image, σ g By estimating L in The average brightness value of all pixels in is obtained, and η controls the global contrast of the image. Their specific calculation process is as follows:

[0029]

[0030] In the above formula, mean(L in ) indicates L inThe brightness values ​​of all pixels in are averaged. exp() represents the natural exponential function, that is, the exponential function with the real number e as the base. Since σ g The value range of is [0,1], then the value of η is in the interval [1,2.7]. k is a parameter that controls the contribution of global and local adjustment factors. It is set to 0.4 according to experience. σ1 and g σ2 are Gaussian filters with standard deviations σ1 and σ2 respectively;

[0031] The enhanced noisy image I in step 3 noise It is expressed as:

[0032] I noise =c·L out ⊙R (6)

[0033] Denoising result I en It is expressed as the following formula:

[0034]

[0035] First, use Gaussian filtering to filter I noise Decompose into high frequency parts and low frequency part g represents a high-frequency noise suppression module based on a gate mechanism, which aims to filter out I noise Part of the noise in, f Θ Represents a denoising network with network parameters Θ, which is used to reconstruct high-frequency details of the image and further suppress noise.

[0036] The specific operation of the high-frequency noise suppression module based on the gate mechanism in step 3 is as follows:

[0037] The calculation formula for the high-frequency component after noise suppression is:

[0038]

[0039] represents the high-frequency component after noise suppression, ⊙ represents the multiplication of the corresponding elements in the matrix. From formula (7), we can see that the subsequent denoising network f Θ Aims to reconstruct the high-frequency details of the image, which are highly correlated with the low-frequency information, so the input of the network is Splicing is performed on the channel, which is expressed as:

[0040]

[0041] is the value of Function g, Concat means concatenation at the channel layer.

[0042] The denoising network structure based on the attention mechanism in step 3 is as follows:

[0043] The network adopts an encoder-decoder structure. The encoder is divided into a shallow encoding module and a deep encoding module. The shallow encoding module includes three 3×3 convolutional layers and a spatial information processing module. The convolutional layer uses the LeakyReLU activation function. The spatial information processing module adopts a residual structure. The module first uses 3×3 convolution operation + ReLU activation function + 3×3 convolution operation for feature extraction, and then obtains the output feature through residual connection; the deep encoding module is divided into two branches, the spatial domain and the frequency domain. The spatial branch uses 6 spatial information processing blocks to extract local structural features F spatial ; The frequency domain branch extracts the global Fourier frequency domain information F through 5 frequency domain information processing blocks fourier ,Each frequency domain information processing block first uses Fourier transform to convert the input features into Fourier space to obtain the amplitude and phase components, and then uses 1×1 convolution operation + LeakyReLU + 1×1 convolution operation to extract the features of the amplitude and phase components respectively, and then uses the inverse Fourier transform to convert the features corresponding to the amplitude and phase back to the spatial domain, and then obtains the output features through 3×3 convolution operation and residual connection;

[0044] Fusion of local structural features F spatial and the global Fourier frequency domain information F fourier , the fusion rules are as follows:

[0045]

[0046] F represents the fused output feature, Att represents the noise distribution attention map, Att represents the noise intensity, and the noise distribution map is approximated by the underexposure level of the image. Att is calculated using the results before and after the adjustment of the illumination component, and the value of Att is between [0, 1];

[0047] Then, the output feature F is input into the decoder to generate a noise-free high-frequency component, as shown in formula (7), and the obtained high-frequency component is combined with the low-frequency component obtained by the Gaussian filter in formula (7) to obtain the final denoised image;

[0048] When training the denoising network, the loss function used is defined as:

[0049]

[0050] φ is a pre-trained VGG network, I gt is a true normal lighting image, The operation refers to performing Fourier transform on the corresponding image and extracting the image amplitude information, ω 1 and ω 2is the weight factor.

[0051] The beneficial effect of the present invention is that the low-light image decoupling enhancement method based on the Retinex model improves the existing low-light image enhancement method based on deep learning, because the image has complex mixed degradation problems and the illumination distribution is very complex, which makes it difficult to use supervised training and learning. The present invention fully analyzes the characteristics of various degradation problems in low-light images and uses the Retinex model to decouple the low-light image enhancement problem into two sub-problems: illumination adjustment and denoising. Different from previous decoupling methods, the illumination adjustment in this method is implemented using the unsupervised Naka-Rushton (NR) formula, while the denoising module is implemented using supervised deep learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a denoising network of the low-light image decoupling enhancement method based on the Retinex model of the present invention;

[0053] Figure 2 It is the spatial information processing block of the low-light image decoupling enhancement method based on the Retinex model of the present invention;

[0054] Figure 3 It is the frequency domain information processing block of the low-light image decoupling enhancement method based on the Retinex model of the present invention. DETAILED DESCRIPTION

[0055] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] The present invention is based on the low-light image decoupling enhancement method of the Retinex model, combined with Figure 1 , follow the steps below:

[0057] Step 1: Perform Retinex decomposition on the input low-light image in RGB space to obtain the lighting color c, the illumination component L and the reflection component R;

[0058] In step 1, the input low-light image is retinex-decomposed in RGB space as follows:

[0059] I=c·L⊙R (1)

[0060] Where I represents the low-light image, L∈R m×n is the illumination component, m represents the width of the low-light image, n represents the height of the image, R∈R m×n×3 is the reflection component, c=[c r , c g , c b ] represents the lighting color, where c r , c g , cb Respectively represent the lighting values ​​corresponding to the RGB color channels.

[0061] Step 1 is as follows:

[0062] Step 1.1, the illumination color c is regarded as atmospheric light, and the dark channel prior principle can be used to obtain the illumination color c; that is, first estimate the dark channel map of the low-light image, then select the pixels with the top 10% brightness values ​​in the dark channel map, and use the average brightness of these pixels on the RGB color channel as c r , c g , c b The value of .

[0063] Step 1.2: In order to retain the advantages of the traditional explicit prior constraints and improve the computational efficiency on high-resolution images, the process of solving the illumination component L in the present invention is as follows:

[0064] First, the V component of the low-light image I in the HSV color space is downsampled to 1 / 4 of its original size and input into the traditional Retinex decomposition algorithm JieP to obtain the low-resolution illumination component L lr , low-resolution illumination component L lr It can effectively capture the basic features of the explicit prior constraints imposed on the illumination component in the JieP decomposition method;

[0065] Then how to use L lr The problem to be solved is to obtain the illumination component L at the original resolution. To this end, first lr Upsample to the original image size Then build a linear fusion model, the specific calculation is as follows:

[0066]

[0067] Among them, f g is the guided filter, V represents the V component of image I in the HSV color space, Indicates is the result of the guided filtering operation on V by the guided graph, V filter is the result of two consecutive guided filterings. α is a weight value that can dynamically adjust the smoothness of the illumination component L. In the present invention, α is set to 0.5. According to the characteristics of guided filtering, the above formula can make V filter (Filtered image) can obtain image content information from V, and also contains Gradient information of the illumination component L. By using the above steps, we can not only retain the explicit constraint characteristics imposed on the illumination component in the Jiep algorithm, but also quickly obtain L on a high-resolution image.

[0068] Step 1.3, after obtaining the illumination component L and the illumination color c, solve the reflection component R by the following formula:

[0069]

[0070] Indicates the division of corresponding elements in the matrix, ∈ is a constant used to prevent the denominator from being zero, and in this invention, the value is 10 -4 , the max function means finding the maximum value of two numbers.

[0071] Step 2: Inspired by the human visual perception mechanism, the illumination component L obtained in step 1 is adjusted multiple times through the NR (Naka-Rushton) formula to obtain the adjusted illumination component L out , achieving brightness improvement and effectively solving the problem of uneven lighting, that is, the simultaneous presence of partial overexposure and underexposure. After two cycles of lighting adjustment, a good effect can be achieved. out Get the enhanced image I noise =c·L out ⊙R, which contains a lot of noise.

[0072] The specific calculation method for each lighting adjustment in step 2 is as follows:

[0073] Let the illumination component to be adjusted be L in , the adjusted illumination component is L out , the calculation process is as follows:

[0074]

[0075] Among them, σ(x, y) represents the adaptive adjustment factor at the pixel point (x, y), and σ(x, y) is adaptively fused with the local adjustment factor σ l (x,y) and the global adjustment factor σ g Find,ω l (x,y) and ω g (x, y) represent the weights of the local adjustment factor and the global adjustment factor at the pixel point (x, y); σ g Characterizes the brightness level of the input image, σ g By estimating L in The average brightness value of all pixels in is obtained, and η controls the global contrast of the image. Their specific calculation process is as follows:

[0076]

[0077] In the above formula, mean(L in ) indicates L inThe brightness values ​​of all pixels in are averaged. exp() represents the natural exponential function, that is, the exponential function with the real number e as the base. Since σ g The value range of is [0,1], then the value of η is in the interval [1,2.7]. k is a parameter that controls the contribution of global and local adjustment factors. It is set to 0.4 according to experience. σ1 and g σ2 They are Gaussian filters with standard deviations σ1 and σ2 respectively. In the present invention, the values ​​of σ1 and σ2 are 2.5 and 1.5 respectively.

[0078] Step 3: Enhanced noise image I noise Denoising is performed to obtain the final enhanced result I en ,This step consists of two stages. The first stage provides a high-frequency noise suppression method based on a gate mechanism; the second stage provides a denoising network based on attention guidance.

[0079] The enhanced noisy image I in step 3 noise It is expressed as:

[0080] I noise =c·L out ⊙R (6)

[0081] In this step, we introduce noise The denoising process consists of two stages: the first stage provides a high-frequency noise suppression method based on a gate mechanism; the second stage provides a denoising network based on attention guidance. Through the operation of these two stages, the noise in the enhanced image can be removed. The final denoising result I en It is expressed as the following formula:

[0082]

[0083] First, use Gaussian filtering to filter I noise Decompose into high frequency parts and low frequency part g represents a high-frequency noise suppression module based on a gate mechanism, which aims to filter out I noise Part of the noise in, f Θ Represents a denoising network with network parameters Θ, which is used to reconstruct high-frequency details of the image and further suppress noise.

[0084] The specific operation of the high-frequency noise suppression module based on the gate mechanism in step 3 is as follows:

[0085] In natural images, if the high-frequency component is not noise, its color information will be consistent with the low-frequency component. That is, for each pixel (x, y) in color channel i, if Larger, then It may also be larger; if Smaller, then will be smaller. Therefore, Can be used as a gate mechanism to filter out The calculation formula of high frequency component after noise suppression is:

[0086]

[0087] represents the high-frequency component after noise suppression, ⊙ represents the multiplication of the corresponding elements in the matrix. From formula (7), we can see that the subsequent denoising network f Θ Aims to reconstruct the high-frequency details of the image, which are highly correlated with the low-frequency information, so the input of the network is Splicing is performed on the channel, which is expressed as:

[0088]

[0089] is the value of Function g, Concat means concatenation at the channel layer.

[0090] The denoising network structure based on the attention mechanism in step 3 is as follows:

[0091] The network adopts an encoder-decoder structure (such as Figure 1 As shown), the encoder is divided into a shallow encoding module and a deep encoding module. The shallow encoding module includes three 3×3 convolutional layers and a spatial information processing module (as shown in Figure 2 As shown in the figure, the convolution layer uses the LeakyReLU activation function, and the spatial information processing module uses a residual structure. The module first uses 3×3 convolution operation + ReLU activation function + 3×3 convolution operation to extract features, and then obtains the output features through residual connection; the deep encoding module is divided into two branches, spatial domain and frequency domain. The spatial branch uses 6 spatial information processing blocks to extract local structural features F spatial ; The frequency domain branch passes through 5 frequency domain information processing blocks (such as Figure 3 As shown in Figure 2, extract the global Fourier frequency domain information F fourier ,Each frequency domain information processing block first uses Fourier transform to convert the input features into Fourier space to obtain the amplitude and phase components, and then uses 1×1 convolution operation + LeakyReLU + 1×1 convolution operation to extract the features of the amplitude and phase components respectively, and then uses the inverse Fourier transform to convert the features corresponding to the amplitude and phase back to the spatial domain, and then obtains the output features through 3×3 convolution operation and residual connection;

[0092] Generally, for areas with less noise, local image features are sufficient to reconstruct details. On the contrary, for areas with high noise intensity, global features are needed to assist in reconstructing image details. Therefore, local structural features F can be fused spatial and the global Fourier frequency domain information F fourier , the fusion rules are as follows:

[0093] F=F spatial ×(1-Att)×F fourier ×Att

[0094]

[0095] F represents the fused output feature, Att represents the noise distribution attention map, Att represents the noise intensity, and the noise distribution map is approximated by the underexposure level of the image. Att is calculated using the results before and after the adjustment of the illumination component, and the value of Att is between [0, 1];

[0096] Then, the output feature F is input into the decoder to generate a noise-free high-frequency component, as shown in formula (7), and the obtained high-frequency component is combined with the low-frequency component obtained by the Gaussian filter in formula (7) to obtain the final denoised image;

[0097] When training the denoising network, the loss function used is defined as:

[0098]

[0099] φ is a pre-trained VGG network, I gt is a true normal lighting image, The operation refers to performing Fourier transform on the corresponding image and extracting the image amplitude information, ω 1 and ω 2 are weight factors, which are empirically set to 0.01 and 0.1 respectively.

[0100] Example 1

[0101] The present invention is based on the low-light image decoupling enhancement method of the Retinex model, combined with Figure 1 , follow the steps below:

[0102] Step 1: Perform Retinex decomposition on the input low-light image in RGB space to obtain the lighting color c, the illumination component L and the reflection component R;

[0103] Step 2: Inspired by the human visual perception mechanism, the illumination component L obtained in step 1 is adjusted multiple times through the NR (Naka-Rushton) formula to obtain the adjusted illumination component L out, achieving brightness improvement and effectively solving the problem of uneven lighting, that is, the simultaneous presence of partial overexposure and underexposure. After two cycles of lighting adjustment, a good effect can be achieved. out Get the enhanced image I noise =c·L out ⊙R, which contains a lot of noise.

[0104] Step 3: Enhanced noise image I noise Denoising is performed to obtain the final enhanced result I en ,This step consists of two stages. The first stage provides a high-frequency noise suppression method based on a gate mechanism; the second stage provides a denoising network based on attention guidance.

[0105] Example 2

[0106] The present invention is based on the low-light image decoupling enhancement method of the Retinex model, combined with Figure 1 , follow the steps below:

[0107] Step 1: Perform Retinex decomposition on the input low-light image in RGB space to obtain the lighting color c, the illumination component L and the reflection component R;

[0108] In step 1, the input low-light image is retinex-decomposed in RGB space as follows:

[0109] I=c·L⊙R (1)

[0110] Where I represents the low-light image, L∈R m×n is the illumination component, m represents the width of the low-light image, n represents the height of the image, R∈R m×n×3 is the reflection component, c=[c r , c g , c b ] represents the lighting color, where c r , c g , c b Respectively represent the lighting values ​​corresponding to the RGB color channels.

[0111] Step 1 is as follows:

[0112] Step 1.1, the illumination color c is regarded as atmospheric light, and the dark channel prior principle is used to obtain the illumination color c; that is, first estimate the dark channel map of the low-light image, then select the pixels with the top 10% brightness values ​​in the dark channel map, and use the average brightness of these pixels on the RGB color channel as c r , c g , c b The value of .

[0113] Step 1.2: In order to retain the advantages of the traditional explicit prior constraints and improve the computational efficiency on high-resolution images, the process of solving the illumination component L in the present invention is as follows:

[0114] First, the V component of the low-light image I in the HSV color space is downsampled to 1 / 4 of its original size and input into the traditional Retinex decomposition algorithm JieP to obtain the low-resolution illumination component L lr , low-resolution illumination component L lr It can effectively capture the basic features of the explicit prior constraints imposed on the illumination component in the JieP decomposition method;

[0115] Then how to use L lr The problem to be solved is to obtain the illumination component L at the original resolution. To this end, first lr Upsample to the original image size Then build a linear fusion model, the specific calculation is as follows:

[0116]

[0117] Among them, f g is the guided filter, V represents the V component of image I in the HSV color space, Indicates is the result of the guided filtering operation on V by the guided graph, V filter is the result of two consecutive guided filterings. α is a weight value that can dynamically adjust the smoothness of the illumination component L. In the present invention, α is set to 0.5. According to the characteristics of guided filtering, the above formula can make V filter (Filtered image) can obtain image content information from V, and also contains Gradient information of the illumination component L. By using the above steps, we can not only retain the explicit constraint characteristics imposed on the illumination component in the Jiep algorithm, but also quickly obtain L on a high-resolution image.

[0118] Step 1.3, after obtaining the illumination component L and the illumination color c, solve the reflection component R by the following formula:

[0119]

[0120] Indicates the division of corresponding elements in the matrix, ∈ is a constant used to prevent the denominator from being zero, and in this invention, the value is 10 -4 .

[0121] Step 2: Inspired by the human visual perception mechanism, the illumination component L obtained in step 1 is adjusted multiple times through the NR (Naka-Rushton) formula to obtain the adjusted illumination component L out , achieving brightness improvement and effectively solving the problem of uneven lighting, that is, the simultaneous presence of partial overexposure and underexposure. After two cycles of lighting adjustment, a good effect can be achieved. out Get the enhanced image I noise =c·L out ⊙R, which contains a lot of noise.

[0122] Step 3: Enhanced noise image I noise Denoising is performed to obtain the final enhanced result I en ,This step consists of two stages. The first stage provides a high-frequency noise suppression method based on a gate mechanism; the second stage provides a denoising network based on attention guidance.

[0123] The enhanced noisy image I in step 3 noise It is expressed as:

[0124] I noise =c·L out ⊙R (6)

[0125] In this step, we introduce noise The denoising process consists of two stages: the first stage provides a high-frequency noise suppression method based on a gate mechanism; the second stage provides a denoising network based on attention guidance. Through the operation of these two stages, the noise in the enhanced image can be removed. The final denoising result I en It is expressed as the following formula:

[0126]

[0127] First, use Gaussian filtering to filter I noise Decompose into high frequency parts and low frequency part g represents a high-frequency noise suppression module based on a gate mechanism, which aims to filter out I noise Part of the noise in, f Θ Represents a denoising network with network parameters Θ, which is used to reconstruct high-frequency details of the image and further suppress noise.

[0128] The specific operation of the high-frequency noise suppression module based on the gate mechanism in step 3 is as follows:

[0129] In natural images, if the high-frequency component is not noise, its color information will be consistent with the low-frequency component. That is, for each pixel (x, y) in color channel i, if Larger, then It may also be larger; if Smaller, then will be smaller. Therefore, Can be used as a gate mechanism to filter out The calculation formula of high frequency component after noise suppression is:

[0130]

[0131] represents the high-frequency component after noise suppression, ⊙ represents the multiplication of the corresponding elements in the matrix. From formula (7), we can see that the subsequent denoising network f Θ Aims to reconstruct the high-frequency details of the image, which are highly correlated with the low-frequency information, so the input of the network is Splicing is performed on the channel, which is expressed as:

[0132]

[0133] is the value of Function g, Concat means concatenation at the channel layer.

[0134] The denoising network structure based on the attention mechanism in step 3 is as follows:

[0135] The network adopts an encoder-decoder structure (such as Figure 1 As shown), the encoder is divided into a shallow encoding module and a deep encoding module. The shallow encoding module includes three 3×3 convolutional layers and a spatial information processing module (as shown in Figure 2 As shown in the figure, the convolution layer uses the LeakyReLU activation function, and the spatial information processing module uses a residual structure. The module first uses 3×3 convolution operation + ReLU activation function + 3×3 convolution operation to extract features, and then obtains the output features through residual connection; the deep encoding module is divided into two branches, spatial domain and frequency domain. The spatial branch uses 6 spatial information processing blocks to extract local structural features F spatial ; The frequency domain branch passes through 5 frequency domain information processing blocks (such as Figure 3 As shown in Figure 2, extract the global Fourier frequency domain information F fourier ,Each frequency domain information processing block first uses Fourier transform to convert the input features into Fourier space to obtain the amplitude and phase components, and then uses 1×1 convolution operation + LeakyReLU + 1×1 convolution operation to extract the features of the amplitude and phase components respectively, and then uses the inverse Fourier transform to convert the features corresponding to the amplitude and phase back to the spatial domain, and then obtains the output features through 3×3 convolution operation and residual connection;

[0136] Generally, for areas with less noise, local image features are sufficient to reconstruct details. On the contrary, for areas with high noise intensity, global features are needed to assist in reconstructing image details. Therefore, local structural features F can be fused spatial and the global Fourier frequency domain information F fourier , the fusion rules are as follows:

[0137] F=F spatial ×(1-Att)+F fourier ×Att

[0138] F represents the fused output feature, Att represents the noise distribution attention map, Att represents the noise intensity, and the noise distribution map is approximated by the underexposure level of the image. Att is calculated using the results before and after the adjustment of the illumination component, and the value of Att is between [0, 1];

[0139] Then, the output feature F is input into the decoder to generate a noise-free high-frequency component, as shown in formula (7), and the obtained high-frequency component is combined with the low-frequency component obtained by the Gaussian filter in formula (7) to obtain the final denoised image;

[0140] When training the denoising network, the loss function used is defined as:

[0141]

[0142] φ is a pre-trained VGG network, I gt is a true normal lighting image, The operation refers to performing Fourier transform on the corresponding image and extracting the image amplitude information, ω 1 and ω 2 are weight factors, which are empirically set to 0.01 and 0.1 respectively.

[0143] Example 3

[0144] The present invention is based on the low-light image decoupling enhancement method of the Retinex model, combined with Figure 1 , follow the steps below:

[0145] Step 1: Perform Retinex decomposition on the input low-light image in RGB space to obtain the lighting color c, the illumination component L and the reflection component R;

[0146] In step 1, the input low-light image is retinex-decomposed in RGB space as follows:

[0147] I=c·L⊙R (1)

[0148] Where I represents the low-light image, L∈R m×nis the illumination component, m represents the width of the low-light image, n represents the height of the image, R∈R m×n×3 is the reflection component, c=[c r , c g , c b ] represents the lighting color, where c r , c g , c b Respectively represent the lighting values ​​corresponding to the RGB color channels.

[0149] Step 1 is as follows:

[0150] Step 1.1, the illumination color c is regarded as atmospheric light, and the dark channel prior principle is used to obtain the illumination color c; that is, first estimate the dark channel map of the low-light image, then select the pixels with the top 10% brightness values ​​in the dark channel map, and use the average brightness of these pixels on the RGB color channel as c r , c g , c b The value of .

[0151] Step 1.2: In order to retain the advantages of the traditional explicit prior constraints and improve the computational efficiency on high-resolution images, the process of solving the illumination component L in the present invention is as follows:

[0152] First, the V component of the low-light image I in the HSV color space is downsampled to 1 / 4 of its original size and input into the traditional Retinex decomposition algorithm JieP to obtain the low-resolution illumination component L lr , low-resolution illumination component L lr It can effectively capture the basic features of the explicit prior constraints imposed on the illumination component in the JieP decomposition method;

[0153] Then how to use L lr The problem to be solved is to obtain the illumination component L at the original resolution. To this end, first lr Upsample to the original image size Then build a linear fusion model, the specific calculation is as follows:

[0154]

[0155] Among them, f g is the guided filter, V represents the V component of image I in the HSV color space, Indicates is the result of the guided filtering operation on V by the guided graph, V filter is the result of two consecutive guided filterings. α is a weight value that can dynamically adjust the smoothness of the illumination component L. In the present invention, α is set to 0.5. According to the characteristics of guided filtering, the above formula can make Vfilter (Filtered image) can obtain image content information from V, and also contains Gradient information of the illumination component L. By using the above steps, we can not only retain the explicit constraint characteristics imposed on the illumination component in the Jiep algorithm, but also quickly obtain L on a high-resolution image.

[0156] Step 1.3, after obtaining the illumination component L and the illumination color c, solve the reflection component R by the following formula:

[0157]

[0158] Indicates the division of corresponding elements in the matrix, ∈ is a constant used to prevent the denominator from being zero, and in this invention, the value is 10 -4 .

[0159] Step 2: Inspired by the human visual perception mechanism, the illumination component L obtained in step 1 is adjusted multiple times through the NR (Naka-Rushton) formula to obtain the adjusted illumination component L out , achieving brightness improvement and effectively solving the problem of uneven lighting, that is, the simultaneous presence of partial overexposure and underexposure. After two cycles of lighting adjustment, a good effect can be achieved. out Get the enhanced image I noise =c·L out ⊙R, which contains a lot of noise.

[0160] The specific calculation method for each lighting adjustment in step 2 is as follows:

[0161] Let the illumination component to be adjusted be L in , the adjusted illumination component is L out , the calculation process is as follows:

[0162]

[0163] Among them, σ(x, y) represents the adaptive adjustment factor at the pixel point (x, y), and σ(x, y) is adaptively fused with the local adjustment factor σ l (x, y) and a global adjustment factor σ g Find,ω l (x, y) and ω g (x, y) represent the weights of the local adjustment factor and the global adjustment factor at the pixel point (x, y); σ g Characterizes the brightness level of the input image, σ g By estimating L in The average brightness value of all pixels in is obtained, and η controls the global contrast of the image. Their specific calculation process is as follows:

[0164]

[0165]

[0166] In the above formula, mean(L in ) indicates L in The brightness values ​​of all pixels in are averaged. exp() represents the natural exponential function, that is, the exponential function with the real number e as the base. Since σ g The value range of is [0,1], then the value of η is in the interval [1,2.7]. k is a parameter that controls the contribution of global and local adjustment factors. It is set to 0.4 according to experience. σ1 and g σ2 The Gaussian filters have standard deviations σ1 and σ2 respectively. In the present invention, the values ​​of σ1 and σ2 are 2.5 and 1.5 respectively.

[0167] Step 3: Enhanced noise image I noise Denoising is performed to obtain the final enhanced result I en ,This step consists of two stages. The first stage provides a high-frequency noise suppression method based on a gate mechanism; the second stage provides a denoising network based on attention guidance.

[0168] The enhanced noisy image I in step 3 noise It is expressed as:

[0169] I noise =c·L out ⊙R (6)

[0170] In this step, we introduce noise The denoising process consists of two stages: the first stage provides a high-frequency noise suppression method based on a gate mechanism; the second stage provides a denoising network based on attention guidance. Through the operation of these two stages, the noise in the enhanced image can be removed. The final denoising result I en It is expressed as the following formula:

[0171]

[0172] First, use Gaussian filtering to filter I noise Decompose into high frequency parts and low frequency part g represents a high-frequency noise suppression module based on a gate mechanism, which aims to filter out I noise Part of the noise in, f Θ Represents a denoising network with network parameters Θ, which is used to reconstruct high-frequency details of the image and further suppress noise.

[0173] The denoising network structure based on the attention mechanism in step 3 is as follows:

[0174] The network adopts an encoder-decoder structure (such as Figure 1 As shown), the encoder is divided into a shallow encoding module and a deep encoding module. The shallow encoding module includes three 3×3 convolutional layers and a spatial information processing module (as shown in Figure 2 As shown in the figure, the convolution layer uses the LeakyReLU activation function, and the spatial information processing module uses a residual structure. The module first uses 3×3 convolution operation + ReLU activation function + 3×3 convolution operation to extract features, and then obtains the output features through residual connection; the deep encoding module is divided into two branches, spatial domain and frequency domain. The spatial branch uses 6 spatial information processing blocks to extract local structural features F spatial ; The frequency domain branch passes through 5 frequency domain information processing blocks (such as Figure 3 As shown in Figure 2, extract the global Fourier frequency domain information F fourier ,Each frequency domain information processing block first uses Fourier transform to convert the input features into Fourier space to obtain the amplitude and phase components, and then uses 1×1 convolution operation + LeakyReLU + 1×1 convolution operation to extract the features of the amplitude and phase components respectively, and then uses the inverse Fourier transform to convert the features corresponding to the amplitude and phase back to the spatial domain, and then obtains the output features through 3×3 convolution operation and residual connection;

[0175] Generally, for areas with less noise, local image features are sufficient to reconstruct details. On the contrary, for areas with high noise intensity, global features are needed to assist in reconstructing image details. Therefore, local structural features F can be fused spatial and the global Fourier frequency domain information F fourier , the fusion rules are as follows:

[0176] F=F spatial ×(1-Att)+F fourier ×Att

[0177]

[0178] F represents the fused output feature, Att represents the noise distribution attention map, Att represents the noise intensity, and the noise distribution map is approximated by the underexposure level of the image. Att is calculated using the results before and after the adjustment of the illumination component, and the value of Att is between [0, 1];

[0179] Then, the output feature F is input into the decoder to generate a noise-free high-frequency component, as shown in formula (7), and the obtained high-frequency component is combined with the low-frequency component obtained by the Gaussian filter in formula (7) to obtain the final denoised image;

[0180] When training the denoising network, the loss function used is defined as:

[0181]

[0182] φ is a pre-trained VGG network, I gt is a true normal lighting image, The operation refers to performing Fourier transform on the corresponding image and extracting the image amplitude information, ω 1 and ω 2 are weight factors, which are empirically set to 0.01 and 0.1 respectively.

Claims

1. A low-light image decoupling enhancement method based on the Retinex model, characterized in that: Follow the steps below to implement it: Step 1: Perform Retinex decomposition on the input low-light image in RGB space to obtain the lighting color c, the illumination component L and the reflection component R; In step 1, the input low-light image is retinex-decomposed in the RGB space as follows: I=c·L⊙R (1) Where I represents the low-light image, L∈R m×n is the illumination component, m represents the width of the low-light image, n represents the height of the image, R∈R m×n×3 is the reflection component, c=[c r , c g , c b ] represents the lighting color, where c r , c g , c b Respectively represent the illumination values ​​corresponding to the RGB color channels; The step 1 is specifically as follows: Step 1.1, the illumination color c is regarded as atmospheric light, and the dark channel prior principle is used to obtain the illumination color c; that is, first estimate the dark channel map of the low-light image, then select the pixels with the top 10% brightness values ​​in the dark channel map, and use the average brightness of these pixels on the RGB color channel as c r , c g , c b The value of Step 1.2: The process of solving the illumination component L is as follows: First, the V component of the low-light image I in the HSV color space is downsampled to 1 / 4 of its original size and input into the traditional Retinex decomposition algorithm JieP to obtain the low-resolution illumination component L lr , low-resolution illumination component L lr It can effectively capture the basic features of the explicit prior constraints imposed on the illumination component in the JieP decomposition method; First, L lr Upsample to the original image size Then build a linear fusion model, the specific calculation is as follows: Among them, f g is the guided filter, V represents the V component of image I in the HSV color space, Indicates is the result of the guided filtering operation on V by the guided graph, V filter is the result of two consecutive guided filtering, α is a weight value; Step 1.3, after obtaining the illumination component L and the illumination color c, solve the reflection component R by the following formula: Represents the division of corresponding elements in the matrix, ∈ is a constant used to prevent the denominator from being zero; Step 2: The illumination component L obtained in step 1 is adjusted through the NR formula for multiple cycles to obtain the adjusted illumination component L. out , based on L out Get the enhanced image I noise =c·L out ⊙R; Step 3: Enhanced noise image I noise Denoising is performed to obtain the final enhanced result I en .

2. The low-light image decoupling enhancement method based on the Retinex model according to claim 1, characterized in that: The specific calculation method for each illumination adjustment in step 2 is as follows: Let the illumination component to be adjusted be L in , the adjusted illumination component is L out , the calculation process is as follows: Among them, σ(x, y) represents the adaptive adjustment factor at the pixel point (x, y), and σ(x, y) is adaptively fused with the local adjustment factor σ l (x, y) and a global adjustment factor σ g Find,ω l (x,y) and ω g (x, y) represents the weights of the local adjustment factor and the global adjustment factor at the pixel point (x, y); σ g Characterizes the brightness level of the input image, σ g By estimating L in The average brightness value of all pixels in is obtained, and η controls the global contrast of the image. Their specific calculation process is as follows: In the above formula, mean(L in ) indicates L in The brightness values ​​of all pixels in are averaged. exp() represents the natural exponential function, that is, the exponential function with the real number e as the base. Since σ g The value range of is [0,1], then the value of η is in the interval [1, 2.7]. k is a parameter that controls the contribution of global and local adjustment factors. It is set to 0.4 according to experience. σ1 and g σ2 are Gaussian filters with standard deviations σ1 and σ2 respectively.

3. The low-light image decoupling enhancement method based on the Retinex model according to claim 2, characterized in that: The enhanced noise image I in step 3 noise It is expressed as: I noise =c·L out ⊙R (6) Denoising result I en It is expressed as the following formula: First, use Gaussian filtering to filter I noise Decompose into high frequency parts and low frequency part g represents a high-frequency noise suppression module based on a gate mechanism, which aims to filter out I noise Part of the noise in, f Θ Represents a denoising network with network parameters Θ, which is used to reconstruct high-frequency details of the image and further suppress noise.

4. The low-light image decoupling enhancement method based on the Retinex model according to claim 3, characterized in that: The specific operation of the high-frequency noise suppression module based on the gate mechanism in step 3 is as follows: The calculation formula for the high-frequency component after noise suppression is: represents the high-frequency component after noise suppression, ⊙ represents the multiplication of the corresponding elements in the matrix. From formula (7), we can see that the subsequent denoising network f Θ Aims to reconstruct the high-frequency details of the image, which are highly correlated with the low-frequency information, so the input of the network is Splicing is performed on the channel, which is expressed as: is the value of Function g, Concat means concatenation at the channel layer.

5. The low-light image decoupling enhancement method based on the Retinex model according to claim 4, characterized in that: The denoising network structure based on the attention mechanism in step 3 is as follows: The network adopts an encoder-decoder structure. The encoder is divided into a shallow encoding module and a deep encoding module. The shallow encoding module includes three 3×3 convolutional layers and a spatial information processing module. The convolutional layer uses the LeakyReLU activation function. The spatial information processing module adopts a residual structure. The module first uses 3×3 convolution operation + ReLU activation function + 3×3 convolution operation for feature extraction, and then obtains the output features through residual connection; The deep coding module is divided into two branches: spatial domain and frequency domain. The spatial branch uses 6 spatial information processing blocks to extract local structural features F spatial ; The frequency domain branch extracts the global Fourier frequency domain information F through 5 frequency domain information processing blocks fourier ,Each frequency domain information processing block first uses Fourier transform to convert the input features into Fourier space to obtain the amplitude and phase components, and then uses 1×1 convolution operation + LeakyReLU + 1×1 convolution operation to extract the features of the amplitude and phase components respectively, and then uses the inverse Fourier transform to convert the features corresponding to the amplitude and phase back to the spatial domain, and then obtains the output features through 3×3 convolution operation and residual connection; Fusion of local structural features F spatial and the global Fourier frequency domain information F fourier , the fusion rules are as follows: F=F spatial ×(1-At)+F fourier ×To F represents the fused output feature, Att represents the noise distribution attention map, Att represents the noise intensity, and the noise distribution map is approximated by the underexposure level of the image. Att is calculated using the results before and after the adjustment of the illumination component, and the value of Att is between [0, 1]; Then, the output feature F is input into the decoder to generate a noise-free high-frequency component, as shown in formula (7), and the obtained high-frequency component is combined with the low-frequency component obtained by the Gaussian filter in formula (7) to obtain the final denoised image; When training the denoising network, the loss function used is defined as: φ is a pre-trained VGG network, I gt is a true normal lighting image, The operation refers to performing Fourier transform on the corresponding image and extracting the image amplitude information, where ω1 and ω2 are weight factors.

Images