Dark light image enhancement method

Abstract

The invention provides a dark-light image enhancement method. According to the method, a dark-light image enhancement task in a RAW format is completed by constructing a two-stage image enhancement network. Firstly, in a denoising stage, a CNN-based denoising module is designed to process an input image, and the module is combined with a designed wavelet feature extraction module to capture noise features of different frequencies so as to enhance the denoising effect. And then color recovery and brightness correction are carried out on the denoised image. In the second stage, a Transform global correction module containing wavelet convolution is designed so as to better utilize colors to recover required low-frequency information. In order to realize dynamic brightness adjustment of different noise areas, a long and short distance interaction module introducing an SNR (Signal to Noise Ratio) graph is added in a network hidden layer in a second stage, and the module consists of a detail processing module and a Transform global correction module. And finally, the processed RAW format image is converted into an RGB format to be output.

Description

Technical Field

[0001] The invention belongs to the technical field of image enhancement, and in particular relates to a dark-light image enhancement method. Background Art

[0002] When using photographic equipment to shoot in environments with poor lighting conditions, such as at night when there is no light, in dim light, or in the shadow of a building, the images taken often perform poorly due to the limited dynamic range of the camera and reduced contrast. These photos in low-light environments usually show the characteristics of global dimness, blurred details, and color distortion, which has a negative impact on subsequent downstream tasks such as object detection and semantic segmentation. Image enhancement technology is used to improve image brightness and contrast, restore true colors, and eliminate potential noise, thereby significantly improving the visual experience of the image, playing an important preprocessing role for high-level visual tasks.

[0003] Compared with RGB images, RAW images have higher color depth, record more distinguishable low-intensity signals, and the noise distribution on RAW images is easy to handle before the image signal processing pipeline. For application scenarios that require high-quality images and fine control, RAW images are the first choice. Therefore, enhancing RAW low-light images can help users conduct more in-depth data mining and analysis in the industrial field.

[0004] Previous image enhancement methods mostly rely on distribution mapping techniques, such as curve transformation and histogram equalization, to improve the pixel distribution of the image. These methods have limitations in accurately capturing and using semantic information because they fail to establish effective models for specific lighting conditions and ignore the intrinsic connections between image pixels. With the rise of convolutional neural networks (CNNs), deep learning-based image enhancement schemes have shown advantages over traditional model-driven methods not only in processing speed but also in the quality of enhanced images. CNN networks are good at learning local features and focusing on detail features in image enhancement tasks, but they have limitations in capturing long-range correlations and non-local self-similarity, which is crucial for image restoration. Currently, most RAW image enhancements use CNN networks to map RAW images to the RGB domain through color channel conversion to complete the enhancement task of RAW dark-light images. Transformer was first proposed in the field of natural language processing for machine translation. With the emergence of its various variants, Transformer has also been applied to various machine vision tasks, such as image classification, image restoration, and image synthesis. Through the self-attention mechanism, Transformer can capture the long-distance dependencies between various parts of the image, which is crucial for image enhancement. When processing images, it can better understand and process the global information of the image, thereby maintaining a more coherent visual effect when enhancing the image. Transformer can effectively integrate multi-scale and multi-level features, which is a key advantage for image enhancement. During the image enhancement process, Transformer is able to fuse features at different levels to generate higher quality output images.

[0005] Wavelet transform theory inherits the idea of ​​Fourier transform theory and provides a more flexible and localized method to analyze the characteristics of signals. It is used to analyze data in both the time domain and the frequency domain for signals and images. Wavelet transform starts with a function called mother wavelet ψ(t), which is a waveform of finite length whose energy is concentrated in a finite interval of time. The mother wavelet must meet certain conditions to ensure that the wavelet transform is reversible. The wavelet transform function is shown below:

[0006]

[0007] Discrete wavelet transform is a type of wavelet transform. Discrete wavelet transform discretizes the scale and displacement of wavelet function, making the transform more efficient in calculation and suitable for practical application. Discrete wavelet transform performs multi-resolution analysis by decomposing the signal into approximate coefficients and detail coefficients of different scales. The approximate coefficient captures the low-frequency components of the signal, while the detail coefficient captures the high-frequency details of the signal. In the process of image enhancement, wavelet transform can be used to remove or reduce noise in the image. This is because noise often appears on the high-frequency detail layer. Through threshold processing, the coefficients of these layers can be selectively reduced to achieve the purpose of noise reduction. Wavelet transform allows the algorithm to adaptively adjust the enhancement parameters according to the content of the image, which means that different enhancement strategies can be used for different types of images or different parts of the image. Wavelet transform can provide localized information of the signal in time and frequency. It can enhance specific areas of the image without affecting other parts, which is very useful when enhancing specific areas of interest in the image.

[0008] In order to better combine the advantages of CNN network in processing detailed features and Transformer in processing global information, the present invention proposes a multi-stage RAW format dark light image enhancement network combining CNN and Transformer. According to the characteristics of the task, the present invention designs a denoising module based on CNN in the denoising stage, and introduces wavelet information to enhance the frequency characteristics. In the color and white balance restoration stage, a correction module based on Transformer is designed, and the color and white balance are dynamically adjusted using the SNR map. Summary of the invention

[0009] In view of this, the present invention aims to overcome the deficiencies of the above-mentioned problems in the prior art and proposes a dark light image enhancement method.

[0010] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0011] A first aspect of the present invention provides a dark light image enhancement method, comprising:

[0012] Construct a two-stage U-Net network. The first-stage U-Net network includes encoder 1 and decoder 1. The second-stage U-Net network includes encoder 2, long-short-range interaction module, and decoder 2. The encoder 1 includes a 4-layer network, which is composed of the designed denoising module and wavelet feature enhancement module stacked 4 times and connected by 3 layers of downsampling modules. The decoder 1 includes a 4-layer network, which is composed of the designed denoising module and wavelet feature enhancement module stacked 4 times and connected by 3 layers of upsampling modules. The encoder 2 includes a 4-layer network, and its structure is the same as that of the encoder 1. The long-short-range interaction module is located in the hidden layer part of the second stage, which is used to introduce the SNR map that guides the image color and white balance restoration. It is composed of a designed 1-layer Transformer-based correction module (long-range) and a 3-layer CNN detail processing (short-range) module in parallel. The decoder 2 includes a 4-layer network, which is composed of the designed Transformer-based correction module stacked 4 times and connected by 3 layers of upsampling modules.

[0013] Acquire the image to be enhanced and perform amplification processing, and obtain an SNR map of the amplified image to be enhanced;

[0014] The amplified image to be enhanced is input to encoder 1 for denoising, wavelet feature enhancement, downsampling, and output image features.

[0015] The image features obtained after passing through encoder 1 are transmitted to decoder 1, and decoder 1 is used to perform denoising, wavelet feature enhancement, and upsampling processing;

[0016] The image features after each layer of the decoder 1 are fused with the image to be enhanced after the amplification process and then output to the encoder 2, which performs denoising, wavelet feature enhancement, downsampling, and outputs the image features;

[0017] The image features obtained after encoder 2 are weighted fused with the signal-to-noise ratio (SNR) map of the unenhanced image, and the adjusted features are then input into the hidden layer. Then, these features are further extracted by the long-short range interaction module, and the optimized image features are output;

[0018] It is used to restore the color and white balance of the image using decoder 2, convert the color channels of the image after upsampling, and output the enhanced image.

[0019] Furthermore, the denoising module is implemented as follows:

[0020] I 1 =f CCC (f Dc (I in ));

[0021]

[0022] Among them, I 1 is the input feature, I out is the output feature, f DC It is a parallel 7×7 and 3×3 convolution operation, f CGC is the nonlinear operation of convolution and GELU function, represents element-wise addition, ⊙ represents element-wise multiplication, and σ(·) represents that the feature is passed through an activation function symmetric about the origin to obtain the attention map.

[0023] Furthermore, the wavelet feature enhancement module is implemented as follows:

[0024]

[0025] F in is the input feature, F out is the output feature, DWT and IDWT represent wavelet transform and inverse wavelet transform respectively, f CD is the feature enhancement process, f CGC is a fully connected layer.

[0026] Further, the obtaining of the SNR map of the amplified image to be enhanced includes:

[0027] For the input image Input in the first stage, calculate its gray value I y , and then calculate its SNR graph S. The process is as follows:

[0028]

[0029] where f BLUR is the Gaussian blur calculation, f ANS To calculate the absolute value, I y After denoising by 5×5 Gaussian blur convolution block, the N is the estimated noise distribution.

[0030] A second aspect of the present invention provides a low-light image enhancement device, comprising:

[0031] The first processing unit is used to construct a two-stage U-Net network. The U-Net network of the first stage includes an encoder 1 and a decoder 1. The U-Net network of the second stage includes an encoder 2, a Transformer correction module, a detail processing module, a long-short-range interaction module, and a decoder 2. The encoder 1 includes a 4-layer network, which is composed of the designed denoising module and wavelet feature enhancement module stacked 4 times and connected by 3 layers of downsampling modules. The decoder 1 includes a 4-layer network, which is composed of the designed denoising module and wavelet feature enhancement module stacked 4 times and connected by 3 layers of upsampling modules. The encoder 2 includes a 4-layer network, and its structure is the same as that of the encoder 1. The long-short-range interaction module is located in the hidden layer part of the second stage, and is used to introduce an SNR map to guide the image color and white balance restoration. It is composed of a designed 1-layer Transformer-based correction module (long-range) and a 3-layer CNN detail processing (short-range) module in parallel. The decoder 2 includes a 4-layer network, which is composed of the designed Transformer-based correction module stacked 4 times and connected by 3 layers of upsampling modules.

[0032] The second processing unit is used to obtain the image to be enhanced and perform amplification processing, and obtain a signal-to-noise ratio (SNR) map of the amplified image to be enhanced;

[0033] The third processing unit is used to input the image to be enhanced after the amplification processing to the encoder 1 for denoising, wavelet feature enhancement, downsampling processing, and output image features.

[0034] The fourth processing unit is used to transfer the image features obtained after passing through the encoder 1 to the decoder 1, and use the decoder 1 to perform denoising, wavelet feature enhancement, and upsampling processing;

[0035] A fifth processing unit is used to perform feature fusion on the image features after each layer of the decoder 1 and the image to be enhanced after the amplification process, and then output the features to the encoder 2. The encoder 2 performs denoising, wavelet feature enhancement, and downsampling processes, and outputs the image features.

[0036] The sixth processing unit is used to perform weighted fusion of the image features obtained after processing by encoder 2 with the signal-to-noise ratio (SNR) map of the unenhanced image, and the adjusted features are then input into the hidden layer. Then, these features are further extracted through the long-short range interaction module, and the optimized image features are output.

[0037] The seventh processing unit is used to restore the color and white balance of the image using the decoder 2, convert the color channels of the image after upsampling, and output the enhanced image.

[0038] A third aspect of the present invention provides an electronic device, comprising a processor and a memory that is communicatively connected to the processor and is used to store instructions executable by the processor, wherein the processor is used to execute the above-mentioned low-light image enhancement method.

[0039] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program implements the above-mentioned low-light image enhancement method when executed by a processor.

[0040] Compared with the prior art, the dark light image enhancement method described in the present invention has the following advantages:

[0041] The multi-stage RAW format low-light image enhancement network proposed in the present invention combines the CNN and Transformer architectures. The CNN module designed by the present invention contains a parameter-free attention mechanism, and the Transformer-based global correction module designed fully utilizes the local detail processing advantages of CNN and the global feature processing advantages of Transformer, and specifically solves the noise removal and color and white balance restoration problems of RAW format low-light images.

[0042] In addition, the present invention also introduces a wavelet feature extraction block that can adaptively process the different frequency detail information in the RAW image to ensure that the image features at all levels can be properly enhanced. At the same time, in order to achieve dynamic optimization of the brightness of different regions, the present invention designs a long-short range interaction module guided by the signal-to-noise ratio (SNR) map, which further improves the image quality and visual effect.

[0043] In summary, the present invention provides a comprehensive and efficient solution to significantly improve the quality of RAW format low-light images. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0045] Figure 1 This is the overall network structure diagram designed by the present invention for RAW format dark light image enhancement;

[0046] Figure 2 It is a schematic diagram of a CNN denoising module based on a parameter-free attention mechanism designed by the present invention;

[0047] Figure 3 Schematic diagram of the wavelet feature enhancement module designed by the present invention integrating coordinate attention;

[0048] Figure 4It is a schematic diagram of the Transformer global correction module including wavelet convolution designed by the present invention;

[0049] Figure 5 It is a schematic diagram of the CNN detail processing module designed by the present invention;

[0050] Figure 6 It is an algorithm flow chart of a dark light image enhancement method designed by the present invention. DETAILED DESCRIPTION

[0051] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0052] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0053] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0054] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0055] Embodiment 1:

[0056] like Figure 6 As shown, a dark light image enhancement method comprises:

[0057] Construct a two-stage dark light image enhancement network. Figure 1 shown.

[0058] The first stage is the denoising stage, which adopts the U-net architecture. Its encoder and decoder are composed of a CNN-based denoising module and a wavelet transform feature enhancement module stacked alternately 4 times, and connected through upsampling and downsampling layers. The denoising module adopts a multi-scale parallel CNN, which can effectively capture various noise features of RAW images. In order to further improve the local feature recognition performance, the present invention adds a parameter-free attention mechanism to the denoising module to optimize the increased computational cost brought by the Transformer. In view of the diversity and complexity of RAW image noise, the present invention adopts a wavelet transform feature enhancement module to dynamically extract high-frequency details and low-frequency structures in the image, so that the model can effectively distinguish between noise and signals.

[0059] The second stage is the white balance and color restoration stage. The encoder part of the U-net used is the same as the first stage, which is designed to further denoise the remaining noise. The decoder part is composed of four stacked Transformer correction modules fused with wavelet convolution. It can not only capture a wider range of global context information, but also enhance the focus on low-frequency features rich in color information through wavelet convolution, thereby improving the effects of white balance and color restoration. In addition, a long-short-range interaction module that introduces a signal-to-noise ratio (SNR) map is added to the hidden layer of the U-net to adaptively adjust the brightness enhancement effect of areas with different noise levels.

[0060] The input image is enhanced by building a two-stage U-Net network. The specific implementation steps are as follows:

[0061] The first step is to input the initial RAW format dark light image, enlarge the input image, and pass it to the encoder part of the first stage.

[0062] After the RAW format dark light image is passed to encoder 1, it first enters the CNN denoising module for denoising. The input features are processed in parallel using 7×7 and 3×3 depthwise separable convolutions to extract information of different scales of the image, such as Figure 2 As shown. The features are then processed nonlinearly by an MLP layer consisting of a 1×1 convolution and a GELU function. Then, a parameter-free attention mechanism is introduced, which dynamically generates an attention map of image features by using a convolution kernel combined with an activation function. The present invention uses an origin-symmetric nonlinear activation function to generate an attention map, and identifies key areas based on the output value of the feature map after convolution, thereby effectively reducing the additional parameter burden brought by the traditional attention mechanism. Specifically, the input feature I in After being processed by the denoising module, I out The process is expressed as:

[0063] I 1 =f CGC (f DC (Iin ));

[0064]

[0065] Among them, I 1 is the input feature, I out is the output feature, f DC It is a parallel 7×7 and 3×3 convolution operation, f CGC is the nonlinear operation of convolution and GELU function, represents element-wise addition, ⊙ represents element-wise multiplication, and σ(·) represents that the feature is passed through an activation function symmetric about the origin to obtain the attention map.

[0066] Next, we will elaborate on the role of the parameter-free attention mechanism in the denoising module. During the model training process, the gradient accumulation generated by the denoising module can be expressed as:

[0067]

[0068] Where L is the loss during training, I in is the input image feature, D is all the convolution modules in the denoising module, and F D It is a convolution operation on image features. The gradient generated for each training. By introducing the parameter-free attention map and superimposing it with the input residual connection, the gradient accumulation can be adjusted to:

[0069]

[0070] Where σ°(·) and σ(·) are the center-symmetric attention maps generated by the activation function. The center-symmetric activation function satisfies the following formula:

[0071] |σ(x)|=|σ(|x|)|=|σ(-x)|

[0072] When extracting feature maps through convolution operations, the absolute value of the gradient represents the size of the feature quantity. Therefore, in areas with rich feature information, σ can be reduced by a larger I 1 The output obtains a larger eigenvalue, and thus the gradient value is also larger. During the training process, the gradient of the information-rich area is amplified, so that the model gives more attention weight to the information-rich area. In addition, by introducing residual connections to avoid information loss problems, the output of the denoising module is (I 1 ⊕I in )⊙σ(I 1 ). The intensity of each pixel in the generated self-attention map is proportional to the feature recognition ability in the denoising process, helping the denoising module to achieve dynamic denoising based on the noise intensity level.

[0073] The image features after denoising are passed to the wavelet feature enhancement module to enhance the features of the image features at different frequencies. Figure 4 As shown in Figure 1, the wavelet feature enhancement module uses discrete wavelet transform (DWT) to decompose the different frequency components of the signal, thereby enhancing the extraction of frequency features of different intensities. First, for a given image input feature Discrete wavelet transform is used for decomposition. The decomposition process is as follows:

[0074] F LL ,{F LH ,F HL ,F HH = DWT(F in )

[0075] The decomposed image features are divided into four sub-bands: F LL (low frequency component, containing color information), (High frequency components, reflecting global structure and detail information). The present invention uses the coordinate attention mechanism to perform feature weighting on the four frequency sub-bands obtained after decomposition to generate enhanced wavelet features. This adapts to the different noise distribution and intensity in the image. The coordinate attention mechanism can efficiently identify and enhance the local details of the features after wavelet transformation, while suppressing the noise in the high-frequency details. The present invention applies inverse wavelet transformation to the enhanced wavelet features of each frequency band and reconstructs them into weighted features in the spatial domain. The conversion from frequency domain information to spatial domain structural information is realized.

[0076] In addition, in order to compensate for the information loss caused by wavelet transform, the input F in As the residual feature and F E Finally, the output is obtained through the MLP layer composed of the convolution layer and the GELU function. The action process of the wavelet feature enhancement module is expressed as the following formula:

[0077]

[0078] F in is the input feature, F out is the output feature. DWT and IDWT represent wavelet transform and inverse wavelet transform respectively. CD is the feature enhancement process of coordinate attention, f CGC is a fully connected layer.

[0079] The image is then downsampled to compress the image features, the image size is halved, and the number of channels is doubled. After undergoing four consecutive denoising, wavelet feature enhancement, and three downsampling processes, the image features are output by encoder 1.

[0080] In the second step, the image features obtained after encoder 1 are passed to decoder 1. First, the image features are input into the denoising module. After denoising, wavelet feature enhancement is performed. The processing process is the same as the first step. Then upsampling is performed to restore the spatial dimension of the image feature map. The image size is doubled and the number of channels is halved. The above denoising, wavelet feature enhancement and downsampling are performed 4 times in succession, and finally the image features are output to encoder 2.

[0081] In the third step, the image features output from decoder 1 are fused with the RAW format dark light image after amplification and then passed to encoder 2. In order to further process the remaining noise, the processing process of encoder 2 is similar to that of encoder 1. The image features are continuously denoised 4 times, wavelet feature enhanced and downsampled 3 times, and the processed image features are output to encoder 2.

[0082] The fourth step is to calculate the gray value I of the input image Input in the first stage y , and then calculate the eigenvalue S of its SNR graph. The process is shown in the following formula:

[0083]

[0084] Among them, f BLUR is the Gaussian blur calculation, f ABS To calculate the absolute value, I y After denoising by 5×5 Gaussian blur convolution block, the N is the estimated noise distribution. Then, the output feature f after encoder 2 is 2 They are respectively input into the long-short-range interaction module composed of the Transformer's global correction module and the 3×3 convolution detail processing module.

[0085] The following is a detailed description of the working process of Transformer's global correction module.

[0086] like Figure 4 As shown in the figure, in the multi-head self-attention part of the Transformer's generated attention map, the present invention adopts the transposed attention suitable for image restoration tasks. First, 1x1 convolution is used to aggregate the cross-channel information of each pixel. In order to prevent the model from being over-parameterized and expand the receptive field of convolution, wavelet convolution is used instead of traditional deep convolution to help better obtain the low-frequency components required for color restoration. The features after convolution processing are reorganized into And get the attention map through the operation of Q and V The calculation formula is:

[0087]

[0088] where γ is for K T ·Q scaling factor introduced for numerical stabilization.

[0089] Next, for the transposed attention map A T Perform matrix point multiplication A with V T V and point-wise convolution to get For its feedforward network part, the present invention designs a dual-scale gated feedforward module that also incorporates wavelet convolution to extract multi-range color information, and integrates two different multi-scale convolution paths during the transmission process by utilizing the expansion mechanism.

[0090] The feedforward network propagation process is expressed as follows:

[0091] O a ,O b =f Split (f PC (O 1 ))

[0092] O out =f PU ((f Mish (f WT3 (O b )))⊙(f WT5 (O a )))

[0093] where f PC is a 1×1 convolution and pixel rearrangement process, f Split Enter O 1 Separate into two equal parts according to the number of channels O a ,O b , f WT3 is a 3×3 wavelet convolution of the feature, f WT5 is a 5×5 wavelet convolution of the feature, f Mish is the Mish activation function, f PU It is pixel inverse rearrangement and 1×1 convolution.

[0094] The entire propagation process of the TWC module can be described as:

[0095]

[0096] where f LN is the layer normalization, f MDTA is the module MDTA self-attention calculation, f DSFN It is the calculation process of the dual-scale feedforward network.

[0097] f 2 Input the global correction module to extract its long-distance features and obtain fL ;f 2 Input the short-range features of the detail processing module extractor stacked three times, and the detail processing module obtains f s .like Figure 5 As shown, the detail processing module consists of a simple two-layer 3×3 convolution and GELU function.

[0098] Then, the SNR map S is adjusted to the same size as the two eigenvalues ​​and normalized to obtain S' as the weight to fuse f L and f s The weighted feature F is obtained, and the specific process is expressed as follows:

[0099] F=f S ×S'+f L ×(1-S');

[0100] Finally, the designed Transformer correction module is stacked with four layers in the decoder 2 part of the second-stage U-shaped network to perform white balance and color restoration on feature F. At the same time, the spatial resolution of the image is gradually improved through upsampling. After the restoration is completed, the RAW domain image is converted into an RGB domain image through color channel transformation for output.

[0101] Embodiment 2:

[0102] A low-light image enhancement device, comprising:

[0103] The first processing unit is used to construct a two-stage U-Net network. The U-Net network of the first stage includes an encoder 1 and a decoder 1. The U-Net network of the second stage includes an encoder 2, a long-short-range interaction module, and a decoder 2. The encoder 1 includes a 4-layer network, which is composed of a denoising module and a wavelet feature enhancement module stacked 4 times and connected by 3 layers of downsampling modules. The decoder 1 includes a 4-layer network, which is composed of a denoising module and a wavelet feature enhancement module stacked 4 times and connected by 3 layers of upsampling modules. The encoder 2 includes a 4-layer network, and its structure is the same as that of the encoder 1. The long-short-range interaction module is located in the hidden layer part of the second stage, and is used to introduce an SNR map for guiding image color and white balance restoration. It is composed of a 1-layer Transformer-based correction module and a 3-layer CNN detail processing module in parallel. The decoder 2 includes a 4-layer network, which is composed of a Transformer-based correction module stacked 4 times and connected by 3 layers of upsampling modules.

[0104] The second processing unit is used to obtain the image to be enhanced and perform amplification processing, and obtain an SNR map of the amplified image to be enhanced;

[0105] The third processing unit is used to input the image to be enhanced after the amplification processing to the encoder 1 for denoising, wavelet feature enhancement, downsampling processing, and output image features.

[0106] The fourth processing unit is used to transfer the image features obtained after passing through the encoder 1 to the decoder 1, and use the decoder 1 to perform denoising, wavelet feature enhancement, and upsampling processing;

[0107] A fifth processing unit is used to perform feature fusion on the image features after each layer of the decoder 1 and the image to be enhanced after the amplification process, and then output the features to the encoder 2. The encoder 2 performs denoising, wavelet feature enhancement, and downsampling processes, and outputs the image features.

[0108] The sixth processing unit is used to perform weighted fusion of the image features obtained after the encoder 2 is processed and the signal-to-noise ratio map of the unenhanced image. The adjusted features are then input into the hidden layer. These features are further extracted through the long-short range interaction module, and the optimized image features are output;

[0109] The seventh processing unit is used to restore the color and white balance of the image using the decoder 2, convert the color channels of the image after upsampling, and output the enhanced image.

[0110] Embodiment three:

[0111] An electronic device comprises a processor and a memory which is communicatively connected with the processor and is used to store instructions executable by the processor, wherein the processor is used to execute the above-mentioned low-light image enhancement method.

[0112] Embodiment 4:

[0113] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned low-light image enhancement method is implemented.

[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A dark light image enhancement method, characterized in that: include: A two-stage U-Net network is constructed. The U-Net network of the first stage includes an encoder 1 and a decoder 1. The U-Net network of the second stage includes an encoder 2, a long-short-range interaction module, and a decoder 2. The encoder 1 includes a 4-layer network, which is composed of a denoising module and a wavelet feature enhancement module stacked 4 times and connected by 3 layers of downsampling modules. The decoder 1 includes a 4-layer network, which is composed of a denoising module and a wavelet feature enhancement module stacked 4 times and connected by 3 layers of upsampling modules. The encoder 2 includes a 4-layer network, and its structure is the same as that of the encoder 1. The long-short-range interaction module is located in the hidden layer part of the second stage, and is used to introduce an SNR map for guiding image color and white balance restoration. It is composed of a 1-layer Transformer-based correction module and a 3-layer CNN detail processing module in parallel. The decoder 2 includes a 4-layer network, which is composed of a Transformer-based correction module stacked 4 times and connected by 3 layers of upsampling modules. Acquire the image to be enhanced and perform amplification processing, and obtain an SNR map of the amplified image to be enhanced; The amplified image to be enhanced is input to encoder 1 for denoising, wavelet feature enhancement, downsampling, and output image features. The image features obtained after passing through encoder 1 are transmitted to decoder 1, and decoder 1 is used to perform denoising, wavelet feature enhancement, and upsampling processing; The image features after each layer of the decoder 1 are fused with the image to be enhanced after the amplification process and then output to the encoder 2, which performs denoising, wavelet feature enhancement, downsampling, and outputs the image features; The image features obtained after encoder 2 processing are weighted fused with the signal-to-noise ratio map of the unenhanced image. The adjusted features are then input into the hidden layer. These features are further extracted through the long-short range interaction module to output the optimized image features. Decoder 2 is used to restore the image color and white balance, and after upsampling, the image is converted into color channels to output an enhanced image.

2. The dark light image enhancement method according to claim 1, characterized in that: The denoising module implementation process is as follows: I1=f CGC (f DC (I in )); I out =(I1⊕I in )⊙σ(I1); Among them, I1 is the input feature, I out is the output feature, f DC It is a parallel 7×7 and 3×3 convolution operation, f CGC It is a nonlinear operation of convolution and GELU function, ⊕ represents element-wise addition, ⊙ represents element-wise multiplication, and σ(·) represents that the feature is passed through an activation function symmetric about the origin to obtain the attention map.

3. The dark light image enhancement method according to claim 1, characterized in that: The wavelet feature enhancement module implementation process is as follows: F out =f CGC (IDWT(f CD (DWT(F in ))⊕F in ); F in is the input feature, F out is the output feature, DWT and IDWT represent wavelet transform and inverse wavelet transform respectively, f CD is the feature enhancement process, f CGC is a fully connected layer.

4. The dark light image enhancement method according to claim 1, characterized in that: The step of obtaining the SNR graph of the amplified image to be enhanced comprises: For the input image Input in the first stage, calculate its gray value I y , and then calculate its SNR graph S. The process is as follows: where f BLUR is the Gaussian blur calculation, f ABS To calculate the absolute value, I y After denoising by 5×5 Gaussian blur convolution block, the N is the estimated noise distribution.

5. A low-light image enhancement device, characterized in that: include: The first processing unit is used to construct a two-stage U-Net network. The U-Net network of the first stage includes an encoder 1 and a decoder 1. The U-Net network of the second stage includes an encoder 2, a long-short-range interaction module, and a decoder 2. The encoder 1 includes a 4-layer network, which is composed of a denoising module and a wavelet feature enhancement module stacked 4 times and connected by 3 layers of downsampling modules. The decoder 1 includes a 4-layer network, which is composed of a denoising module and a wavelet feature enhancement module stacked 4 times and connected by 3 layers of upsampling modules. The encoder 2 includes a 4-layer network, and its structure is the same as that of the encoder 1. The long-short-range interaction module is located in the hidden layer part of the second stage, and is used to introduce an SNR map for guiding image color and white balance restoration. It is composed of a 1-layer Transformer-based correction module and a 3-layer CNN detail processing module in parallel. The decoder 2 includes a 4-layer network, which is composed of a Transformer-based correction module stacked 4 times and connected by 3 layers of upsampling modules. The second processing unit is used to obtain the image to be enhanced and perform amplification processing, and obtain an SNR map of the amplified image to be enhanced; The third processing unit is used to input the image to be enhanced after the amplification processing to the encoder 1 for denoising, wavelet feature enhancement, downsampling processing, and output image features. The fourth processing unit is used to transfer the image features obtained after passing through the encoder 1 to the decoder 1, and use the decoder 1 to perform denoising, wavelet feature enhancement, and upsampling processing; A fifth processing unit is used to perform feature fusion on the image features after each layer of the decoder 1 and the image to be enhanced after the amplification process, and then output the features to the encoder 2. The encoder 2 performs denoising, wavelet feature enhancement, and downsampling processes, and outputs the image features. The sixth processing unit is used to perform weighted fusion of the image features obtained after being processed by the encoder 2 and the signal-to-noise ratio map of the unenhanced image. The adjusted features are then input into the hidden layer. These features are further extracted through the long-short range interaction module, and the optimized image features are output; The seventh processing unit is used to restore the color and white balance of the image using the decoder 2, convert the color channels of the image after upsampling, and output the enhanced image.

6. An electronic device, comprising a processor and a memory connected to the processor for storing instructions executable by the processor, characterized in that: The processor is used to execute a dark light image enhancement method as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, a dark-light image enhancement method according to any one of claims 1 to 4 is implemented.

Images