Low-light image enhancement method of noise adaptive network based on signal-to-noise ratio guidance

Abstract

A low-light image enhancement method based on a signal-to-noise ratio guided noise adaptive network, comprising 1: constructing a low-light image enhancement system based on the SNA-Net network; 2: the image acquisition module acquires a low-light image; 3: the input layer of the SNA-Net acquires the low-light image and transmits it to the CNN encoder, the SNA encoder and the signal-to-noise ratio image calculation module; 4: the signal-to-noise ratio image calculation module calculates the low-light image to obtain a signal-to-noise ratio image; 5: the CNN encoder encodes the low-light image at a short distance to obtain short-distance features; the SNA encoder encodes the low-light image at a long distance to obtain long-distance features; 6: the convolution residual block performs convolution residual operation on the short-distance features to obtain convolution residual data; the feature fusion module performs feature fusion operation on the short-distance features and the long-distance features to obtain fused features; 7: the CNN decoder decodes the convolution residual data and the fused features to obtain an enhanced image. Effect: clear image enhancement can be performed under low-light conditions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image enhancement, in particular to a low-light image enhancement method based on a noise adaptive network guided by signal-to-noise ratio. BACKGROUND

[0002] Images captured under low-light conditions are affected by poor visibility. On the one hand, this affects the quality of human visual perception; on the other hand, it leads to a decline in the performance of other advanced visual tasks, such as object detection, image segmentation and recognition.

[0003] There are many advanced methods for low-light image enhancement. These methods can generally be divided into non-learning-based methods and learning-based methods. In the early stage, more methods only focus on enhancing brightness, contrast and color factors. In recent years, some work has begun to consider the noise problem in the enhancement process by exploring the signal-to-noise ratio prior to guide the model to focus on different regions of the image. Although noise is still a difficult problem to overcome, considering the information difference of different light regions, corresponding methods should be considered for image enhancement. As shown in Figure 1 As shown, the well-lit (green region) region tends to have more information, and local information is sufficient for local LLIE. In contrast, the very low light region (red region) is dominated by noise, and has less information, and non-local information is needed to effectively enhance the image.

[0004] Due to the lack of information in the very low signal-to-noise ratio noise region, and the fact that convolutional neural network (CNN) based methods are not good at handling long distance dependencies, relying solely on a local learning paradigm based on CNN is not sufficient to reconstruct high-quality images. Therefore, it is beneficial to use a deep learning model based on the self-attention mechanism, Transformer, to capture non-local information in these regions. At the same time, in regions with a high signal-to-noise ratio, local information is already sufficient, and the high computational cost of traditional transformers limits their widespread application, and using CNN is sufficient to achieve the desired reconstruction effect. Therefore, a hybrid model that combines the advantages of CNN and transformers brings new possibilities for low-light enhancement. However, there are still obstacles in designing an effective attention mechanism, because the standard transformer uses dense attention computation, which introduces noise interaction. Recently, in order to solve this problem, several sparse transformer methods have emerged. Methods based on Top-k selection attention mechanism and adaptive sparse attention mechanism have made efforts in this regard. Unfortunately, there is currently no method specifically for the low-light enhancement task that can efficiently filter out redundant information.

[0005] The prior art has the following disadvantages: The Transformer has important significance in low-light image enhancement. The low-light image has certain noise, especially in the extreme low-light region. Since the Transformer usually calculates the self-attention scores of all available labels, the low-light image enhancement method based on the Transformer is difficult to avoid the interference of noise, which is not conducive to clear image enhancement in low-light conditions. SUMMARY

[0006] The low-light image enhancement method based on the noise adaptive network guided by the signal-to-noise ratio provided by the application can achieve clear image enhancement in low-light conditions.

[0007] To achieve the above-mentioned purpose, the low-light image enhancement method based on the noise adaptive network guided by the signal-to-noise ratio provided by the application comprises the following steps:

[0008] Step 1: Construct a low-light image enhancement system based on a noise adaptive network guided by a signal-to-noise ratio, wherein the low-light image enhancement system is provided with an image acquisition module and a noise adaptive network SNA-Net guided by a signal-to-noise ratio, the noise adaptive network SNA-Net guided by the signal-to-noise ratio is provided with an input layer, the output end of the input layer is connected to the input ends of a CNN encoder, an SNA encoder and a signal-to-noise ratio map calculation module, the output ends of the CNN encoder, the SNA encoder and the signal-to-noise ratio map calculation module are connected to the input end of a feature fusion module, and the output end of the CNN encoder is also connected to the input end of a CNN decoder through a convolution residual block, and the input end of the CNN decoder is also connected to the output end of the feature fusion module.

[0009] Step 2: The image acquisition module acquires a low-light image I and transmits it to the noise adaptive network SNA-Net guided by the signal-to-noise ratio.

[0010] Step 3: The input layer of the noise adaptive network SNA-Net guided by the signal-to-noise ratio acquires the low-light image I and transmits it to the CNN encoder, the SNA encoder and the signal-to-noise ratio map calculation module.

[0011] Step 4: The signal-to-noise ratio map calculation module calculates the low-light image I to obtain a corresponding signal-to-noise ratio map S, and then transmits the signal-to-noise ratio map S to the feature fusion module and the SNA encoder.

[0012] Step 5: The CNN encoder encodes the low-light image I at a short distance to capture the local information of the low-light image I to obtain short-distance feature data a i , and transmits the short-distance feature data a to the feature fusion module and the convolution residual block.

[0013] The SNA encoder uses the signal-to-noise ratio map S as prior knowledge to calculate a self-attention score, and then performs long-distance encoding on the low-light image I to filter out noise and redundant information, thereby obtaining long-distance feature data b i , and is transmitted to a feature fusion module;

[0014] Step 6: The convolution residual block performs convolution residual operation on the short-distance feature data a i , to obtain convolution residual data d, and transmits it to a CNN decoder;

[0015] The feature fusion module performs feature fusion operation on the short-distance feature data a i and long-distance feature data b i , to obtain fusion feature data c i , and transmits it to the CNN decoder;

[0016] Step 7: The CNN decoder performs decoding operation on the convolution residual data d and the fusion feature data c i , to obtain an enhanced image

[0017] Through the above design, in the SNA-Net network, considering the signal-to-noise ratio change in different regions, a CNN is used for short-distance encoding in the CNN encoder, and a Transformer is used for long-distance encoding in the SNA encoder. In order to better integrate the feature information of the two, a signal-to-noise ratio guided feature fusion module SGFF is constructed to guide the integration process. The feature fusion module SGFF guides the adaptive fusion of features by the signal-to-noise ratio map S through two parallel encoders CNN block and SNA block, and then sends these features to the decoder through the residual connection to produce an enhanced image.

[0018] Through the above design, the present application not only can perform clear image enhancement under low light conditions, but also can use information most effectively, filter out irrelevant information, and retain the feature representation of key information as much as possible.

[0019] As a preferred: in the step 1, the CNN encoder is provided with a first CNN encoding block, a second CNN encoding block, a third CNN encoding block and a fourth CNN encoding block connected in sequence, the SNA encoder is provided with a first SNA block, a second SNA block, a third SNA block and a fourth SNA block connected in sequence, the feature fusion module is provided with a first feature fusion block, a second feature fusion block, a third feature fusion block and a fourth feature fusion block, and the CNN decoder is provided with a first CNN decoding block, a second CNN decoding block, a third CNN decoding block and a fourth CNN decoding block connected in sequence;

[0020] The output ends of the first CNN encoding block and the first SNA block are connected to the input ends of a first feature fusion block, and the output end of the first feature fusion block is connected to the input end of a fourth CNN decoding block;

[0021] The output ends of the second CNN encoding block and the second SNA block are connected to the input ends of a second feature fusion block, and the output end of the second feature fusion block is connected to the input end of a third CNN decoding block;

[0022] The output ends of the third CNN encoding block and the third SNA block are connected to the input ends of a third feature fusion block, and the output end of the third feature fusion block is connected to the input end of a second CNN decoding block;

[0023] The output ends of the fourth CNN encoding block and the fourth SNA block are connected to the input ends of a fourth feature fusion block, and the output end of the fourth feature fusion block is connected to the input end of a first CNN decoding block.

[0024] As a preferred: in the step 4, first calculate the gray image I corresponding to the low light image I g , and then calculate the signal-to-noise ratio image S according to the following formula, S ∈ R H×W ;

[0025]

[0026] Where Denoise() represents the denoising operation using the mean filter, abs represents the absolute value, and N ∈ R H×W The estimated noise image.

[0027] As a preferred: the first CNN encoding block, the second CNN encoding block, the third CNN encoding block and the fourth CNN encoding block have consistent structures and the same working logic, and are each provided with a 3 × 3 convolution layer and a down-sampling layer, and the output end of the 3 × 3 convolution layer is built-in with a LeakyReLU activation function.

[0028] Each level in the CNN encoder is implemented by a CNN block and a single convolution layer for down-sampling. The CNN block is composed of a 3 × 3 convolution layer and a LeakyReLU activation function, for capturing local information.

[0029] As a preferred: the first SNA block, the second SNA block, the third SNA block and the fourth SNA block have consistent structures and the same working logic, and are each provided with a noise adaptive self-attention module NASA and a dual-domain refinement feedforward network DRFN;

[0030] The noise adaptive self-attention module NASA in each SNA block calculates the self-attention score Attn of the corresponding SNA block by taking the signal-to-noise ratio map S as prior knowledge, and then encodes the output data of the low-light image I or the previous SNA block through the self-attention score Attn to obtain the self-attention feature of the current SNA block, and then further refines the self-attention feature through the dual-domain refinement feedforward network DRFN to obtain the long-distance feature data b of the current SNA block i .

[0031] The noise adaptive self-attention module NASA can eliminate the possibility of information integrity loss, retain as much valid information as possible, and filter out irrelevant features to the greatest extent. The dual-domain refinement feedforward network DRFN can refine the features in the spatial and frequency domains to eliminate potential redundant information, that is, enhance the features most useful for image restoration in both domains while suppressing potential redundant features.

[0032] The dual-domain refinement feedforward network DRFN suppresses redundant information in the channel dimension, while the noise adaptive self-attention module NASA suppresses noise interaction in irrelevant regions in the spatial dimension. The complementarity of the two components enables the SNA-Net to suppress irrelevant features to a certain extent while obtaining the most informative feature representation.

[0033] As a preferred embodiment, the noise adaptive self-attention module NASA obtains the self-attention feature of the current SNA block through the following steps:

[0034] Step A1: The first normalization layer in the noise adaptive self-attention module NASA obtains the output data of the low-light image I or the previous SNA block and generates tensor data, which is then passed to the query matrix Q encoding unit, the key matrix K encoding unit, the value matrix V encoding unit, and the first addition unit;

[0035] Step A2: The query matrix encoding unit encodes the tensor data through a first 1x1 convolutional layer and a first 3x3 deep convolutional layer to generate a query matrix Q, which is then passed to the first multiplication unit;

[0036] The key matrix K encoding unit encodes the tensor data through a second 1x1 convolutional layer and a second 3x3 deep convolutional layer to generate a key matrix K, which is then transposed and passed to the first multiplication unit;

[0037] The value matrix V encoding unit encodes the tensor data through a third 1x1 convolutional layer and a third 3x3 deep convolutional layer to generate a value matrix V, which is then passed to the second multiplication unit;

[0038] The expression of the query matrix Q, the key matrix K and the value matrix V is as follows:

[0039]

[0040] wherein W d represents a 1x1 point-wise convolution, W p represents a 3x3 depth-wise convolution, and X represents tensor data;

[0041] Step A3: the first multiplication unit multiplies the query matrix Q and the transposed key matrix K T to obtain a first multiplication matrix and passes it to the sparse self-attention branch and the dense self-attention branch;

[0042] Step A4: the dense self-attention branch calculates first attention data from the first multiplication matrix through a first Softmax function and passes it to a second addition unit;

[0043] The sparse self-attention branch uses the normalized signal-to-noise ratio map S' as a mask to filter out the areas in the first multiplication matrix with a signal-to-noise ratio value lower than a signal-to-noise ratio threshold, and then calculates the scores of the areas with a signal-to-noise ratio value higher than or equal to the signal-to-noise ratio threshold through a second Softmax function to obtain second attention data and pass it to the second addition unit;

[0044] Step A5: the second addition unit weights the first attention data with a first attention weight w1 and weights the second attention data with a second attention weight w2, and adds the weighted first attention score and the second attention score, and then multiplies it with the value matrix V through a second multiplication unit to obtain a self-attention score Attn, the expression is as follows:

[0045] Attn=w1*D attn +w2*S attn

[0046] D attn =Softmax(QK T / α)V

[0047] S attn =Softmax(QK T / α+(1-S′)σ)V

[0048] wherein α is a learnable scaling parameter for controlling the dot product size of the query matrix Q and the key matrix K; σ is a small negative scalar -1e9, D attn is the dense self-attention score, and S attnis the multiplication operation. At the end of the noise adaptive self-attention module, a position encoding with learnable parameters is added to produce the final output. This design can guarantee that both irrelevant feature interactions such as noise can be filtered out while having enough information features. That is, the model can well control the sparsity degree of the input tokens of a specific task.

[0049] Step A6: The second multiplication unit passes the self-attention score Attn to a fourth 1x1 convolutional layer for convolution operation, and then passes the convolution result to the tensor data through the first addition unit for element-wise addition to obtain the self-attention feature of the current SNA block and pass it to the dual-domain refinement feedforward network DRFN.

[0050] In order to weaken the negative impact of noise regions or other irrelevant information, while preventing the possibility of excessive sparsity, a noise adaptive self-attention module NASA is proposed. The noise adaptive self-attention module NASA adopts a channel-level self-attention mechanism to calculate the self-attention score in the channel dimension, thereby reducing the complexity. The NASA consists of a dense branch and a sparse branch, which can be adaptively fused. The sparse branch filters out tokens from low signal-to-noise ratio regions to avoid noise interference, while the dense branch ensures the integrity of the key information.

[0051] The sparse self-attention branch uses a normalized signal-to-noise ratio map as a mask to filter out low signal-to-noise ratio regions and only calculates the score of the relatively high signal-to-noise ratio region. This sparse method can filter out redundant information and irrelevant features. At the same time, considering the possibility of excessive sparsity, another dense self-attention branch is introduced. This branch uses standard softmax attention to retain complete information. The NASA adaptively obtains features from the two branches and propagates information flow through the network.

[0052] As preferred: the dual-domain refinement feedforward network DRFN further refines the self-attention feature by the following steps:

[0053] Step B1: the second normalization layer in the dual-domain refinement feedforward network DRFN performs a normalization operation on the self-attention feature, and then a fifth 1x1 convolutional layer is used to perform convolution operation on the normalized feature to obtain fifth convolutional data, and pass it to a sixth 1x1 convolutional layer, a fourth 3x3 deep convolutional layer and a fast Fourier transform unit FFT;

[0054] The expression of the fifth convolutional data is as follows:

[0055] Y' = W d (LN(Y))

[0056] wherein Y is a self-attention feature, LN is a layer normalization, Y' is the fifth convolutional data, W d represents a 1x1 point-wise convolution;

[0057] Step B2: the sixth 1x1 convolutional layer convolves the fifth convolutional data, and then the fifth 3x3 deep convolutional layer performs a deep convolutional operation to obtain the fifth deep convolutional data, and passes it to the first element-wise multiplication unit;

[0058] The expression of the fifth deep convolutional data is as follows:

[0059] Y'' = W p W d (Y')

[0060] wherein W p represents a 3x3 deep convolution, and Y'' represents the fifth deep convolutional data;

[0061] The fourth 3x3 deep convolutional layer performs a deep convolutional operation on the fifth convolutional data, and then performs an activation through the GELU activation function to obtain the first activation data, and passes it to the element-wise multiplication unit;

[0062] The expression of the first activation data is as follows:

[0063]

[0064] wherein, is the GELU activation function, and Y''' represents the first activation data;

[0065] Step B3: the fast Fourier transform unit FFT performs a fast Fourier transform on the fifth convolutional data to obtain FFT data, then the FFT data is element-wise multiplied with the weight w of the dual-domain refinement feedforward network DRFN through the second element-wise multiplication unit, then an inverse fast Fourier transform is performed through the inverse fast Fourier transform unit IFFT, and then an activation is performed through the GEGLU activation function to obtain the second activation data, and the second activation data is passed to the connection unit;

[0066] The expression of the second activation data is as follows:

[0067]

[0068] wherein, is the fast Fourier transform, is the inverse fast Fourier transform, w is the weight of the dual-domain refinement feedforward network DRFN, and Y f is the second activation data, and ζ() represents the GEGLU function;

[0069] The first element-by-element multiplication unit element-by-element multiplies the fifth deep convolution data and the first activation data to obtain first multiplication data and delivers the first multiplication data to a connection unit;

[0070] The expression of the first multiplication data is as follows:

[0071] Y s =Y″⊙Y″′

[0072] Wherein, ⊙ represents element-by-element multiplication, Y s is the first multiplication data;

[0073] Step B4: The connection unit performs a connection operation on the second activation data and the first multiplication data, and then performs a convolution operation through a seventh 1x1 convolution layer to obtain seventh convolution data and deliver the seventh convolution data to a third addition unit;

[0074] Step B5: The third addition unit element-by-element adds the self-attention feature and the seventh convolution data to obtain long-distance feature data b i of the current SNA block.

[0075] The expression of the long-distance feature data b i of the current SNA block is as follows:

[0076]

[0077] Wherein, represents a connection operation.

[0078] The dual-domain refinement feed-forward network DRFN refines features in the spatial domain and the frequency domain to enhance feature representation, thereby achieving better potential image enhancement. Specifically, in the frequency domain, the DRFN uses a learnable global filter to determine which low-frequency and high-frequency information should be retained to recover the underlying clear image. In the spatial domain, a gating mechanism is used to reduce redundant features.

[0079] As preferred: the first feature fusion block, the second feature fusion block, the third feature fusion block and the fourth feature fusion block have consistent structures and the same working logic, and are each provided with a third multiplication unit, a fourth multiplication unit, a fourth addition unit, a global average pooling layer, a first full connection layer, a second full connection layer, a fifth multiplication unit, a sixth multiplication unit, and a fifth addition unit, and the output end of the second full connection layer is built-in with a Sigmoid function;

[0080] In the step 6, the feature fusion block obtains corresponding short-distance feature data a i and long-distance feature data b i and performs a feature fusion operation to obtain fusion feature data c i, comprising the following steps:

[0081] Step C1: the third multiplication unit in the current feature fusion block multiplies the normalized signal-to-noise ratio map as a mask with the short-distance feature data a i to obtain third multiplication data and passes it to the fourth addition unit and the fifth multiplication unit;

[0082] The fourth multiplication unit in the current feature fusion block multiplies the normalized signal-to-noise ratio map with the long-distance feature data b i to obtain fourth multiplication data and passes it to the fourth addition unit and the sixth multiplication unit;

[0083] Step C2: the fourth addition unit element-wise adds the third multiplication data and the fourth multiplication data, then performs a global average pooling operation through a global average pooling layer, then performs a full connection operation through a first full connection layer and a second full connection layer in turn, and then obtains the short-distance feature fusion weight w a and the long-distance feature fusion weight w d after normalization through a Sigmoid function, and passes the short-distance feature fusion weight w a to the fifth multiplication unit and passes the long-distance feature fusion weight w d to the sixth multiplication unit;

[0084] Step C3: the fifth multiplication unit weights the third multiplication data through the short-distance feature fusion weight w a to obtain fifth multiplication data and passes it to the fifth addition unit;

[0085] The sixth multiplication unit weights the fourth multiplication data through the long-distance feature fusion weight w d to obtain sixth multiplication data and passes it to the fifth addition unit;

[0086] Step C4: the fifth addition unit element-wise adds the fifth multiplication data and the sixth multiplication data to obtain the feature fusion data c i of the current feature fusion block, expressed as follows:

[0087] c i = SGFF(a i ×(1-S′)+b i ×S′)

[0088] Wherein, SGFF represents a feature fusion operation, S' represents a normalized signal-to-noise ratio map, and 1-S' represents the normalized signal-to-noise ratio map as a mask.

[0089] Simply concatenating the features of the two parallel encoders can lead to an over-contribution of unimportant information. To solve this problem and naturally combine the features of the two encoders, a signal-to-noise ratio guided feature fusion module SGFF is developed to dynamically integrate the information flow. Since the signal-to-noise ratio map reflects the different noise levels of different image regions, SGFF can adaptively combine the long-distance features from the SNA block and the short-distance features from the CNN block.

[0090] Specifically, the feature fusion module uses the normalized signal-to-noise ratio map as the interpolation weight to selectively fuse the features, using their respective capabilities in global and local feature extraction to produce two merged features. Subsequently, further processing is performed by applying global average pooling to the features. Next, the dimensions are first reduced and then increased through two fully connected layers to enhance the expressiveness of the information. Finally, after softmax normalization, two weights are obtained, which are used to weight and sum the features to obtain the final fused features.

[0091] Advantages of the present application: A new hybrid CNN-Transformer dual encoder model is proposed, named signal-to-noise ratio guided noise adaptive network SNA-Net, which can adapt to the enhancement needs of different regions of the image by using the respective strengths of convolutional neural network CNN and deep learning model Transformer based on self-attention mechanism according to the signal-to-noise ratio changes of different illumination regions. SNA-Net mainly includes two components, noise adaptive self-attention module NASA and dual-domain refinement feedforward network DRFN. NASA adaptively calculates the attention scores from the dense and sparse branches, where the sparse branch filters out negative marker interactions from low signal-to-noise ratio regions guided by the signal-to-noise ratio map, and the dense branch ensures the integrity of the information. At the same time, DRFN eliminates feature redundancy in the dual domain to improve the recovery of the underlying clear image. In addition, in order to better integrate the feature information from CNN and Transformer, a signal-to-noise ratio guided feature fusion module SGFF is designed. BRIEF DESCRIPTION OF DRAWINGS

[0092] Figure 1 Structure diagram of signal-to-noise ratio guided noise adaptive network SNA-Net in the embodiment;

[0093] Figure 2 Structure diagram of noise adaptive self-attention module NASA in the embodiment;

[0094] Figure 3 Structure diagram of dual-domain refinement feedforward network DRFN in the embodiment;

[0095] Figure 4 Structure diagram of feature fusion block in the embodiment;

[0096] Figure 5 A comparison chart of the results of different models in the example on LOLv2-Real (top) and LOLv2-Synthetic (bottom);

[0097] Figure 6 A comparison chart of the quantitative results of different models in the example on the no-reference dataset. DETAILED DESCRIPTION

[0098] The application will be further described in conjunction with the drawings and specific examples. The following examples or drawings are used to illustrate the application, but not to limit the scope of the application.

[0099] As shown in FIG. 1: a low-light image enhancement method based on a signal-to-noise ratio guided noise adaptive network, comprising the following steps:

[0100] Step 1: Construct a low-light image enhancement system based on a signal-to-noise ratio guided noise adaptive network, which is provided with an image acquisition module and a signal-to-noise ratio guided noise adaptive network SNA-Net, wherein the input layer of the signal-to-noise ratio guided noise adaptive network SNA-Net is connected to the input ends of a CNN encoder, an SNA encoder and a signal-to-noise ratio map calculation module, the output ends of the CNN encoder, the SNA encoder and the signal-to-noise ratio map calculation module are connected to the input end of a feature fusion module, the output end of the CNN encoder is also connected to the input end of a CNN decoder through a convolution residual block, and the input end of the CNN decoder is also connected to the output end of the feature fusion module;

[0101] Step 2: The image acquisition module acquires a low-light image I and transmits it to the signal-to-noise ratio guided noise adaptive network SNA-Net;

[0102] Step 3: The input layer of the signal-to-noise ratio guided noise adaptive network SNA-Net acquires the low-light image I and transmits it to the CNN encoder, the SNA encoder and the signal-to-noise ratio map calculation module;

[0103] Step 4: The signal-to-noise ratio map calculation module calculates the low-light image I to obtain a corresponding signal-to-noise ratio map S, and then transmits the signal-to-noise ratio map S to the feature fusion module and the SNA encoder;

[0104] Step 5: The CNN encoder encodes the low-light image I at a short distance to capture the local information of the low-light image I to obtain short-distance feature data a i , and transmits it to the feature fusion module and the convolution residual block;

[0105] The SNA encoder uses the signal-to-noise ratio map S as prior knowledge to calculate a self-attention score, and then performs long-distance encoding on the low-light image I to filter out noise and redundant information, obtaining long-distance feature data b i , and passes it to a feature fusion module.

[0106] Step 6: The convolution residual block performs convolution residual operation on the short-distance feature data a i , obtains convolution residual data d, and passes it to a CNN decoder.

[0107] The feature fusion module performs feature fusion operation on the short-distance feature data a i and long-distance feature data b i , obtains fusion feature data c i , and passes it to the CNN decoder.

[0108] Step 7: The CNN decoder performs decoding operation on the convolution residual data d and fusion feature data c i , to obtain an enhanced image

[0109] For the low-light image I obtained by the image acquisition module, first use a learning-free method to obtain a signal-to-noise ratio map S. Then input I into two parallel encoders composed of CNN blocks and SNA blocks. Subsequently, use the signal-to-noise ratio map to guide the fusion process between the CNN blocks and the SNA blocks, and pass the fused features to the corresponding layers of the decoder through the jump connection to produce an enhanced image

[0110] This embodiment uses Charbonnier loss and perceptual loss to train the SNA-Net network, and the overall loss function is:

[0111]

[0112] where I' is the real data, ε is set to 10 -3 , λ is a hyperparameter, ‖‖2 represents the L2 norm, and ‖‖1 represents the L1 norm.

[0113] In the step 4, first calculate the gray image I g corresponding to the low-light image I, and then calculate the signal-to-noise ratio map S according to the following formula, S∈R H×W .

[0114]

[0115] Wherein, Denoise() represents a denoising operation using a mean filter, abs represents an absolute value, N is in R H×W estimated noise map.

[0116] The CNN encoder is provided with a first CNN encoding block, a second CNN encoding block, a third CNN encoding block and a fourth CNN encoding block connected in sequence, the SNA encoder is provided with a first SNA block, a second SNA block, a third SNA block and a fourth SNA block connected in sequence, the feature fusion module is provided with a first feature fusion block, a second feature fusion block, a third feature fusion block and a fourth feature fusion block, and the CNN decoder is provided with a first CNN decoding block, a second CNN decoding block, a third CNN decoding block and a fourth CNN decoding block connected in sequence.

[0117] The output ends of the first CNN encoding block and the first SNA block are connected to the input end of the first feature fusion block, and the output end of the first feature fusion block is connected to the input end of the fourth CNN decoding block.

[0118] The output ends of the second CNN encoding block and the second SNA block are connected to the input end of the second feature fusion block, and the output end of the second feature fusion block is connected to the input end of the third CNN decoding block.

[0119] The output ends of the third CNN encoding block and the third SNA block are connected to the input end of the third feature fusion block, and the output end of the third feature fusion block is connected to the input end of the second CNN decoding block.

[0120] The output ends of the fourth CNN encoding block and the fourth SNA block are connected to the input end of the fourth feature fusion block, and the output end of the fourth feature fusion block is connected to the input end of the first CNN decoding block.

[0121] The first CNN encoding block, the second CNN encoding block, the third CNN encoding block and the fourth CNN encoding block have consistent structures and same working logic, and are each provided with a 3*3 convolution layer and a down-sampling layer, and the output end of the 3*3 convolution layer is built-in with a LeakyReLU activation function.

[0122] The first SNA block, the second SNA block, the third SNA block and the fourth SNA block have consistent structures and same working logic, and are each provided with a noise adaptive self-attention module NASA and a dual-domain refinement feedforward network DRFN.

[0123] The noise adaptive self-attention module NASA in each SNA block calculates the self-attention score Attn of the corresponding SNA block by taking the signal-to-noise ratio map S as prior knowledge, and then encodes the output data of the low-light image I or the output data of the previous SNA block through the self-attention score Attn to obtain the self-attention feature of the current SNA block, and then further refines the self-attention feature through the dual-domain refinement feedforward network DRFN to obtain the long-distance feature data b of the current SNA block i .

[0124] As shown in FIG. 2, the noise adaptive self-attention module NASA obtains the self-attention feature of the current SNA block through the following steps:

[0125] Step A1: The first normalization layer in the noise adaptive self-attention module NASA obtains the output data of the low-light image I or the output data of the previous SNA block, and generates tensor data, which is then transmitted to the query matrix encoding Q unit, the key matrix K encoding unit, the value matrix V encoding unit and the first addition unit;

[0126] Step A2: The query matrix encoding unit encodes the tensor data through the first 1x1 convolution layer and the first 3x3 deep convolution layer to generate the query matrix Q, which is then transmitted to the first multiplication unit;

[0127] The key matrix K encoding unit encodes the tensor data through the second 1x1 convolution layer and the second 3x3 deep convolution layer to generate the key matrix K, which is then transmitted to the first multiplication unit after being transposed;

[0128] The value matrix V encoding unit encodes the tensor data through the third 1x1 convolution layer and the third 3x3 deep convolution layer to generate the value matrix V, which is then transmitted to the second multiplication unit;

[0129] The expressions for generating the query matrix Q, the key matrix K and the value matrix V are as follows:

[0130]

[0131] wherein W d represents a 1x1 pointwise convolution, W p represents a 3x3 deep convolution, and X represents tensor data;

[0132] Step A3: The first multiplication unit multiplies the query matrix Q and the transposed key matrix K T to obtain the first multiplication matrix, which is then transmitted to the sparse self-attention branch and the dense self-attention branch;

[0133] Step A4: the dense self-attention branch calculates first attention data according to the first multiplication matrix through a first Softmax function and passes the first attention data to a second addition unit;

[0134] The sparse self-attention branch uses the normalized signal-to-noise ratio map S' as a mask to filter out the areas in the first multiplication matrix with a signal-to-noise ratio value lower than the signal-to-noise ratio threshold, and then calculates the scores of the areas with a signal-to-noise ratio value higher than or equal to the signal-to-noise ratio threshold through a second Softmax function to obtain second attention data, which is passed to the second addition unit;

[0135] Step A5: the second addition unit weights the first attention data with a first attention weight w1, weights the second attention data with a second attention weight w2, and adds the weighted first attention scores and second attention scores, and then multiplies the result through a second multiplication unit with a value matrix V to obtain self-attention scores Attn, which is expressed as follows:

[0136] Attn=w1*D attn +w2*S attn

[0137] D attn =Softmax(QK T / α)V

[0138] S attn =Softmax(QK T / α+(1-S′)σ)V

[0139] wherein α is a learnable scaling parameter for controlling the dot product size of the query matrix Q and the key matrix K; σ is a small negative scalar -1e9, D attn is the dense self-attention score, S attn is the sparse self-attention score; w1 and w2 are two normalized weights for adaptively combining the two branches; * is a multiplication operation.

[0140] The noise adaptive self-attention module NASA divides the channel into multiple heads to learn separate attention maps in parallel. Therefore, it can be ensured that the long-distance attention comes from the image area with a relatively high signal-to-noise ratio.

[0141] Step A6: the second multiplication unit passes the self-attention scores Attn to a fourth 1x1 convolution layer for convolution operation, and then passes the convolution result to a first addition unit for element-wise addition with the tensor data to obtain the self-attention feature of the current SNA block, which is passed to the dual-domain refinement feedforward network DRFN.

[0142] For a given input feature map and a signal-to-noise ratio map First, the size of the signal-to-noise ratio map S is adjusted pixel by pixel to match the input feature map I, and then it is normalized. In the self-attention score calculation, each pixel point is taken as a mask, which can effectively suppress the influence of the image area with very low signal-to-noise ratio in the enhancement process. The value of each pixel point is:

[0143]

[0144] where T represents the threshold of the signal-to-noise ratio map, S * represents the signal-to-noise ratio map after pixel-by-pixel adjustment, and S' represents the normalized signal-to-noise ratio map.

[0145] First, a layer normalization tensor is generated, and then a 1x1 convolution and a 3x3 deep convolution are cascaded to encode the local spatial context across channels and generate a query matrix Q, a key matrix K and a value matrix V. Then, Q and K are reshaped so that the size of the transposed attention map obtained by the dot product interaction between them is R C×C , instead of the size R HW×HW of the regular huge attention map. Since not all query matrices Q are closely related to the key matrix K, it is not the best choice to calculate the attention score for all tokens in the enhancement process. Sparse self-attention score calculation guided by signal-to-noise ratio map may be a reasonable solution, which filters out irrelevant information in areas with very low signal-to-noise ratio and propagates the most useful information flow.

[0146] As shown in Figure 3 : the dual-domain refinement feedforward network DRFN further refines the self-attention feature by the following steps:

[0147] Step B1: the second normalization layer in the dual-domain refinement feedforward network DRFN performs a normalization operation on the self-attention feature, and then a fifth 1x1 convolution layer is used to perform convolution operation on the normalized feature to obtain fifth convolution data, which is passed to a sixth 1x1 convolution layer, a fourth 3x3 deep convolution layer and a fast Fourier transform unit FFT;

[0148] The expression of the fifth convolution data is as follows:

[0149] Y' = W d (LN(Y))

[0150] where Y is the self-attention feature, LN is the layer normalization, Y' is the fifth convolution data, W d represents a 1x1 point-by-point convolution;

[0151] Step B2: the sixth 1x1 convolutional layer convolves the fifth convolutional data, and then a fifth 3x3 depth convolutional layer performs a depth convolution operation on the fifth convolutional data to obtain fifth depth convolutional data, which is transmitted to the first element-wise multiplication unit;

[0152] The expression of the fifth depth convolutional data is as follows:

[0153] Y'' = W p W d (Y')

[0154] wherein W p represents a 3x3 depth convolution, and Y'' represents the fifth depth convolutional data;

[0155] The fourth 3x3 depth convolutional layer performs a depth convolution operation on the fifth convolutional data, and then a GELU activation function is performed to obtain first activation data, which is transmitted to the element-wise multiplication unit;

[0156] The expression of the first activation data is as follows:

[0157]

[0158] wherein, is a GELU activation function, and Y''' represents the first activation data;

[0159] Step B3: the fast Fourier transform unit FFT performs a fast Fourier transform on the fifth convolutional data to obtain FFT data, which is then element-wise multiplied with the weight w of the dual-domain refinement feedforward network DRFN through the second element-wise multiplication unit, and then an inverse fast Fourier transform unit IFFT is performed to obtain inverse fast Fourier transform data, which is then activated by a GEGLU activation function to obtain second activation data, which is transmitted to the connection unit;

[0160] The expression of the second activation data is as follows:

[0161]

[0162] wherein, is a fast Fourier transform, is an inverse fast Fourier transform, w is a weight of the dual-domain refinement feedforward network DRFN, and Y f is the second activation data, and ζ() represents a GEGLU function;

[0163] The first element-wise multiplication unit element-wise multiplies the fifth depth convolutional data and the first activation data to obtain first multiplication data, which is transmitted to the connection unit;

[0164] The expression of the first multiplication data is as follows:

[0165] Y s =Y″⊙Y″′

[0166] wherein, ⊙ represents element-wise multiplication, Y s is the first multiplication data;

[0167] Step B4: the connection unit performs a connection operation on the second activation data and the first multiplication data, then performs a convolution operation through a seventh 1x1 convolution layer to obtain seventh convolution data, and passes the seventh convolution data to a third addition unit;

[0168] Step B5: the third addition unit performs element-wise addition on the self-attention feature and the seventh convolution data to obtain long-distance feature data b i of the current SNA block;

[0169] The expression of the long-distance feature data b i of the current SNA block is as follows:

[0170]

[0171] wherein, represents a connection operation.

[0172] Common fast Fourier transform processes information at each pixel position separately, and plays a very important role in improving feature representation. Therefore, in the image restoration task, it is essential to design an effective feedforward network to generate features that are helpful for reconstructing high-quality images. In order to further refine the features generated by NASA, a dual-domain refinement feedforward network DRFN is designed to enhance the flow of effective information in the features. Specifically, the present application constructs DRFN by performing two parallel operations in the spatial and frequency domains. In the spatial branch, the information flow is adjusted by a gating mechanism, which suppresses features containing redundant information and only allows useful information to propagate further. In the frequency branch, since not all low-frequency and high-frequency information is beneficial to image restoration, a learnable quantization matrix is used as a global filter in the frequency modulation process to adaptively determine which frequency information should be retained.

[0173] As Figure 4 shown: the first feature fusion block, the second feature fusion block, the third feature fusion block and the fourth feature fusion block have consistent structures and the same working logic, and are each provided with a third multiplication unit, a fourth multiplication unit, a fourth addition unit, a global average pooling layer, a first full connection layer, a second full connection layer, a fifth multiplication unit, a sixth multiplication unit, and a fifth addition unit, and a Sigmoid function is built in an output end of the second full connection layer;

[0174] In the step 6, the feature fusion block obtains the corresponding short-distance feature data ai and long distance feature data b i and a feature fusion operation is performed to obtain fusion feature data c of the current feature fusion block i , comprising the following steps:

[0175] Step C1: a third multiplication unit in the current feature fusion block multiplies the normalized signal-to-noise ratio map as a mask with short distance feature data a i to obtain third multiplication data and passes it to a fourth addition unit and a fifth multiplication unit;

[0176] A fourth multiplication unit in the current feature fusion block multiplies the normalized signal-to-noise ratio map with long distance feature data b i to obtain fourth multiplication data and passes it to the fourth addition unit and a sixth multiplication unit;

[0177] Step C2: the fourth addition unit element-wise adds the third multiplication data and the fourth multiplication data, then performs a global average pooling operation through a global average pooling layer, and then sequentially performs a full connection operation through a first full connection layer and a second full connection layer, and then normalizes through a Sigmoid function to obtain short distance feature fusion weight w a and long distance feature fusion weight w d , passes short distance feature fusion weight w a to the fifth multiplication unit, and passes long distance feature fusion weight w d to the sixth multiplication unit;

[0178] Step C3: the fifth multiplication unit weights the third multiplication data through short distance feature fusion weight w a to obtain fifth multiplication data and passes it to a fifth addition unit;

[0179] The sixth multiplication unit weights the fourth multiplication data through long distance feature fusion weight w d to obtain sixth multiplication data and passes it to the fifth addition unit;

[0180] Step C4: the fifth addition unit element-wise adds the fifth multiplication data and the sixth multiplication data to obtain feature fusion data c i of the current feature fusion block, expressed as follows:

[0181] c i = SGFF(a i ×(1-S′)+b i ×S′)

[0182] Wherein, SGFF represents a feature fusion operation, S' represents a normalized signal-to-noise ratio map, and 1-S' represents the normalized signal-to-noise ratio map as a mask.

[0183] Next, the performance of the SNR-guided noise adaptive network SNA-Net proposed in the present application is evaluated through specific experiments.

[0184] Dataset: The effectiveness of the present embodiment is tested on two reference datasets, LOLv2-Real and LOLv2-Synthetic. Among them, the LOLv2-Real dataset contains 689 pairs of low-light / normal-light images for training and 100 pairs for testing. The LOLv2-Synthetic dataset contains 900 pairs of images for training and 100 pairs for testing.

[0185] In addition, the present embodiment also tests our method on four no-reference datasets, LIME, DICM, MEF, and NPE.

[0186] Implementation details: The framework of the present application is implemented by the open-source deep learning framework PyTorch for machine learning and deep learning, and is trained and tested on an NVIDIA GeForce RTX 3090 GPU. For SNA-Net, the number of first SNA blocks, second SNA blocks, third SNA blocks, and fourth SNA blocks is 4, 6, 6, and 8, respectively, and the number of self-attention heads in the same level of SNA blocks is set to {1, 2, 4, 8}. During training, the Adam optimizer is used to optimize the SNA-Net network model, the batch size is set to 8, the patch size is set to 128, and a total of 600k iterations are performed. For data augmentation, vertical and horizontal flips are randomly applied. On the reference dataset, the peak signal-to-noise ratio PSNR and the structural similarity SSIM are used as evaluation indicators, and on the no-reference dataset, the natural image quality evaluator NIQE and the blind / reference image spatial quality evaluator BRISQUE are used.

[0187] Qualitative results: The present application is compared with some latest technologies SOTA to show its superiority. As shown in Figure 6 , first, Restormer has color distortion and image blurring during the enhancement process on the LOLv2-Real dataset. Second, SNR-Net shows severe artificial artifacts on the no-reference dataset. Third, as shown in the results of LOLv2-Synthetic in Figure 5 , FourLLIE loses details and blurs images when enhancing images, and the enhancement results on the no-reference dataset also show unnatural colors, such as Figure 6The results of LLformer also show its noise problem. Unfortunately, previous methods have some problems in color distortion, over / under exposure or image blur. In contrast, the SNA-Net network can stably enhance the low visibility while keeping as much detail information as possible, without causing over / under exposure, additional noise or artificial artifacts.

[0188] Quantitative results:

[0189] SNR-guided noise adaptive network

[0190]

[0191] Table 1 Quantitative results comparison on LOLv2-Real and LOLv2-Synthetic datasets

[0192] SNR-guided noise adaptive network

[0193]

[0194] Table 2 Quantitative results on no-reference dataset

[0195] In this section, the quantitative results of all models are presented in Table 1 and Table 2. The performance of the SNA-Net network is significantly improved on most datasets. Specifically, on the LOLv2-Real dataset, for PSNR, SNA-Net improves 0.91 dB and 0.05 dB compared with SNR-Net and FourLLIE respectively, indicating better improvement in noise suppression. For structural similarity SSIM, the present application also achieves significant improvement on two datasets, indicating that the images enhanced by SNA-Net network have better perceptual quality. In summary, there is a certain degree of improvement in the four evaluation indicators compared with other methods, which proves that the present application achieves satisfactory recovery effect in structural details and noise. The quantitative indicators of no-reference, unpaired datasets also show that the SNA-Net network has certain advantages.

[0196] Low-light target detection:

[0197] Low-light image enhancement as a preprocessing step for advanced vision tasks plays a crucial role in improving their performance. To this end, the present invention conducts low-light target detection experiments using the Exdark dataset. And compares the effects of using different low-light enhancement algorithms as preprocessing steps on low-light detection performance. The Exdark dataset contains 7363 images, covering 10 different low-light environments. In this experiment, YOLOv5s is used as the target detection model, and it is combined with various enhancers as preprocessing modules with fixed parameters. By training the model, the effects of these preprocessing steps on detection performance are evaluated. As shown in Figure 3 the model of the present invention achieves the best performance, which improves by 3.76% in mAR 0.5 and 4.64% in mAR 0.5:0.95 compared to LLformer and SNR-Net, respectively. These improvements may be due to the adaptive noise filtering of NASA and the refinement of DRFN, which enable the enhancer to recover enhanced images with reduced noise interference and more rich details, which are crucial for the detector.

[0198] Ablation experiments:

[0199] To verify the effectiveness of each component in the SNA-Net network, three major ablation studies are considered by removing three different components from the SNA-Net network framework on the LOLv2-Real dataset. In addition, some ablation studies on two components of the SNA block are also considered.

[0200] • No long branch: The Transformer encoder branch is removed, so the framework only has a CNN-based encoder.

[0201] • No short branch: The CNN encoder branch is removed, so the framework only has a Transformer-based encoder.

[0202] • No SGFF: The signal-to-noise ratio guided feature fusion module is removed, and only a simple concatenation is used.

[0203] Table 3 compares the performance of low-light target detection on the Exdark dataset preprocessed by different enhancers.

[0204]

[0205] Table 3 compares the performance of low-light target detection on the Exdark dataset preprocessed by different enhancers.

[0206] As shown in Table 4, the complete setting achieves the highest PSNR and superior SSIM value. "w / o S" and "w / o L" prove the effectiveness of the two parallel encoders. "w / o SGFF" proves that the SGFF module can integrate two features from the SNA block and the CNN encoding block more finely.

[0207]

[0208] Table 4 Ablation experiment results of SNA-Net

[0209] Effectiveness of NASA: To study the effectiveness of the NASA component, several existing self-attention mechanisms are replaced for comparison. As shown in Table 5, compared with MDTA in Restormer, the PSNR and SSIM are improved by 0.71dB and 0.01, respectively. Comparison with TKSA and ASSA is also made, and certain performance improvement is obtained. In addition, since the sparse method does not introduce additional parameters, it does not lead to an increase in complexity while improving performance.

[0210]

[0211] Table 5 Ablation experiment for different self-attention mechanisms

[0212] Effectiveness of DRFN: Since not all features contribute positively to restoring a clear image, simply applying feature conversion can result in too much redundant information. DREN estimates information useful for image restoration in the frequency and spatial domains. To demonstrate the effectiveness in the dual domain, an ablation study is conducted by removing the frequency domain and the spatial domain, respectively. Specifically, the experimental settings include (1) without DRFN, (2) without the frequency branch, and (3) without the spatial branch. The quantitative comparison results are shown in Table 6. The results show that DRFN can select more effective information in the two domains, reduce several redundant features, and thus better complement the NASA design.

[0213]

[0214] Table 6 Ablation experiment of dual-domain refinement feedforward network

[0215] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A low-light image enhancement method of a noise adaptive network based on signal-to-noise ratio guidance, characterized in that, It comprises the following steps: Step 1: Construct a low-light image enhancement system based on a signal-to-noise ratio guided noise adaptive network, which is provided with an image acquisition module and a signal-to-noise ratio guided noise adaptive network SNA-Net, wherein the input layer of the signal-to-noise ratio guided noise adaptive network SNA-Net is connected with the output end of a CNN encoder, the input end of a SNA encoder and the input end of a signal-to-noise ratio map calculation module, the output ends of the CNN encoder, the SNA encoder and the signal-to-noise ratio map calculation module are connected with the input end of a feature fusion module, the output end of the CNN encoder is also connected with the input end of a CNN decoder through a convolution residual block, and the input end of the CNN decoder is also connected with the output end of the feature fusion module; Step 2: The image acquisition module acquires a low-light image I and transmits it to the signal-to-noise ratio guided noise adaptive network SNA-Net; Step 3: The input layer of the signal-to-noise ratio guided noise adaptive network SNA-Net acquires the low-light image I and transmits it to the CNN encoder, the SNA encoder and the signal-to-noise ratio map calculation module; Step 4: The signal-to-noise ratio map calculation module calculates the low-light image I to obtain a corresponding signal-to-noise ratio map S, and then transmits the signal-to-noise ratio map S to the feature fusion module and the SNA encoder; Step 5: The CNN encoder encodes the low-light image I at a short distance to capture local information of the low-light image I, to obtain short-distance feature data a i and is passed to the feature fusion module and the convolution residual block; The SNA encoder adopts the signal-to-noise ratio map S as prior knowledge to calculate a self-attention score, and then performs long-distance encoding on the low-illumination image I to filter out noise and redundant information, thereby obtaining long-distance feature data b i , and is transmitted to a feature fusion module; Step 6: the convolution residual block performs a convolution residual operation on the short distance feature data a i a convolution residual operation to obtain convolution residual data d, and passes to the CNN decoder; The feature fusion module performs feature fusion operation on the short-distance feature data a i and the long-distance feature data b i to obtain fusion feature data c i and deliver to the CNN decoder. Step 7: The CNN decoder decodes the convolutional residual data d and the fusion feature data c i to obtain an enhanced image 2. The low-light image enhancement method of claim 1, wherein: In the step 1, the CNN encoder is provided with a first CNN encoding block, a second CNN encoding block, a third CNN encoding block and a fourth CNN encoding block connected in sequence, the SNA encoder is provided with a first SNA block, a second SNA block, a third SNA block and a fourth SNA block connected in sequence, the feature fusion module is provided with a first feature fusion block, a second feature fusion block, a third feature fusion block and a fourth feature fusion block, and the CNN decoder is provided with a first CNN decoding block, a second CNN decoding block, a third CNN decoding block and a fourth CNN decoding block connected in sequence; The output ends of the first CNN encoding block and the first SNA block are connected with the input end of the first feature fusion block, and the output end of the first feature fusion block is connected with the input end of the fourth CNN decoding block; The output ends of the second CNN encoding block and the second SNA block are connected with the input end of the second feature fusion block, and the output end of the second feature fusion block is connected with the input end of the third CNN decoding block; The output ends of the third CNN encoding block and the third SNA block are connected with the input end of the third feature fusion block, and the output end of the third feature fusion block is connected with the input end of the second CNN decoding block; The output ends of the fourth CNN encoding block and the fourth SNA block are connected with the input end of the fourth feature fusion block, and the output end of the fourth feature fusion block is connected with the input end of the first CNN decoding block.

3. The low-light image enhancement method of claim 1, wherein: In said step 4, first a gray scale image I corresponding to the low light image I is computed g Then the signal to noise ratio map S, S e R, is computed according to the following formula H×W ; where Denoise() denotes a denoising operation using a mean filter, abs denotes an absolute value, and N e R H×W estimated noise map.

4. The low-light image enhancement method of claim 2, wherein: The first CNN encoding block, the second CNN encoding block, the third CNN encoding block and the fourth CNN encoding block are consistent in structure and are each provided with a 3x3 convolution layer and a down-sampling layer, and the output end of the 3x3 convolution layer is built-in with a LeakyReLU activation function.

5. The low-light image enhancement method of claim 2, wherein: The first SNA block, the second SNA block, the third SNA block and the fourth SNA block are of the same structure and are each provided with a noise adaptive self-attention module (NASA) and a double-domain refined feedforward network (DRFN); The noise adaptive self-attention module NASA in each SNA block calculates a self-attention score Attn of the corresponding SNA block by taking the signal-to-noise ratio map S as prior knowledge, and then encodes the low-illumination image I or the output data of the previous SNA block in a long distance through the self-attention score Attn to obtain self-attention features of the current SNA block, and then further refines the self-attention features through a dual-domain refinement feedforward network DRFN to obtain long-distance feature data b of the current SNA block i .

6. The low-light image enhancement method of claim 5, wherein: The noise adaptive self-attention module (NASA) obtains the self-attention feature of the current SNA block through the following steps: Step A1: The first normalization layer in the noise adaptive self-attention module (NASA) obtains the output data of the low-illumination image I or the previous SNA block, generates tensor data, and then transmits the tensor data to a query matrix encoding unit, a key matrix K encoding unit, a value matrix V encoding unit and a first addition unit; Step A2: The query matrix encoding unit encodes the tensor data through a first 1x1 convolution layer and a first 3x3 deep convolution layer to generate a query matrix Q, and transmits the query matrix Q to a first multiplication unit; The key matrix K encoding unit encodes the tensor data through a second 1x1 convolution layer and a second 3x3 deep convolution layer to generate a key matrix K, and transmits the key matrix K to the first multiplication unit after matrix transposition; The value matrix V encoding unit encodes the tensor data through a third 1x1 convolution layer and a third 3x3 deep convolution layer to generate a value matrix V, and transmits the value matrix V to a second multiplication unit; The expressions for generating the query matrix Q, the key matrix K and the value matrix V are as follows: where W d represents a 1x1 point-wise convolution, W p represents a 3x3 depth-wise convolution, and X represents tensor data; Step A3: the first multiplication unit multiplies the query matrix Q and the transposed key matrix K T to obtain a first multiplication matrix, and passes it to the sparse self-attention branch and the dense self-attention branch; Step A4: The dense self-attention branch calculates the first attention data according to the first multiplication matrix through a first Softmax function, and transmits the first attention data to a second addition unit; The sparse self-attention branch uses the normalized signal-to-noise ratio image S' as a mask to filter out the areas in the first multiplication matrix where the signal-to-noise ratio is lower than the signal-to-noise ratio threshold, and then calculates the scores of the areas where the signal-to-noise ratio is higher than or equal to the signal-to-noise ratio threshold through a second Softmax function to obtain the second attention data, and transmits the second attention data to the second addition unit; Step A5: The second addition unit weights the first attention data with a first attention weight w1, weights the second attention data with a second attention weight w2, and adds the weighted first attention score and the weighted second attention score, and then multiplies the sum by the value matrix V through a second multiplication unit to obtain the self-attention score Attn, and the expression is as follows: Attn = w1 * D attn + w2 * S attn D attn = Softmax(QK T / α)V S attn = Softmax(QK T / α+(1-S')σ)V where a is a scaling parameter to control the dot product size of query matrix Q and key matrix K; s is a negative scalar -1e9, D attn is dense self-attention score, S attn is sparse self-attention score; Step A6: The second multiplication unit transmits the self-attention score Attn to a fourth 1x1 convolution layer for convolution operation, and then adds the convolution result and the tensor data element by element through a first addition unit to obtain the self-attention feature of the current SNA block, and transmits the self-attention feature to the double-domain refined feedforward network (DRFN).

7. The low-light image enhancement method of claim 6, wherein: The double-domain refined feedforward network (DRFN) further refines the self-attention feature through the following steps: Step B1: The second normalization layer in the double-domain refined feedforward network (DRFN) performs a normalization operation on the self-attention feature, and then performs a convolution operation on the normalized feature through a fifth 1x1 convolution layer to obtain fifth convolution data, and transmits the fifth convolution data to a sixth 1x1 convolution layer, a fourth 3x3 deep convolution layer and a fast Fourier transform unit (FFT); The expression for obtaining the fifth convolution data is as follows: Y' = W d (LN(Y)) where Y is a self-attention feature, LN is a layer normalization, Y' is a fifth convolution data, W d represents a 1x1 point-wise convolution; Step B2: the sixth 1x1 convolutional layer convolves the fifth convolutional data, then the fifth 3x3 depth convolutional layer performs depth convolution operation to obtain fifth depth convolutional data, and the first element-wise multiplication unit is passed; The expression of the fifth depth convolutional data is as follows: Y" = W p W d (Y') wherein W p represents a 3x3 depth convolution, Y" represents fifth depth convolution data; The fourth 3x3 depth convolutional layer performs depth convolution operation on the fifth convolutional data, then the GELU activation function is activated to obtain the first activation data, and the element-wise multiplication unit is passed; The expression of the first activation data is as follows: wherein, is a GELU activation function, Y”’ represents first activation data; Step B3: the fast Fourier transform unit FFT performs fast Fourier transform on the fifth convolutional data to obtain FFT data, then the FFT data is multiplied by the weight w of the dual-domain refinement feedforward network DRFN through the second element-wise multiplication unit, then the inverse fast Fourier transform unit IFFT is performed, then the GEGLU activation function is activated to obtain the second activation data, and the connection unit is passed; The expression of the second activation data is as follows: wherein, is a fast Fourier transform, is an inverse fast Fourier transform, w is a weight of a dual-domain refinement feed-forward network DRFN, Y f is second activation data, ζ() denotes a GEGLU function; The first element-wise multiplication unit element-wise multiplies the fifth depth convolutional data and the first activation data to obtain the first multiplication data, and the connection unit is passed; The expression of the first multiplication data is as follows: Y s = Y" O Y'" wherein, ⊙ denotes element-wise multiplication, Y s is the first multiplication data; Step B4: the connection unit connects the second activation data and the first multiplication data, then the seventh 1x1 convolutional layer is convolved to obtain the seventh convolutional data, and the third addition unit is passed; Step B5: the third adding unit element-wise adds the self-attention feature and the seventh convolution data to obtain long-distance feature data b of the current SNA block i ; obtaining long-range feature data b of the current SNA block i The expression is as follows: wherein represents a concatenation operation.

8. The low-light image enhancement method of claim 2, wherein: The first feature fusion block, the second feature fusion block, the third feature fusion block and the fourth feature fusion block have the same structure, and are provided with a third multiplication unit, a fourth multiplication unit, a fourth addition unit, a global average pooling layer, a first full connection layer, a second full connection layer, a fifth multiplication unit, a sixth multiplication unit, and a fifth addition unit. The output end of the second full connection layer is built-in Sigmoid function; In the step 6, the feature fusion block obtains the corresponding short-range feature data a i and long-range feature data b i and performs a feature fusion operation to obtain the fusion feature data c i of the current feature fusion block, including the following steps: Step C1: the third multiplication unit in the current feature fusion block multiplies the normalized signal-to-noise ratio map as a mask with the short-distance feature data a i to obtain third multiplication data, and passes the third multiplication data to the fourth addition unit and the fifth multiplication unit; The fourth multiplication unit in the current feature fusion block multiplies the normalized signal-to-noise ratio map with the long-distance feature data b i The fourth multiplication unit in the current feature fusion block multiplies the normalized signal-to-noise ratio map with the long-distance feature data b Step C2: the fourth adding unit element-wise adds the third multiplication data and the fourth multiplication data, then performs a global average pooling operation through a global average pooling layer, and then sequentially performs a full connection operation through a first full connection layer and a second full connection layer, and then performs normalization through a Sigmoid function to obtain a short-distance feature fusion weight w a and a long-distance feature fusion weight w d , and the short-distance feature fusion weight w a is transmitted to a fifth multiplication unit, and the long-distance feature fusion weight w d is transmitted to a sixth multiplication unit; Step C3: the fifth multiplication unit weights the fifth multiplication data by short-distance feature fusion weight w a The third multiplication data is weighted to obtain fifth multiplication data, and is transmitted to a fifth addition unit; The sixth multiplication unit weights the long-distance feature fusion weight w d The fourth multiplication data is weighted to obtain sixth multiplication data and is transmitted to a fifth addition unit; Step C4: the fifth adder unit element-wise adds the fifth multiplication data and the sixth multiplication data to obtain feature fusion data c of the current feature fusion block i The expression is as follows: c i = SGFF(a i ×(1-S′)+b i ×S′) Wherein, SGFF represents feature fusion operation, S' represents normalized signal-to-noise ratio map, and 1-S' represents normalized signal-to-noise ratio map as mask.

Images