A low-light image enhancement method and system based on space-frequency fusion

Abstract

The present invention discloses a space-frequency fused low-light image enhancement method and system, which relate to the field of image enhancement technology. The acquired original low-light image is preprocessed and then input into a trained image enhancement model to output a high-resolution image corresponding to the original low-light image. The training process of the image enhancement model is as follows: the original low-light image is acquired and preprocessed to construct a training data set, and the low-light image in the training data set is sequentially input into a shared weight encoder component, a denoising decoding component and a color correction decoding component to output fusion features; after the fusion features are convolved, an RGB image is obtained using a pixel shuffle operation; the loss of the obtained RGB image and a clean image corresponding to the original low-light image is calculated to update the model parameters of the image enhancement model; the low-light image enhancement method and system are helpful to extract the texture details and denoising capability of the image.

Description

Technical Field

[0001] The present invention relates to the field of image enhancement technology, and in particular to a space-frequency fusion low-light image enhancement method and system. Background Art

[0002] Since images under low light conditions have a lower signal-to-noise ratio, the task of low-light Raw domain denoising has a place in the low-level tasks of computer vision. Since the conversion from Raw domain to sRGB domain involves complex nonlinear ISP (image signal processing) transformations, such as white balance, color correction, gamma correction, etc., it is difficult to achieve simply through data transformation. With the development of deep learning, the use of neural networks to solve this problem has shown good results.

[0003] Chen et al. first proposed a set of methods for Raw domain denoising based on deep learning in the paper Learning to See in the Dark and released the SID dataset. Since then, researchers have made great achievements in this field, and a variety of methods have emerged, such as using the classic UNet network in the field of image segmentation, or introducing some efficient convolution modules to extract features, and some using the decoupling idea to complete the task in stages, with Raw domain denoising in the first stage and color correction and conversion to sRGB space in the second stage. These methods have shown good results on public datasets such as SID and MCR.

[0004] Most of the existing deep learning methods that complete this task are almost all full convolutional networks, which leads to two problems. First, the receptive field of the convolution kernel is limited by the size of the convolution kernel. Commonly used small convolution kernels such as 3×3 are not suitable for large-sized Raw images. Second, the convolution operation is often accompanied by a large number of parameters, which is computationally intensive, resulting in high costs and poor image enhancement effects. Summary of the invention

[0005] Based on the technical problems existing in the background technology, the present invention proposes a low-light image enhancement method and system with space-frequency fusion, which is helpful to extract the texture details and denoising ability of the image.

[0006] The present invention proposes a low-light image enhancement method using space-frequency fusion, which pre-processes the acquired original low-light image and then inputs it into a trained image enhancement model to output a high-resolution image corresponding to the original low-light image.

[0007] The training process of the image enhancement model is as follows:

[0008] Step 1: Obtain the original low-light image and preprocess it to construct a training dataset;

[0009] Step 2: Input the low-light images in the training data set into the shared weight encoder component, the denoising decoding component and the color correction decoding component in sequence. In the shared weight encoder component, the layers are downsampled, and in the denoising decoding component and the color correction decoding component, the layers are upsampled.

[0010] Step 3: Input the low-light image to the shared weight encoder component and the denoising decoding component in sequence. After encoding and convolution denoising, each layer in the denoising decoding component outputs a first transient coding feature map.

[0011] Step 4: Output the low-light image to the shared weight encoder component again, fuse the first transient coding feature map in each layer and encode it layer by layer, and the shared weight encoder component outputs the final coding feature map;

[0012] Step 5: Input the final state encoding feature map into the color correction decoding component. After layer-by-layer correction, the color correction decoding component outputs the fusion feature.

[0013] Step 6. After convolution operation on the fused features, use pixel shuffling operation to obtain RGB image; calculate the loss of the obtained RGB image and the clean image corresponding to the original low-light image to update the model parameters of the image enhancement model.

[0014] Further, in the shared weight encoder component, the first layer of shared weight encoders is connected to the Nth layer of shared weight encoders in sequence, and in the denoising decoder component, the Nth layer of denoising decoders is connected to the first layer of denoising decoders in sequence;

[0015] The generation process of the first transient coding feature map is as follows:

[0016] Each layer of shared weight encoder includes a spatial domain branch and a frequency domain branch. The spatial domain branch adopts a state space model. In the current layer shared weight encoder, the input two-dimensional feature map is pulled into a one-dimensional vector through Z-shaped scanning. The one-dimensional vector is transformed into a spatial domain feature through a potential state using a linear ordinary differential equation framework. The two-dimensional feature map of the current layer input outputs a frequency domain feature through a frequency domain branch. The spatial domain feature is denoised through convolution and fused with the frequency domain feature to obtain a temporary fused feature map output by the current layer shared weight encoder.

[0017] The temporary fused feature map output by the Nth layer shared weight encoder is input to the Nth layer denoising decoder, and the denoised feature map generated by upsampling by the next layer denoising decoder of the current layer is fused with the temporary fused feature map output by the current same layer shared weight encoder as the input of the current layer denoising decoder. After convolution denoising, the current layer denoising decoder outputs the first transient coding feature map.

[0018] Furthermore, the input two-dimensional feature map is pulled into a one-dimensional vector through Z-shaped scanning, specifically:

[0019] Scan the two-dimensional feature map horizontally and vertically by pixel to obtain feature vectors in two directions;

[0020] Reverse the obtained eigenvectors in two directions to obtain eigenvectors in the other two directions;

[0021] Flatten the obtained eigenvectors in four directions to obtain a one-dimensional vector.

[0022] Furthermore, in step 4, the final state encoding feature map is generated as follows:

[0023] The second transient coding feature map downsampled by the shared weight encoder of the previous layer of the current layer and the first transient coding feature map output by the current same-layer denoising decoder are fused as the input of the shared weight encoder of the current layer. After encoding, the shared weight encoder of the current layer outputs the third transient coding feature map. After layer-by-layer encoding, the shared weight encoder of the Nth layer outputs the final state coding feature map.

[0024] Furthermore, in the color correction decoding component, the Nth layer color correction decoder is connected to the first layer color correction decoder in sequence. In step 5, the fusion feature is generated as follows:

[0025] The features generated by upsampling of the next layer of color correction decoder of the current layer are fused with the third transient coding feature map output by the current same layer shared weight encoder as the input of the current layer color correction decoder. After layer-by-layer correction, the first layer color correction decoder outputs the fused features.

[0026] Furthermore, in step six, the loss of the obtained RGB image and the clean image corresponding to the original low-light image is calculated, specifically:

[0027] Perform fast Fourier transform on the RGB image to obtain the amplitude spectrum and phase spectrum corresponding to the RGB image;

[0028] The amplitude spectrum and phase spectrum corresponding to the RGB image and the amplitude spectrum and phase spectrum corresponding to the clean image are calculated to calculate the loss function to update the model parameters of the image enhancement model.

[0029] Furthermore, the loss function The details are as follows:

[0030] ;

[0031] ;

[0032] ;

[0033] in, represents the spatial domain loss, represents the frequency domain loss, is the equalization coefficient of the spatial domain loss term, is the equalization coefficient of the frequency domain loss term, represents a norm, Represents the Raw image output by the image enhancement model, Represents a Raw image without noise, represents the sRGB image output by the image enhancement model, represents a noise-free sRGB image, , They respectively represent the amplitude spectrum A and phase spectrum P after fast Fourier transform of the input.

[0034] Furthermore, in step one, the original low-light image is preprocessed as follows: data transformation and data enhancement are performed in sequence after the black level is subtracted from the original low-light image, and a training data set is constructed based on the preprocessed image.

[0035] A low-light image enhancement system with space-frequency fusion, which pre-processes the acquired original low-light image and inputs it into a trained image enhancement model to output a high-resolution image corresponding to the original low-light image;

[0036] The training process of the image enhancement model is as follows:

[0037] Step 1: Obtain the original low-light image and preprocess it to construct a training dataset;

[0038] Step 2: Input the low-light images in the training data set into the shared weight encoder component, the denoising decoding component and the color correction decoding component in sequence. In the shared weight encoder component, the layers are downsampled, and in the denoising decoding component and the color correction decoding component, the layers are upsampled.

[0039] Step 3: Input the low-light image to the shared weight encoder component and the denoising decoding component in sequence. After encoding and convolution denoising, each layer in the denoising decoding component outputs a first transient coding feature map.

[0040] Step 4: Output the low-light image to the shared weight encoder component again, fuse the first transient coding feature map in each layer and encode it layer by layer, and the shared weight encoder component outputs the final coding feature map;

[0041] Step 5: Input the final state encoding feature map into the color correction decoding component. After layer-by-layer correction, the color correction decoding component outputs the fusion feature.

[0042] Step 6. After convolution operation on the fused features, use pixel shuffling operation to obtain RGB image; calculate the loss of the obtained RGB image and the clean image corresponding to the original low-light image to update the model parameters of the image enhancement model.

[0043] The advantages of the space-frequency fusion low-light image enhancement method and system provided by the present invention are: an image enhancement model WNet is constructed to decouple the tasks, first remove noise, and then perform color correction. CNN (convolutional neural network) is used to extract local features of the image, and SSM (state space model) is introduced to increase the receptive field of the network to extract global features. At the same time, its scanning method is improved to Z-shaped scanning, emphasizing the continuity of the image space, which helps the image surface enhancement model to learn more efficiently, and introduces a diagonal scanning mechanism to make full use of the information of adjacent pixels to supplement the shortcomings of the naive scanning method. In addition, this embodiment introduces a space-frequency fusion mechanism, so that the image enhancement model can not only learn information in the spatial domain, but also learn information in the frequency domain, which helps to extract the texture details and denoising ability of the image; at the same time, the amount of calculation is reduced compared to the existing full convolution network, the cost is low, and the image enhancement effect is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of the process of the present invention;

[0045] Figure 2 This is a diagram of the training and testing process of the image enhancement model;

[0046] Figure 3 Figure 3 is a schematic diagram of the processing results of the SID dataset, which is the most commonly used dataset in the field of low-light image enhancement. (a), (b), (c), and (d) are schematic diagrams of the results of directly visualizing different low-light images in the SID dataset; (e), (f), (g), and (h) are schematic diagrams of the output visualization results after (a), (b), (c), and (d) are sent to the trained image enhancement model after preprocessing. DETAILED DESCRIPTION

[0047] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.

[0048] like Figures 1 to 3As shown, the present invention proposes a space-frequency fusion low-light image enhancement method, which preprocesses the acquired original low-light image and inputs it into the trained image enhancement model to output a high-resolution image corresponding to the original low-light image.

[0049] The preprocessing of the original low-light image includes: performing data transformation after subtracting the black level from the original low-light image, so as to realize the preprocessing process of the original low-light image.

[0050] The training process of the image enhancement model Wnet is as follows:

[0051] Step 1: Obtain the original low-light image and preprocess it to construct a training dataset;

[0052] The original low-light image is preprocessed as follows: after subtracting the black level from the original low-light image, data transformation and data enhancement are performed in sequence, and a training data set is constructed based on the preprocessed image.

[0053] For example, when preprocessing a noisy low-light image, the noisy low-light image may be subtracted from the black level (the black level is set to 512 in this embodiment) and then multiplied by the exposure time ratio, wherein the exposure time ratio (the maximum exposure time ratio setting in this embodiment is 300) is the exposure time of the clean low-light image (clean Raw domain image) divided by the exposure time of the noisy low-light image (noisy Raw domain image), and the Rawpy library is used to obtain a clean image in the sRGB domain corresponding to the clean low-light image.

[0054] In addition, the preprocessed original low-light image is packed from a single channel to a four-channel image, and then cropped into a 4*512*512 image. Similarly, the corresponding position of the clean image in the sRGB domain is cropped into a 3*1024*1024 image, and a fast Fourier transform is performed on it to obtain the corresponding amplitude spectrum and phase spectrum; the cropped low-light image and clean image are used to construct a training data set.

[0055] Step 2: Input the low-light images in the training data set into the shared weight encoder component, the denoising decoding component and the color correction decoding component in sequence. In the shared weight encoder component, downsampling is performed between layers, and in the denoising decoding component and the color correction decoding component, upsampling is performed between layers.

[0056] That is, in the shared weight encoder component, the first layer shared weight encoder is connected to the Nth layer shared weight encoder in sequence, in the denoising decoding component, the Nth layer denoising decoder is connected to the first layer denoising decoder in sequence, and in the color correction decoding component, the Nth layer color correction decoder is connected to the first layer color correction decoder in sequence; the clean image corresponding to the low-light image is also synchronously input into the image enhancement model, and the clean image corresponds to the high-resolution image.

[0057] The shared weight encoder component adopts a space-frequency gating structure for target shared weight encoding. The shared weight encoder component contains spatial domain branches and frequency domain branches for extracting features from different domains. Since the frequency domain is only supervised in the sRGB domain, the frequency domain branch is only turned on in the color correction decoding component, i.e., the secondary encoding stage, and is turned off in the convolution denoising stage, i.e., the first encoding stage, so as to reduce the ambiguity of features between different domains. The convolution denoising stage consists of a series of convolutional layers, and a gated fusion mechanism is used for feature fusion with the shared weight encoder. The color correction decoding component also has spatial domain branches and frequency domain branches similar to the shared weight encoder component, but the gating mechanism is cancelled. The uniform distribution initialization method is used to initialize the network parameters. In addition, experimental hyperparameters are set, such as initial learning rate, learning rate update algorithm, batch size, optimizer type, optimizer parameters, number of iterations, etc.; in this embodiment, the initial learning rate is set to 0.0001, the learning rate update algorithm is the cosine annealing algorithm, and it is reduced to 0.00001 at the 500th epoch (epoch is a training process in which all samples in the training data set are passed once (and only once)), the batch size is set to 1, the optimizer type is set to Adamw, the optimizer parameter betas=[0.9, 0.999], and the number of iterations is set to 500, where betas is a tuple containing two values, beta1 and beta2, beta1=0.9 is used to calculate the first-order moment estimate (momentum term), and is usually set to a value close to 1, such as 0.9, which controls the influence of the past gradient on the current momentum; beta2 is used to calculate the second-order moment estimate, and is usually set to a value close to 1, such as 0.999, which controls the influence of the past gradient square on the current second-order moment.

[0058] Step 3: Input the low-light image to the shared weight encoder component and the denoising decoding component in sequence. After encoding and convolution denoising, each layer in the denoising decoding component outputs a first transient coding feature map.

[0059] That is, the low-light image is output to the shared weight encoder component, each layer of shared weight encoder outputs a temporary fused feature map, the temporary fused feature map output by the Nth layer of shared weight encoder is input to the Nth layer of denoising decoder, the denoised feature map generated by upsampling the next layer of denoising decoder of the current layer is fused with the temporary fused feature map output by the current same layer of shared weight encoder as the input of the current layer of denoising decoder, after convolution denoising, the current layer of denoising decoder outputs the first transient coded feature map;

[0060] Each layer of shared weight encoder includes a spatial domain branch and a frequency domain branch. The spatial domain branch adopts a state space model. In the shared weight encoder of the current layer, the input two-dimensional feature map is pulled into a one-dimensional vector through Z-shaped scanning. The one-dimensional vector is transformed into spatial domain features through the potential state using the linear ordinary differential equation framework. The two-dimensional feature map of the current layer input outputs frequency domain features through the frequency domain branch. The spatial domain features are denoised by convolution and fused with the frequency domain features to obtain a temporary fused feature map.

[0061] The input two-dimensional feature map is stretched into a one-dimensional vector through Z-shaped scanning, specifically: after each row / column is scanned, it is immediately scanned from the next adjacent row / column, with horizontal scanning and vertical scanning in two directions, thereby obtaining feature vectors in two directions, and reversing the feature vectors in the two directions to obtain another two directions, a total of four one-dimensional vectors, and the flattened one-dimensional vectors are then processed by the state space model (SSM) to obtain the result; for example, firstly, by horizontal scanning and vertical scanning in two directions, two feature vectors of 123 and 456 are obtained, and then the two feature vectors are reversed respectively to obtain 321 and these two feature vectors, thereby obtaining four one-dimensional vectors: 123, 456, 321, 654.

[0062] Among them, the state space model (SSM) uses the linear ordinary differential equation framework to transform a one-dimensional vector into an output through a potential state. Specifically, for a one-dimensional vector , the output is The dynamic characteristics of the state space model (SSM) can be described by the following equation:

[0063] ; ; (1)

[0064] in, represents the state vector, is the one-dimensional vector input, They are the model parameters of the state-space model, such as Mamba. The model parameters are usually adjusted by the zero-order hold (ZOH) method. To discretize, the steps are as follows:

[0065] ; ; (2)

[0066] in, Represent the model parameters The discretized parameters, Represents model parameters Discretize the time step, Represents model parameters Discretize the time step, represents the identity matrix;

[0067] After discretization, Equation (1) has a step size The discretized version of can be rewritten as the following recurrent neural network RNN ​​form:

[0068] ; ; (3)

[0069] Indicates The state vector of step length is Indicates The state vector of step length is Indicates The one-dimensional vector input by the step size, For the Step output;

[0070] In addition, formula (3) can also be mathematically equivalently transformed into the following convolutional neural network CNN form:

[0071] ; (4)

[0072] ; (5)

[0073] in, is the length of the input one-dimensional vector, represents the convolution operation, is the structured convolution kernel, In is the size of the power, that is .

[0074] Step 4: Output the low-light image to the shared weight encoder component again, fuse the first transient coding feature map in each layer and encode it layer by layer, and the shared weight encoder component outputs the final coding feature map;

[0075] That is, the low-light image is output to the shared weight encoder component again, and the second transient coding feature map downsampled by the shared weight encoder of the previous layer of the current layer is fused with the first transient coding feature map output by the current same-layer denoising decoder as the input of the shared weight encoder of the current layer. After encoding, the shared weight encoder of the current layer outputs the third transient coding feature map. After layer-by-layer encoding, the shared weight encoder of the Nth layer outputs the final state coding feature map.

[0076] Step 5: Input the final state encoding feature map into the color correction decoding component. After layer-by-layer correction, the color correction decoding component outputs the fusion feature.

[0077] That is, the final state coding feature map is input into the Nth layer color correction decoder, the features generated by upsampling of the next layer color correction decoder of the current layer are fused with the third transient coding feature map output by the current same layer shared weight encoder as the input of the current layer color correction decoder, and after layer-by-layer correction, the first layer color correction decoder outputs the fused features.

[0078] It can be understood that steps two to five are used to reduce the amount of parameters, which can be briefly described as follows: in the shared weight encoder component, the output of the shared weight encoder of the current layer is used as the input of the shared weight encoder of the next layer, the shared weight encoder of the first layer inputs the low-light image, and the shared weight encoder of each layer outputs a temporary fused feature map; the output of the shared weight encoder of the Nth layer is connected to the denoising decoder of the Nth layer, and at the same time, the temporary fused feature map output by the shared weight encoder of each layer is passed to the denoising decoder of the same layer at the same layer as an auxiliary input of the denoising decoder, and then the low-light image is input into the shared weight encoder component again for encoding, and the features output by the shared weight encoder of the Nth layer are input into the color correction decoder of the Nth layer, and finally the color correction decoder of the first layer outputs the fused features.

[0079] That is, steps 2 to 5 correspond to two stages:

[0080] The first stage: The input feature image first passes through the shared weight encoder component, and is encoded by each layer of the shared weight encoder to generate a temporary fused feature map , Indicates the number of layers, the output of the Nth layer shared weight encoder Then go through the denoising decoding component, starting from the Nth layer denoising decoder, the Denoised feature map produced by upsampling of the layer denoising decoder With temporary fusion feature map Fusion as the first The input of the layer denoising decoder is denoised by convolution. Layer denoising decoder outputs the first transient coding feature map , the first transient coding feature map obtained after the first layer of denoising decoder This is the output of the first stage.

[0081] The second stage: The input of the second stage is the same as the first stage. The input feature image enters the shared weight encoder component, starting from the first layer of shared weight encoder. The second transient coding feature map generated by downsampling of the layer shared weight encoder With The first transient coding feature map output by the layer denoising decoder Fusion, as the first The input of the layer shared weight encoder is passed through the The layer shared weight encoder processes the third transient coding feature map , after the Nth layer shared weight encoder outputs the third transient coding feature map , the third transient coding feature map As the final state encoding feature map, it is input to the Nth layer color correction decoder. Layer color correction decoder upsampling features With The third transient coded feature map output by the layer-shared weight encoder Fusion is performed until the output of the second stage is obtained after passing through the first layer of color correction decoder, that is, the fused feature is obtained.

[0082] Step 6. After convolution operation on the fused features, use pixel shuffling operation to obtain an RGB image with shape (1, 3, 1024, 1024); calculate the loss of the obtained RGB image and the clean image corresponding to the original low-light image to update the model parameters of the image enhancement model.

[0083] The L1 loss of the RGB image and the clean image corresponding to the original low-light image is calculated, and the peak signal-to-noise ratio PSNR, structural similarity SSIM and other indicators are calculated by comparison. That is, the RGB image is subjected to a fast Fourier transform to obtain the amplitude spectrum and phase spectrum corresponding to the RGB image; the amplitude spectrum and phase spectrum corresponding to the RGB image and the amplitude spectrum and phase spectrum corresponding to the clean image are calculated to calculate the loss function to update the model parameters of the image enhancement model.

[0084] Loss Function The details are as follows:

[0085] ;

[0086] ;

[0087] ;

[0088] in, represents the spatial domain loss, represents the frequency domain loss, is the equalization coefficient of the spatial domain loss term, is the equalization coefficient of the frequency domain loss term, represents a norm, Represents the Raw image output by the image enhancement model, Represents a Raw image without noise, represents the sRGB image output by the image enhancement model, represents a noise-free sRGB image, , They respectively represent the amplitude spectrum A and phase spectrum P after fast Fourier transform of the input.

[0089] The best image enhancement model is selected as the subsequent execution model through the peak signal-to-noise ratio (PSNR) indicator of the image enhancement model on the validation set; in subsequent tests, the selected image enhancement model obtains the original low-light image, subtracts the black level from the original low-light image, and performs data transformation processing in sequence as the test data set, which is directly sent to the image enhancement model to obtain the corresponding output, and calculates the PSNR and SSIM between the output and the clean image to determine the generalization performance of the model. At the same time, the output is also visualized and saved as a picture for visual comparison.

[0090] Therefore, the low-light images in the training data set and the test data set are both obtained by preprocessing the original low-light images, but the difference is that the low-light images in the training data set are obtained by cropping the original low-light images after black level reduction, data transformation, and data enhancement, while the low-light images in the test data set are obtained after black level and data transformation.

[0091] In this embodiment, the SID data set is processed to obtain Figure 3 The results shown, Figure 3 (a), (b), (c), and (d) correspond to different images. (e) is a schematic diagram of the output visualization result of (a) after preprocessing and feeding it into the trained image enhancement model. Similarly, (f) corresponds to (b), (g) corresponds to (c), and (h) corresponds to (d). Figure 3 It can be clearly seen that this embodiment achieves good results in both the Raw domain (low-light domain) denoising task and the low-light enhancement, i.e., ISP (image signal processing) simulation task.

[0092] This embodiment constructs an image enhancement model WNet, decouples the tasks, removes noise first, and then performs color correction. CNN (convolutional neural network) is used to extract local features of the image, and SSM (state space model) is introduced to increase the receptive field of the network to extract global features. At the same time, its scanning method is improved to Z-shaped scanning, emphasizing the continuity of the image space, which helps the image surface enhancement model to learn more efficiently, and introduces a diagonal scanning mechanism to make full use of the information of adjacent pixels to supplement the shortcomings of the naive scanning method. In addition, this embodiment introduces a space-frequency fusion mechanism, so that the image enhancement model can not only learn information in the space domain, but also learn information in the frequency domain, which helps to extract the texture details and denoising ability of the image.

[0093] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A low-light image enhancement method based on space-frequency fusion, characterized in that: The acquired original low-light image is preprocessed and then input into the trained image enhancement model to output a high-resolution image corresponding to the original low-light image; The training process of the image enhancement model is as follows: Step 1: Obtain the original low-light image and preprocess it to construct a training dataset; Step 2: Input the low-light images in the training data set into the shared weight encoder component, the denoising decoding component and the color correction decoding component in sequence. In the shared weight encoder component, there are N layers with downsampling between layers. In the denoising decoding component and the color correction decoding component, there are N layers with upsampling between layers. Step 3, input the low-light image into the shared weight encoder component, each layer of the shared weight encoder includes a spatial domain branch and a frequency domain branch, the spatial domain branch adopts a state space model, in the current layer shared weight encoder, the input two-dimensional feature map is pulled into a one-dimensional vector through Z-shaped scanning, and the one-dimensional vector is transformed into a spatial domain feature through a potential state using a linear ordinary differential equation framework, and the two-dimensional feature map of the current layer input outputs a frequency domain feature through a frequency domain branch, and the spatial domain feature is denoised by convolution and fused with the frequency domain feature to obtain a temporary fused feature map output by the shared weight encoder of the current layer; the temporary fused feature map output by the Nth layer shared weight encoder is input to the Nth layer denoising decoder, and the denoised feature map generated by upsampling the next layer denoising decoder of the current layer is fused with the temporary fused feature map output by the current same layer shared weight encoder as the input of the current layer denoising decoder, and after convolution denoising, the current layer denoising decoder outputs the first transient coding feature map; Step 4: input the low-light image into the shared weight encoder component again, fuse the second transient coding feature map obtained by downsampling the shared weight encoder of the previous layer of the current layer with the first transient coding feature map output by the current same-layer denoising decoder as the input of the shared weight encoder of the current layer. After encoding, the shared weight encoder of the current layer outputs the third transient coding feature map. After layer-by-layer encoding, the shared weight encoder of the Nth layer outputs the final state coding feature map. Step 5: Input the final state encoding feature map into the color correction decoding component. After layer-by-layer correction, the color correction decoding component outputs the fusion feature. Step 6. After convolution operation on the fused features, use pixel shuffling operation to obtain RGB image; calculate the loss based on the obtained RGB image and the clean image corresponding to the original low-light image to update the model parameters of the image enhancement model.

2. The low-light image enhancement method based on space-frequency fusion according to claim 1, characterized in that: The input two-dimensional feature map is pulled into a one-dimensional vector through Z-shaped scanning, specifically: Scan the two-dimensional feature map horizontally and vertically by pixel to obtain feature vectors in two directions; Reverse the obtained eigenvectors in two directions to obtain eigenvectors in the other two directions; Flatten the obtained eigenvectors in four directions to obtain a one-dimensional vector.

3. The low-light image enhancement method of space-frequency fusion according to claim 1, characterized in that: In the color correction decoding component, the Nth layer color correction decoder is connected to the first layer color correction decoder in sequence. In step 5, the fusion feature is generated as follows: The final state coding feature map is input into the Nth layer color correction decoder. The features generated by upsampling of the next layer color correction decoder of the current layer are fused with the third transient coding feature map output by the current same layer shared weight encoder as the input of the current layer color correction decoder. After layer-by-layer correction, the first layer color correction decoder outputs the fused features.

4. The low-light image enhancement method of space-frequency fusion according to claim 1, characterized in that: In step six, the loss is calculated based on the obtained RGB image and the clean image corresponding to the original low-light image, specifically: Perform fast Fourier transform on the RGB image to obtain the amplitude spectrum and phase spectrum corresponding to the RGB image; The loss function is calculated based on the amplitude spectrum and phase spectrum corresponding to the RGB image and the amplitude spectrum and phase spectrum corresponding to the clean image to update the model parameters of the image enhancement model.

5. The low-light image enhancement method of space-frequency fusion according to claim 4, characterized in that: The loss function The details are as follows: ； ； ； in, represents the spatial domain loss, represents the frequency domain loss, is the equalization coefficient of the spatial domain loss term, is the equalization coefficient of the frequency domain loss term, represents a norm, Represents the Raw image output by the image enhancement model, Represents a Raw image without noise, represents the sRGB image output by the image enhancement model, represents a noise-free sRGB image, , They respectively represent the amplitude spectrum A and phase spectrum P after fast Fourier transform of the input.

6. The low-light image enhancement method of space-frequency fusion according to claim 1, characterized in that: In step one, the original low-light image is preprocessed as follows: data transformation and data enhancement are performed in sequence after the black level is subtracted from the original low-light image, and a training data set is constructed based on the preprocessed image.

7. A low-light image enhancement system with space-frequency fusion, characterized in that: The acquired original low-light image is preprocessed and then input into the trained image enhancement model to output a high-resolution image corresponding to the original low-light image; The training process of the image enhancement model is as follows: Step 1: Obtain the original low-light image and preprocess it to construct a training dataset; Step 2: Input the low-light images in the training data set into the shared weight encoder component, the denoising decoding component and the color correction decoding component in sequence. In the shared weight encoder component, there are N layers with downsampling between layers. In the denoising decoding component and the color correction decoding component, there are N layers with upsampling between layers. Step 3, input the low-light image into the shared weight encoder component, each layer of the shared weight encoder includes a spatial domain branch and a frequency domain branch, the spatial domain branch adopts a state space model, in the current layer shared weight encoder, the input two-dimensional feature map is pulled into a one-dimensional vector through Z-shaped scanning, and the one-dimensional vector is transformed into a spatial domain feature through a potential state using a linear ordinary differential equation framework, and the two-dimensional feature map of the current layer input outputs a frequency domain feature through a frequency domain branch, and the spatial domain feature is denoised by convolution and fused with the frequency domain feature to obtain a temporary fused feature map output by the shared weight encoder of the current layer; the temporary fused feature map output by the Nth layer shared weight encoder is input to the Nth layer denoising decoder, and the denoised feature map generated by upsampling the next layer denoising decoder of the current layer is fused with the temporary fused feature map output by the current same layer shared weight encoder as the input of the current layer denoising decoder, and after convolution denoising, the current layer denoising decoder outputs the first transient coding feature map; Step 4: input the low-light image into the shared weight encoder component again, fuse the second transient coding feature map obtained by downsampling the shared weight encoder of the previous layer of the current layer with the first transient coding feature map output by the current same-layer denoising decoder as the input of the shared weight encoder of the current layer. After encoding, the shared weight encoder of the current layer outputs the third transient coding feature map. After layer-by-layer encoding, the shared weight encoder of the Nth layer outputs the final state coding feature map. Step 5: Input the final state encoding feature map into the color correction decoding component. After layer-by-layer correction, the color correction decoding component outputs the fusion feature. Step 6. After convolution operation on the fused features, use pixel shuffling operation to obtain RGB image; calculate the loss based on the obtained RGB image and the clean image corresponding to the original low-light image to update the model parameters of the image enhancement model.

Images