Low-rank recovery and deep diffusion fusion-based low-light image enhancement method

Abstract

The invention discloses a low-light image enhancement method based on low-rank recovery and deep diffusion fusion, and relates to the technical field of image processing. Comprising the following steps: inputting a to-be-processed low-light image into a trained low-light image enhancement model, and transforming a pixel value of the to-be-processed low-light image through a discrete cosine transform module to obtain a spectrogram in a spectral domain; performing forward noise addition at each time step through a Markov chain mechanism, capturing global features of noise distribution through a Transform network, and removing noise components in the current time step according to the global features of the noise distribution to obtain an intermediate image; taking the intermediate image corresponding to the last time step as a new spectrogram; and performing inverse discrete cosine transform on the new spectrogram through an inverse discrete cosine transform module, and converting the new spectrogram from the frequency domain to the spatial domain to obtain an enhanced low-light image. The method pays attention to the globality of noise components, and can improve the definition and quality of the enhanced low-light image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a low-light image enhancement method based on low-rank recovery and deep diffusion fusion. Background Art

[0002] The visual quality of images acquired under low-light conditions is often greatly reduced, which not only makes the image look dim and blurry, but also loses a lot of detail information and significantly increases noise. Such image quality undoubtedly brings great challenges to downstream visual tasks such as subsequent image processing, feature extraction, and target recognition. Therefore, image enhancement of low-light images and improving the visual quality of images are the key to ensuring that downstream visual tasks can be performed accurately and efficiently.

[0003] In the prior art, the diffusion model has attracted attention for its performance in image generation and restoration tasks. First, the diffusion model analyzes the noise distribution characteristics in low-light images, and grasps the noise components in low-light images by accurately modeling the noise; then, the reverse diffusion process is used to gradually guide the image from a noisy low-light state to a clear, bright state; during the transformation process, the noise in the image is gradually eliminated, and the potential information and structural features in the image are used to gradually restore and enhance the details, brightness and contrast of the image to obtain an enhanced image.

[0004] The defects of the above-mentioned existing technology are: when the low-light image enhancement algorithm based on the diffusion model performs denoising and denoising on the low-light image, the texture and detail features captured are mainly concentrated in the spatial domain, resulting in obvious residual noise traces on the denoised image, which in turn affects the clarity and quality of the enhanced image. Summary of the invention

[0005] Based on this, it is necessary to provide a low-light image enhancement method based on low-rank recovery and deep diffusion fusion to address the above technical problems.

[0006] The embodiment of the present invention provides a low-light image enhancement method based on low-rank restoration and deep diffusion fusion, including: Obtain historical low-light images and historical high-light images corresponding to the historical low-light images; A Markov chain mechanism is added before the Transformer network to form a global depth diffusion generation module. The input end of the global depth diffusion generation module is connected to the output end of the discrete cosine transform module, and the output end of the global depth diffusion generation module is connected to the input end of the inverse discrete cosine transform module. The input end of the discrete cosine transform module is used as the input end of the low-light image enhancement model, and the output end of the inverse discrete cosine transform module is used as the output end of the low-light image enhancement model to obtain the low-light image enhancement model. Inputting the historical low-light image into the low-light image enhancement model to obtain an enhancement result of the historical low-light image; training the low-light image enhancement model based on the enhancement result of the historical low-light image and the historical high-light image corresponding to the historical low-light image; The low-light image to be processed is input into the trained low-light image enhancement model, and the pixel values ​​of the low-light image to be processed are transformed through the discrete cosine transform module to obtain a spectrum map in the spectral domain; the spectrum map is forward denoised at each time step through the Markov chain mechanism in the global deep diffusion generation module, and then the global characteristics of the noise distribution are captured through the Transformer network in the global deep diffusion generation module; in each time step, the noise component in the current time step is removed according to the global characteristics of the noise distribution to obtain an intermediate image; the intermediate image corresponding to the last time step is used as the new spectrum map, and the new spectrum map is inversely discretely transformed through the inverse discrete cosine transform module to convert the new spectrum map from the spectral domain back to the spatial domain to obtain an enhanced low-light image.

[0007] Optionally, the pixel values ​​of the low-light image to be processed are transformed by a discrete cosine transform module, specifically including: Discrete cosine transform decomposes the pixel values ​​of the low-light image to be processed into a weighted sum of different frequency components; The formula for the transformation process is: ; in, represents the coefficients in the frequency domain, Represents the pixel value of the original image, x and y Respectively represent the horizontal and vertical coordinates of the image, and represents a frequency variable, N Indicates the size of the image. represents the normalization factor; The definition of the normalization factor is: ; in, i Represents a formal parameter, numerically equal to u and v .

[0008] Optionally, the spectrogram is forward-noised at each time step through the Markov chain mechanism in the global deep diffusion generation module. The formula for forward noising is: ; in, t represents the time step, T represents the last time step, Represents the pictures generated at each step, Represents the original image, represents pure Gaussian noise, represents Gaussian noise that satisfies the (0,1) distribution, represents a variable that gets smaller as the time step increases; The relationship between variables and hyperparameters is: ; in, represents the variable at time step t, represents a hyperparameter that gets larger as the number of time steps increases.

[0009] Optionally, the global features of the noise distribution are captured by the Transformer network in the global deep diffusion generation module. The calculation formula of the global features is: ; in, represents the local feature matrix after dynamic convolution processing, Represents the local feature matrix The transposed matrix of Represents global features The dimension of .

[0010] Optionally, the noise component in the current time step is removed according to the global characteristics of the noise distribution. The removal process is formulated as: ; in, represents the time step in the diffusion process t The sample at represents the noise prediction function of the model, Represents the time step t The coefficient that controls the amount of noise, represents random noise from a standard normal distribution, represents a variable that gets smaller as the time step increases, Represents the time step t Variables when .

[0011] Optionally, performing an inverse discrete cosine transform on the new spectrum graph through an inverse discrete cosine transform module specifically includes: The frequency domain information is converted back to the spatial domain through inverse discrete cosine transform, so that the image can be reconstructed into a visible form; The inverse discrete cosine transform formula is expressed as: ; in, represents the coefficients in the frequency domain, Represents the pixel value of the original image, x and yRespectively represent the horizontal and vertical coordinates of the image, and represents a frequency variable, N Indicates the size of the image. Represents the normalization factor.

[0012] The low-light image enhancement method based on low-rank restoration and deep diffusion fusion provided by the embodiment of the present invention has the following beneficial effects compared with the prior art: The present invention transforms the pixel values ​​of the low-light image to be processed through a discrete cosine transform module, so that the structural information of the low-light image to be processed can be analyzed more clearly in the spectral domain, effectively improving the generation efficiency; forward noise is performed at each time step through the Markov chain mechanism in the global depth diffusion generation module, and then the global characteristics of the noise distribution are captured through the Transformer network in the global depth diffusion generation module, and the noise components in the current time step are removed according to the global characteristics of the noise distribution to obtain an intermediate image; the intermediate image corresponding to the last time step is used as a new spectrum diagram. In this process, the globality of the noise components is emphasized, which can improve the clarity and quality of the enhanced low-light image. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A processing flow chart of a low-light image enhancement method based on low-rank restoration and deep diffusion fusion provided in an embodiment; Figure 2 A schematic diagram of the discrete cosine transform effect of a low-light image enhancement method based on low-rank recovery and deep diffusion fusion provided in an embodiment; Figure 3 It is a loss function convergence graph of a low-light image enhancement method based on low-rank recovery and deep diffusion fusion provided in an embodiment. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0015] In one embodiment, a low-light image enhancement method based on low-rank restoration and deep diffusion fusion is provided, the method comprising: Obtain the historical low-light image and the historical high-light image corresponding to the historical low-light image.

[0016] A Markov chain mechanism is added before the Transformer network to form a global depth diffusion generation module. The input of the global depth diffusion generation module is connected to the output of the discrete cosine transform module, and the output of the global depth diffusion generation module is connected to the input of the inverse discrete cosine transform module. The input of the discrete cosine transform module is used as the input of the low-light image enhancement model, and the output of the inverse discrete cosine transform module is used as the output of the low-light image enhancement model to obtain a low-light image enhancement model, such as Figure 1 shown.

[0017] The historical low-light image is input into the low-light image enhancement model to obtain the enhancement result of the historical low-light image. The low-light image enhancement model is trained based on the enhancement result of the historical low-light image and the historical high-light image corresponding to the historical low-light image.

[0018] The low-light image to be processed is input into the trained low-light image enhancement model, and the pixel values ​​of the low-light image to be processed are transformed through the discrete cosine transform module to obtain a spectrum map in the spectral domain. The spectrum map is forward-noised at each time step through the Markov chain mechanism in the global deep diffusion generation module, and then the global characteristics of the noise distribution are captured through the Transformer network in the global deep diffusion generation module. In each time step, the noise component in the current time step is removed according to the global characteristics of the noise distribution to obtain an intermediate image. The intermediate image corresponding to the last time step is used as the new spectrum map, and the new spectrum map is inversely discretely transformed through the inverse discrete cosine transform module to convert the new spectrum map from the spectral domain back to the spatial domain to obtain an enhanced low-light image.

[0019] A specific embodiment of the present invention is provided: S1. Construct a low-light image enhancement model based on low-rank recovery and deep diffusion fusion; wherein the low-light image enhancement model includes a discrete cosine transform module, a global deep diffusion generation module, and an inverse discrete cosine transform module; S2. Preprocess the low-light images and normal images in the dataset, including resizing and normalization, to ensure the quality and consistency of the input data. The preprocessed images are sent to the discrete cosine transform module to obtain the spectrum corresponding to each image. Each point in the spectrum represents the components of different frequencies in the image. Figure 2 As shown in the figure, this module converts the information of the image spatial domain into the spectral domain, performs high-frequency truncation to reduce high-frequency data, and finally converts the information back to the spatial domain through the inverse discrete cosine transform, thus realizing the conversion of information between the spatial domain and the spectral domain.

[0020] Specifically, the discrete cosine transform decomposes the pixel values ​​of the low-light image to be processed and expresses it as a weighted sum of different frequency components. In this way, the structural information of the image can be analyzed more clearly in the spectrum domain. The formula for discrete cosine transform is: ; in, represents the coefficients in the frequency domain, Represents the pixel value of the original image, x and y Respectively represent the horizontal and vertical coordinates of the image, and represents a frequency variable, N Indicates the size of the image. represents the normalization factor, defined as: ; in, i is a formal parameter, which is numerically equal to u and v .

[0021] Specifically, the data matrix of natural images is usually low-rank or approximately low-rank, with most of the energy concentrated in a few low-frequency components, which reflect the main features of the image. However, the noise and errors generated during the acquisition or processing process appear in the form of high-frequency components, destroying the low-rank nature of the image. In order to restore the low-rank nature of the image and denoise it, high-frequency truncation can be performed in the spectral domain to retain the low-frequency components after DCT transformation, thereby effectively removing noise and retaining core features, and effectively accelerating the training and inference process.

[0022] S3, the spectrum graph obtained from S2 is input into the diffusion model, and the Markov chain mechanism is used for forward noise addition. At each time step, the model gradually adds noise to the spectrum graph until it approaches Gaussian noise. The Transformer network is used to learn the noise distribution at each time step, capture the global characteristics of the noise distribution, and extract relevant information from the noise through the self-attention mechanism, so as to better simulate the noise characteristics of the real data at each time step.

[0023] Specifically, the spectrogram after discrete cosine transformation is forward-noised using the Markov chain mechanism. At each time step, the model gradually adds noise to the spectrogram to generate noisy samples. The specific formula is as follows: ; in, t represents the time step, T represents the last time step, Represents the pictures generated at each step, Represents the original image, represents pure Gaussian noise, represents Gaussian noise that satisfies the (0,1) distribution, Represents a variable that gets smaller as the time step increases.

[0024] The relationship between variables and hyperparameters is: ; in, Represents the time step t Variables when represents a hyperparameter that gets larger as the number of time steps increases.

[0025] At the same time, the Transformer network obtains global information through the residual self-attention mechanism. This mechanism enhances the weight of the original features and the results of the previous layer attention mechanism in the subsequent attention operation, making the global information more robust. Incorporating it into the global deep diffusion generation module, the formula is as follows: ; in, represents the local feature matrix after dynamic convolution processing, Represents the local feature matrix The transposed matrix of Represents global features The dimension of .

[0026] S4. Initialize a noise combined with low-light images, using the noise distribution learned by the Transformer network, at each time step t In the process, the model refers to the current input noisy image and predicts and removes the noise component of the current step based on the learned noise distribution. This denoising process is carried out step by step. The model gradually removes the noise through a series of time steps, and each step generates a clearer intermediate image until a noise-free spectrum close to the real data is restored.

[0027] Specifically, it includes: removing the noise component in the current time step according to the global characteristics of the noise distribution, and the calculation formula is: ; in, represents the time step in the diffusion process t The sample at represents the noise prediction function of the model, Represents the time step t The coefficient that controls the amount of noise, represents random noise from a standard normal distribution, represents a variable that gets smaller as the time step increases, Represents the time step t Variables when .

[0028] S5. Perform an inverse discrete cosine transform on the spectrum graph to convert the frequency domain information back to the spatial domain so that the image can be reconstructed into a visual form. This method improves the clarity and visual quality of the image, making the restored highlight image more realistic and natural.

[0029] Specifically, by performing an inverse discrete cosine transform on the spectrum graph, the frequency domain information can be effectively converted back to the spatial domain. Because the spectrum image only contains frequency components and lacks direct visual information. The implementation of the inverse discrete cosine transform not only allows the visual form of the image to be reconstructed, but also ensures the integrity of its main features. The specific formula is as follows: ; in, represents the coefficients in the frequency domain, Represents the pixel value of the original image, x and y Respectively represent the horizontal and vertical coordinates of the image, which is the same as the case in discrete spectrum transform. N Indicates the size of the image. Represents the normalization factor.

[0030] In this process, the inverse discrete cosine transform recombines all frequency components so that each pixel value can reflect the corresponding frequency information. This conversion not only restores the visibility of the image, but also maintains the overall quality and details of the image, making the generated image visually clearer, more natural and more detailed.

[0031] Exemplarily, the loss function change curve and convergence diagram of the algorithm based on the diffusion model and the present invention when trained on the low-light LOL dataset are provided as follows: Figure 3 As shown, it can be seen that the loss function converges faster and the training loss is smaller in the present invention, which shows that the present invention can effectively improve the model training speed and accuracy.

[0032] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A low-light image enhancement method based on low-rank recovery and deep diffusion fusion, characterized in that: include: Obtain historical low-light images and historical high-light images corresponding to the historical low-light images; A Markov chain mechanism is added before the Transformer network to form a global depth diffusion generation module. The input end of the global depth diffusion generation module is connected to the output end of the discrete cosine transform module, and the output end of the global depth diffusion generation module is connected to the input end of the inverse discrete cosine transform module. The input end of the discrete cosine transform module is used as the input end of the low-light image enhancement model, and the output end of the inverse discrete cosine transform module is used as the output end of the low-light image enhancement model to obtain the low-light image enhancement model. Input the historical low-light image into the low-light image enhancement model to obtain the enhancement result of the historical low-light image; Based on the enhancement results of historical low-light images and the historical high-light images corresponding to the historical low-light images, a low-light image enhancement model is trained; The low-light image to be processed is input into the trained low-light image enhancement model, and the pixel values ​​of the low-light image to be processed are transformed through the discrete cosine transform module to obtain a spectrum map in the spectral domain; the spectrum map is forward denoised at each time step through the Markov chain mechanism in the global deep diffusion generation module, and then the global characteristics of the noise distribution are captured through the Transformer network in the global deep diffusion generation module; in each time step, the noise component in the current time step is removed according to the global characteristics of the noise distribution to obtain an intermediate image; the intermediate image corresponding to the last time step is used as the new spectrum map, and the new spectrum map is inversely discretely transformed through the inverse discrete cosine transform module to convert the new spectrum map from the spectral domain back to the spatial domain to obtain an enhanced low-light image.

2. The low-light image enhancement method based on low-rank restoration and deep diffusion fusion as claimed in claim 1, characterized in that: The transforming of the pixel values ​​of the low-light image to be processed by the discrete cosine transform module specifically includes: Discrete cosine transform decomposes the pixel values ​​of the low-light image to be processed into a weighted sum of different frequency components; The formula for the transformation process is: ； in, represents the coefficients in the frequency domain, Represents the pixel value of the original image, x and y Respectively represent the horizontal and vertical coordinates of the image, and represents a frequency variable, N Indicates the size of the image. represents the normalization factor; The definition of the normalization factor is: ； in, i Represents a formal parameter, numerically equal to u and v .

3. The low-light image enhancement method based on low-rank restoration and deep diffusion fusion as claimed in claim 1, characterized in that: The Markov chain mechanism in the global deep diffusion generation module performs forward noise addition on the spectrum graph at each time step. The formula for forward noise addition is: ； in, t represents the time step, T represents the last time step, Represents the pictures generated at each step, Represents the original image, represents pure Gaussian noise, represents Gaussian noise that satisfies the (0,1) distribution, represents a variable that gets smaller as the time step increases; The relationship between variables and hyperparameters is: ； in, Represents the time step t Variables when represents a hyperparameter that gets larger as the number of time steps increases.

4. The low-light image enhancement method based on low-rank restoration and deep diffusion fusion as claimed in claim 1, characterized in that: The global features of the noise distribution are captured by the Transformer network in the global deep diffusion generation module. The calculation formula of the global features is: ； in, represents the local feature matrix after dynamic convolution processing, Represents the local feature matrix The transposed matrix of Represents global features The dimension of .

5. The low-light image enhancement method based on low-rank restoration and deep diffusion fusion as claimed in claim 1, characterized in that: The noise component in the current time step is removed according to the global characteristics of the noise distribution, and the formula of the removal process is: ； in, represents the time step in the diffusion process t The sample at represents the noise prediction function of the model, Represents the time step t The coefficient that controls the amount of noise, represents random noise from a standard normal distribution, represents a variable that gets smaller as the time step increases, Represents the time step t Variables when .

6. The low-light image enhancement method based on low-rank restoration and deep diffusion fusion as claimed in claim 1, characterized in that: The inverse discrete cosine transform of the new spectrum graph is performed by the inverse discrete cosine transform module, specifically comprising: The frequency domain information is converted back to the spatial domain through inverse discrete cosine transform, so that the image can be reconstructed into a visible form; The inverse discrete cosine transform formula is expressed as: ； in, represents the coefficients in the frequency domain, represents the pixel value of the original image, x and y Respectively represent the horizontal and vertical coordinates of the image, and represents a frequency variable, N Indicates the size of the image. Represents the normalization factor.

Images