A convolutional neural network image denoising method based on self-supervised learning

Abstract

The application relates to a convolutional neural network image denoising method based on self-supervised learning. The method comprises the following steps: acquiring a noisy image sample and a pre-constructed image denoising model; inputting the noisy image sample into a noise removal model and a noise estimation model respectively to obtain a predicted noise-free image and predicted noise; obtaining a variance loss function according to the pixel variance of the noisy image sample, the predicted noise and the predicted noise-free image, and the pixel-by-pixel covariance of the predicted noise; constructing a model loss function according to the variance loss function, a pre-constructed mean loss function, a pre-constructed data loss function and a pre-constructed enhancement loss function; optimizing the model loss function of the image denoising model to obtain a trained image denoising model; and performing image denoising by using the trained image denoising model. The method does not need to collect noise-free images corresponding to the noisy images, and can better retain the original image detail information while removing the noise.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision and pattern recognition, and in particular to a convolutional neural network image denoising method based on self-supervised learning. Background Art

[0002] Image denoising is a fundamental problem in the fields of computer vision and pattern recognition, with important theoretical value and broad application prospects. Traditional image processing and machine learning methods have solved some of the problems in image denoising, especially achieving good results in removing Gaussian white noise.

[0003] In recent years, image denoising methods based on deep neural networks have attracted attention from researchers and industry due to their significant improvements in denoising performance. Existing deep neural network-based image denoising methods are mostly supervised, requiring a large number of noisy-noiseless image pairs to train the models. However, in real-world scenarios, due to factors such as object motion, lighting changes, and camera shake, it is difficult to collect the noise-free images corresponding to noisy images. Therefore, the difficulty of collecting noise-free images corresponding to noisy images in real-world scenarios needs to be addressed urgently. Summary of the Invention

[0004] Based on this, it is necessary to provide a convolutional neural network image denoising method based on self-supervised learning to address the above technical problems.

[0005] A convolutional neural network image denoising method based on self-supervised learning, the method comprising:

[0006] Obtaining a noisy image sample and a pre-built image denoising model; the image denoising model includes a noise removal model and a noise estimation model;

[0007] Inputting the noisy image sample into the noise removal model and the noise estimation model respectively to obtain a predicted noise-free image and predicted noise;

[0008] Obtaining a variance loss function according to the noisy image sample, the predicted noise, and the pixel variance of the predicted noise-free image, and a pixel-by-pixel covariance of the predicted noise, and constructing a model loss function according to the variance loss function, a pre-constructed mean loss function, a pre-constructed data loss function, and a pre-constructed enhancement loss function;

[0009] Optimizing the model loss function of the image denoising model to obtain a trained image denoising model, and using the trained image denoising model to perform image denoising.

[0010] In one embodiment, the method further includes: obtaining a variance loss function based on the pixel variance of the noisy image sample, the predicted noise, and the predicted noise-free image, and the pixel-by-pixel covariance of the predicted noise:

[0011]

[0012] Among them, L var is the variance loss function, is the pixel variance of the k-th noisy image sample, is the pixel variance of the predicted noise, φ is the parameter to be learned in the noise estimation model, To predict the pixel variance of the noise-free image, θ is the parameter to be learned in the noise removal model, is the pixel-wise covariance, is the kth noisy image sample, k=1,…,B, B is the number of samples in the noisy image sample set, To predict a noise-free image, To predict noise.

[0013] In one embodiment, the method further includes: obtaining a mean loss function based on the mean of each channel of the noisy image sample and the predicted noise-free image:

[0014]

[0015] Among them, L mean is the mean loss function, mean(·) means calculating the mean of each channel of the input image, ‖·‖2 means calculating the 2-norm, is the kth noisy image sample, k=1,…,B, B is the number of samples in the noisy image sample set, To predict the noise-free image, θ is the parameter to be learned in the noise removal model.

[0016] In one embodiment, the method further includes: obtaining a data loss function based on the sum of the predicted noise-free image and the predicted noise and the input noisy image sample:

[0017]

[0018] Among them, L data is the data loss function, is the kth noisy image sample, k=1,…,B, B is the number of samples in the noisy image sample set, To predict a noise-free image, To predict noise, φ is the parameter to be learned in the noise estimation model, and θ is the parameter to be learned in the noise removal model.

[0019] In one embodiment, the method further includes: constructing a simulated noise image based on the predicted noise and the predicted noise-free image; and constructing an enhancement loss function based on the denoised simulated noise image and the predicted noise-free image:

[0020]

[0021] in, is the i-th enhancement loss function, To simulate noisy images, α and β are randomly generated coefficients, is the kth noisy image sample, k=1,…,B, B is the number of samples in the noisy image sample set, To predict a noise-free image, To predict noise, φ is the parameter to be learned in the noise estimation model, and θ is the parameter to be learned in the noise removal model.

[0022] In one embodiment, the model loss function is:

[0023]

[0024] Among them, L is the model loss function, L mean is the mean loss function, L var is the variance loss function, L data is the data loss function, is the i-th enhancement loss function.

[0025] In one embodiment, the noise removal model includes multiple convolutional layers and nonlinear restrictive units, and the noise estimation model includes multiple convolutional layers and nonlinear restrictive units.

[0026] A convolutional neural network image denoising device based on self-supervised learning, the device comprising:

[0027] Obtaining a noisy image sample and a pre-built image denoising model; the image denoising model includes a noise removal model and a noise estimation model;

[0028] Inputting the noisy image sample into the noise removal model and the noise estimation model respectively to obtain a predicted noise-free image and predicted noise;

[0029] Obtaining a variance loss function according to the noisy image sample, the predicted noise, and the pixel variance of the predicted noise-free image, and a pixel-by-pixel covariance of the predicted noise, and constructing a model loss function according to the variance loss function, a pre-constructed mean loss function, a pre-constructed data loss function, and a pre-constructed enhancement loss function;

[0030] Optimizing the model loss function of the image denoising model to obtain a trained image denoising model, and using the trained image denoising model to perform image denoising.

[0031] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0032] Obtaining a noisy image sample and a pre-built image denoising model; the image denoising model includes a noise removal model and a noise estimation model;

[0033] Inputting the noisy image sample into the noise removal model and the noise estimation model respectively to obtain a predicted noise-free image and predicted noise;

[0034] Obtaining a variance loss function according to the noisy image sample, the predicted noise, and the pixel variance of the predicted noise-free image, and a pixel-by-pixel covariance of the predicted noise, and constructing a model loss function according to the variance loss function, a pre-constructed mean loss function, a pre-constructed data loss function, and a pre-constructed enhancement loss function;

[0035] Optimizing the model loss function of the image denoising model to obtain a trained image denoising model, and using the trained image denoising model to perform image denoising.

[0036] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0037] Obtaining a noisy image sample and a pre-built image denoising model; the image denoising model includes a noise removal model and a noise estimation model;

[0038] Inputting the noisy image sample into the noise removal model and the noise estimation model respectively to obtain a predicted noise-free image and predicted noise;

[0039] Obtaining a variance loss function according to the noisy image sample, the predicted noise, and the pixel variance of the predicted noise-free image, and a pixel-by-pixel covariance of the predicted noise, and constructing a model loss function according to the variance loss function, a pre-constructed mean loss function, a pre-constructed data loss function, and a pre-constructed enhancement loss function;

[0040] Optimizing the model loss function of the image denoising model to obtain a trained image denoising model, and using the trained image denoising model to perform image denoising.

[0041] The above-mentioned convolutional neural network image denoising method based on self-supervised learning obtains a predicted noise-free image and predicted noise by inputting the noisy image samples into the noise removal model and the noise estimation model respectively, obtains the pixel variance of the noisy image samples, the predicted noise and the predicted noise-free image, and the pixel-by-pixel covariance of the predicted noise, obtains the variance loss function, uses the variance loss function to constrain the predicted noise-free image and the predicted noise so that they do not tend to zero or the noisy image samples, constructs a model loss function according to the variance loss function, a pre-constructed mean loss function, a pre-constructed data loss function and a pre-constructed enhancement loss function, uses the mean loss function to constrain the noisy image samples and the predicted denoised image so that each channel of the two has the same mean, uses the data loss function to make the sum of the predictions of the noise removal model and the noise estimation model close to the noisy image samples, uses the enhancement loss function to enhance the training samples, optimizes the model loss function of the image denoising model to obtain a trained image denoising model, and uses the trained image denoising model to perform image denoising. In the embodiment of the present invention, there is no need to collect a noise-free image corresponding to a noisy image, and the detail information of the original image is better preserved while removing the noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 1 is a flow chart of a convolutional neural network image denoising method based on self-supervised learning in one embodiment;

[0043] Figure 2 Schematic diagram of the structure of an image denoising model in one embodiment;

[0044] Figure 3 Figure 1 is a schematic diagram of the denoising effect of an actual sRGB noise image in a specific embodiment, wherein (a) is the actual sRGB noise image, (b) is a schematic diagram of the denoising effect of the AP-BSN method, and (c) is a schematic diagram of the denoising effect of the method of the present invention;

[0045] Figure 4 1 is a block diagram of a convolutional neural network image denoising device based on self-supervised learning in one embodiment;

[0046] Figure 5 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0048] In one embodiment, Figure 1As shown, a convolutional neural network image denoising method based on self-supervised learning is provided, comprising the following steps:

[0049] Step 102: Obtain a noisy image sample and a pre-built image denoising model.

[0050] like Figure 2 As shown, the image denoising model includes a noise removal model and a noise estimation model. The noise removal model is f θ (·), where θ is the parameter to be learned in the neural network, and the noise estimation model is g φ (·), where φ is the parameter to be learned in the neural network.

[0051] The actual noise image is I N ∈R m×n×c , the clear image to be restored is I C ∈R m×n×c , the additive zero-mean noise is N∈R m×n×c , where m is the number of rows of the image, n is the number of columns of the image, and c is the number of channels of the image. Thus, the imaging model is expressed as:

[0052] I N =I C +N.

[0053] Next, we need to design a suitable training method, especially the loss function, based only on the noisy image I N Optimize network parameters. In order to train the noise removal model f θ (·) and the noise estimation model g φ (·), a series of loss functions need to be designed according to the characteristics of image noise, and batches of image blocks are intercepted from the noisy image dataset as training data, which is recorded as the set The stochastic gradient descent method is used to optimize the convolutional neural network parameters, and finally a convolutional neural network with denoising ability is obtained.

[0054] Step 104: input the noisy image samples into the noise removal model and the noise estimation model respectively to obtain the predicted noise-free image and the predicted noise.

[0055] Step 106, obtain a variance loss function based on the noisy image samples, the pixel variance of the predicted noise and the predicted noise-free image, and the pixel-by-pixel covariance of the predicted noise, and construct a model loss function based on the variance loss function, the pre-constructed mean loss function, the pre-constructed data loss function, and the pre-constructed enhancement loss function.

[0056] Considering that additive noise has the characteristic of zero mean, the noisy image sample and predict noise-free images Each channel should have the same mean, thus establishing a mean loss function. Although the mean loss function is correct in constraining the model, it is still insufficient. The main reason is that the model can degenerate into two special trivial mappings: identity mapping and constant zero mapping. Therefore, this patent considers predicting noise-free images. Prediction noise Constrain it so that it does not approach zero or have noisy image samples Pixel-by-pixel constraints are obtained, allowing the variance loss function to be constructed. Considering that the sum of the predictions from the noise removal and noise estimation models should be a noisy image sample, a data loss function is constructed. To further reduce the fluctuations of the resulting model and improve accuracy, the training data can be augmented using predicted noise and predicted noise-free images. Based on this, an augmentation loss function is constructed.

[0057] Step 108: Optimize the model loss function of the image denoising model to obtain a trained image denoising model, and use the trained image denoising model to perform image denoising.

[0058] In the above-mentioned convolutional neural network image denoising method based on self-supervised learning, the noisy image samples are input into the noise removal model and the noise estimation model respectively to obtain the predicted noise-free image and the predicted noise. The variance loss function is obtained according to the pixel variance of the noisy image samples, the predicted noise and the predicted noise-free image, and the pixel-by-pixel covariance of the predicted noise. The variance loss function is used to constrain the predicted noise-free image and the predicted noise so that they do not tend to zero or the noisy image samples. The model loss function is constructed according to the variance loss function, a pre-constructed mean loss function, a pre-constructed data loss function and a pre-constructed enhancement loss function. The mean loss function is used to constrain the noisy image samples and the predicted denoised image so that each channel of the two has the same mean. The data loss function is used to make the sum of the predictions of the noise removal model and the noise estimation model close to the noisy image samples. The enhancement loss function is used to enhance the training samples. The model loss function of the image denoising model is optimized to obtain a trained image denoising model, and the trained image denoising model is used to perform image denoising. In the embodiment of the present invention, there is no need to collect a noise-free image corresponding to a noisy image, and the detail information of the original image is better preserved while removing the noise.

[0059] In one embodiment, Figure 2As shown, a schematic diagram of the structure of an image denoising model is provided. The noise removal model includes multiple convolutional layers and nonlinear restrictive units, and the noise estimation model includes multiple convolutional layers and nonlinear restrictive units. In this embodiment, the specific structure of the noise removal model is composed of 17 layers of 3×3 convolutional layers (Conv) and nonlinear restrictive units (ReLU) connected in sequence, and the specific structure of the noise estimation model is composed of 14 layers of 3×3 convolutional layers (Conv) and nonlinear restrictive units (ReLU) connected in sequence.

[0060] In one embodiment, the step of constructing the mean loss function includes: obtaining the mean loss function as follows based on the noisy image samples and the mean of each channel of the predicted noise-free image:

[0061]

[0062] Among them, L mean is the mean loss function, mean(·) means calculating the mean of each channel of the input image, ‖·‖2 means calculating the 2-norm, is the kth noisy image sample, k=1,…,B, B is the number of samples in the noisy image sample set, To predict the noise-free image, θ is the parameter to be learned in the noise removal model. In this embodiment, considering that the additive noise has the characteristic of zero mean, the noisy image sample and predict noise-free images Each channel should have the same mean, thus establishing a mean loss function.

[0063] In one embodiment, obtaining a variance loss function based on the noisy image samples, the pixel variance of the predicted noise and the predicted noise-free image, and the pixel-by-pixel covariance of the predicted noise includes: obtaining a variance loss function based on the noisy image samples, the pixel variance of the predicted noise and the predicted noise-free image, and the pixel-by-pixel covariance of the predicted noise:

[0064]

[0065] Among them, L var is the variance loss function, is the pixel variance of the k-th noisy image sample, is the pixel variance of the predicted noise, φ is the parameter to be learned in the noise estimation model, To predict the pixel variance of the noise-free image, θ is the parameter to be learned in the noise removal model, is the pixel-wise covariance, is the kth noisy image sample, k=1,…,B, B is the number of samples in the noisy image sample set, To predict a noise-free image, To predict noise.

[0066] In this embodiment, during the neural network training process, each training optimization needs to extract a batch from the input noisy image. The pixel variance of the input noisy image sample is recorded as and With the same dimensions, the pixel variance calculation formula at the pixel position (x, y) is defined as:

[0067]

[0068] in, is the pixel variance at pixel location (x,y), is the pixel of the k-th noise image sample at the (x,y) coordinate position, Pixels The expected value of is calculated from the mean of all pixels in a neighborhood patch of size p×p centered at the (x,y) position. When the side length p of the segmented patch is small (e.g., p=5), the image content within the patch can be considered locally flat, so the variance between the input noisy image sample and the predicted noise should be approximately equal, and the loss function is:

[0069]

[0070] Considering that the variance is approximately equal but not exactly equal, we design an adversarial loss function with exactly equal variance to mitigate the error caused by approximation:

[0071] Where λ is a constant, is the pixel-by-pixel covariance, and the calculation formula at the pixel position (x, y) is:

[0072]

[0073] In one embodiment, the step of constructing a data loss function includes: obtaining the data loss function as follows based on the sum of the predicted noise-free image and the predicted noise and the input noisy image sample:

[0074]

[0075] Among them, L data is the data loss function, is the kth noisy image sample, k=1,…,B, B is the number of samples in the noisy image sample set, To predict a noise-free image, To predict noise, φ is the parameter to be learned in the noise estimation model, and θ is the parameter to be learned in the noise removal model.

[0076] In one embodiment, the step of constructing the enhancement loss function includes: constructing a simulated noise image based on the predicted noise and the predicted noise-free image; and constructing the enhancement loss function based on the denoised simulated noise image and the predicted noise-free image as follows:

[0077]

[0078] in, is the i-th enhancement loss function, To simulate noisy images, α and β are randomly generated coefficients, is the kth noisy image sample, k=1,…,B, B is the number of samples in the noisy image sample set, To predict a noise-free image, To predict the noise, is the parameter to be learned in the noise estimation model, and θ is the parameter to be learned in the noise removal model. In this embodiment, a simulated noisy image is constructed, and the simulated noisy image should be close to the predicted noise-free image after denoising. The superscripts in indicate that this loss function can be constructed in multiple sets. Specifically, when α = 1 and β = 0, the enhanced loss function states that the image denoised by the convolutional neural network should be noise-free; when α = 0 and β = 1, the enhanced loss function states that the estimated noise output by the noise removal model should be 0. These two special parameter sets are used as the first and second sets, and two more sets of α ~ U (0.2, 1.0) and β ~ U (0, 2) are randomly selected from a uniform distribution.

[0079] In one embodiment, the model loss function is:

[0080]

[0081] Among them, L is the model loss function, L mean is the mean loss function, L var is the variance loss function, L data is the data loss function, is the i-th enhancement loss function.

[0082] In a specific embodiment, Figure 3 The figure shows the denoising effect of an actual sRGB noise image, where (a) is the actual sRGB noise image, (b) is the denoising effect of the AP-BSN method, and (c) is the denoising effect of the method of the present invention. The model training and parameter optimization adopt the stochastic gradient descent method, the batch size is 64, the optimization operator adopts the Adam operator, and the initial learning rate is 1e -4, the learning rate is reduced to 1 / 10 of the current one every 200 iterations; take parameters s = 5, λ = 4, and the total number of training iterations is 500; in each iterative optimization, first calculate the data loss function, and then calculate the gradient through the optimization operator to update the denoising model parameters θ. The noisy image is directly input into the trained model f θ (·), the output is the denoising result. Figure 2 Compared with the AP-BSN method published in CVPR2022, the proposed method can better preserve the original image details while removing noise.

[0083] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0084] In one embodiment, Figure 4 As shown, a convolutional neural network image denoising device based on self-supervised learning is provided, comprising: a model acquisition module 402, a sample input module 404, a loss function construction module 406 and an image denoising module 408, wherein:

[0085] Model acquisition module 402, used to obtain noisy image samples and a pre-built image denoising model; the image denoising model includes a noise removal model and a noise estimation model;

[0086] The sample input module 404 is used to input the noisy image samples into the noise removal model and the noise estimation model respectively to obtain the predicted noise-free image and the predicted noise;

[0087] a loss function construction module 406 for obtaining a variance loss function based on the noisy image samples, the pixel variance of the predicted noise and the predicted noise-free image, and the pixel-by-pixel covariance of the predicted noise, and constructing a model loss function based on the variance loss function, a pre-constructed mean loss function, a pre-constructed data loss function, and a pre-constructed enhancement loss function;

[0088] The image denoising module 408 is used to optimize the model loss function of the image denoising model to obtain a trained image denoising model, and use the trained image denoising model to perform image denoising.

[0089] In one embodiment, the variance loss function is obtained based on the noisy image samples, the pixel variance of the predicted noise and the predicted noise-free image, and the pixel-by-pixel covariance of the predicted noise:

[0090]

[0091] Among them, L var is the variance loss function, is the pixel variance of the k-th noisy image sample, is the pixel variance of the predicted noise, φ is the parameter to be learned in the noise estimation model, To predict the pixel variance of the noise-free image, θ is the parameter to be learned in the noise removal model, is the pixel-wise covariance, is the kth noisy image sample, k=1,…,B, B is the number of samples in the noisy image sample set, To predict a noise-free image, To predict noise.

[0092] In one embodiment, the mean loss function is obtained based on the noisy image samples and the mean of each channel of the predicted noise-free image:

[0093]

[0094] Among them, L mean is the mean loss function, mean(·) means calculating the mean of each channel of the input image, ‖·‖2 means calculating the 2-norm, is the kth noisy image sample, k=1,…,B, B is the number of samples in the noisy image sample set, To predict the noise-free image, θ is the parameter to be learned in the noise removal model.

[0095] In one embodiment, the data loss function is obtained based on the sum of the predicted noise-free image and the predicted noise and the input noisy image sample:

[0096]

[0097] Among them, L data is the data loss function, is the kth noisy image sample, k=1,…,B, B is the number of samples in the noisy image sample set, To predict a noise-free image, To predict noise, φ is the parameter to be learned in the noise estimation model, and θ is the parameter to be learned in the noise removal model.

[0098] In one embodiment, a simulated noisy image is constructed based on the predicted noise and the predicted noise-free image; and an enhancement loss function is constructed based on the denoised simulated noisy image and the predicted noise-free image:

[0099]

[0100] in, is the i-th enhancement loss function, To simulate noisy images, α and β are randomly generated coefficients, is the kth noisy image sample, k=1,…,B, B is the number of samples in the noisy image sample set, To predict a noise-free image, To predict noise, φ is the parameter to be learned in the noise estimation model, and θ is the parameter to be learned in the noise removal model.

[0101] In one embodiment, the loss function used for the model is:

[0102]

[0103] Among them, L is the model loss function, L mean is the mean loss function, L var is the variance loss function, L data is the data loss function, is the i-th enhancement loss function.

[0104] In one embodiment, the noise removal model includes multiple convolutional layers and nonlinear restrictive units, and the noise estimation model includes multiple convolutional layers and nonlinear restrictive units.

[0105] Regarding the specific definition of the convolutional neural network image denoising device based on self-supervised learning, please refer to the definition of the convolutional neural network image denoising method based on self-supervised learning above, which will not be repeated here. The various modules in the above-mentioned convolutional neural network image denoising device based on self-supervised learning can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0106] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a convolutional neural network image denoising method based on self-supervised learning is implemented. The display screen of the computer device can be a liquid crystal display or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0107] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0108] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.

[0109] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.

[0110] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0111] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0112] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and such modifications and improvements are all within the scope of protection of the present application. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A convolutional neural network image denoising method based on self-supervised learning, characterized in that: The method comprises: Obtaining a noisy image sample and a pre-built image denoising model; the image denoising model includes a noise removal model and a noise estimation model; Inputting the noisy image sample into the noise removal model and the noise estimation model respectively to obtain a predicted noise-free image and predicted noise; Obtaining a variance loss function according to the noisy image sample, the predicted noise, and the pixel variance of the predicted noise-free image, and a pixel-by-pixel covariance of the predicted noise, and constructing a model loss function according to the variance loss function, a pre-constructed mean loss function, a pre-constructed data loss function, and a pre-constructed enhancement loss function; Optimizing the model loss function of the image denoising model to obtain a trained image denoising model, and performing image denoising using the trained image denoising model; Obtaining a variance loss function based on the pixel variance of the noisy image sample, the predicted noise, and the predicted noise-free image, and the pixel-by-pixel covariance of the predicted noise includes: According to the pixel variance of the noisy image sample, the predicted noise and the predicted noise-free image, and the pixel-by-pixel covariance of the predicted noise, the variance loss function is obtained as follows: ， in, is the variance loss function, For the The pixel variance of a noisy image sample, is the pixel variance of the predicted noise, are the parameters to be learned in the noise estimation model, To predict the pixel variance of the noise-free image, are the parameters to be learned in the noise removal model, is the pixel-wise covariance, For the Noisy image samples, , is the number of samples in the noisy image sample set, To predict a noise-free image, To predict noise.

2. The method according to claim 1, characterized in that The steps to construct the mean loss function include: According to the mean of each channel of the noisy image sample and the predicted noise-free image, the mean loss function is obtained as follows: ， in, is the mean loss function, Indicates calculating the mean of each channel of the input image. Indicates the calculation of 2-norm, For the Noisy image samples, , is the number of samples in the noisy image sample set, To predict a noise-free image, Parameters to be learned in the noise removal model.

3. The method according to claim 1, characterized in that The steps to construct a data loss function include: According to the sum of the predicted noise-free image and the predicted noise and the input noisy image sample, the data loss function is obtained as follows: ， in, is the data loss function, For the Noisy image samples, , is the number of samples in the noisy image sample set, To predict a noise-free image, To predict the noise, are the parameters to be learned in the noise estimation model, Parameters to be learned in the noise removal model.

4. The method according to claim 1, wherein The steps to construct the enhanced loss function include: constructing a simulated noise image according to the predicted noise and the predicted noise-free image; According to the simulated noisy image and the predicted noise-free image after denoising, the enhancement loss function is constructed as follows: ， in, For the An enhanced loss function, To simulate noisy images, , and are randomly generated coefficients, For the Noisy image samples, , is the number of samples in the noisy image sample set, To predict a noise-free image, To predict the noise, are the parameters to be learned in the noise estimation model, Parameters to be learned in the noise removal model.

5. The method according to any one of claims 1 to 4, characterized in that The model loss function is: ， in, is the model loss function, is the mean loss function, is the variance loss function, is the data loss function, For the An enhanced loss function.

6. The method according to claim 1, characterized in that The noise removal model includes multiple convolutional layers and nonlinear restrictive units, and the noise estimation model includes multiple convolutional layers and nonlinear restrictive units.

7. A convolutional neural network image denoising device based on self-supervised learning implemented in the method according to any one of claims 1 to 6, characterized in that: The device comprises: A model acquisition module is used to acquire noisy image samples and a pre-built image denoising model; the image denoising model includes a noise removal model and a noise estimation model; A sample input module, configured to input the noisy image sample into the noise removal model and the noise estimation model respectively to obtain a predicted noise-free image and predicted noise; A loss function construction module is used to obtain a variance loss function based on the pixel variance of the noisy image sample, the predicted noise and the predicted noise-free image, and the pixel-by-pixel covariance of the predicted noise, and to construct a model loss function based on the variance loss function, a pre-constructed mean loss function, a pre-constructed data loss function, and a pre-constructed enhancement loss function; The image denoising module is used to optimize the model loss function of the image denoising model to obtain a trained image denoising model, and use the trained image denoising model to perform image denoising.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images