Image denoising method, device and storage medium

Abstract

The present application proposes an image denoising method, device and storage medium, the method comprising: obtaining an image to be processed, inputting the image to be processed into an image denoising model, and obtaining a denoised image, wherein the image denoising model is a model formed by combining a U-shaped network, a residual network and a dense network. The image denoising model in this embodiment can combine the characteristics of the residual network and the dense network to better utilize the deep and shallow features of the image to be processed, and can achieve the goal of retaining the details of the image to be processed as much as possible while removing the noise, thereby achieving better denoising performance while taking into account the quality of the denoised image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image denoising method, device and storage medium. Background Art

[0002] Image denoising has always been a very important direction in the field of image processing. In recent years, the surveillance field has been popular with starlight-level shooting. The lighting conditions in the dark environment are very bad, and the sensor is not sensitive enough, resulting in a lot of noise in the collected pictures. The received images or videos have lower resolution than those collected under good lighting conditions, which not only affects the visual effect, but also affects the accuracy of the recognition work for images or videos that need to identify moving targets. Therefore, better denoising algorithms are needed to improve image quality.

[0003] At present, the adaptive denoising algorithms based on neural networks that have better effects are all supervised learning type neural network algorithms, that is, the training samples include input sample and output sample pairs, and the parameters in the neural network are updated through the gradient descent algorithm so that the output of the training sample after the input passes through the neural network is close to the true value sample.

[0004] However, the current adaptive denoising algorithms cannot achieve good denoising performance while also taking into account the quality of the denoised image. Summary of the invention

[0005] The present application provides an image denoising method, device and storage medium to solve the current technical problem of not being able to take into account both denoising performance and the quality of the image after denoising.

[0006] The present application provides an image denoising method, comprising:

[0007] Get the image to be processed;

[0008] The image to be processed is input into an image denoising model to obtain a denoised image; wherein the image denoising model is a model formed by combining a U-type network, a residual network and a dense network.

[0009] The present application provides an image denoising device, comprising:

[0010] A first acquisition module is configured to acquire an image to be processed;

[0011] The second acquisition module is configured to input the image to be processed into an image denoising model to obtain a denoised image; wherein the image denoising model is a model formed by combining a U-type network, a residual network and a dense network.

[0012] The present application provides an image denoising device, comprising:

[0013] one or more processors;

[0014] A memory for storing one or more programs;

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement any one of the image denoising methods in the embodiments of the present application.

[0016] An embodiment of the present application provides a storage medium, which stores a computer program. When the computer program is executed by a processor, any one of the image denoising methods in the embodiments of the present application is implemented.

[0017] With regard to the above embodiments and other aspects of the present application and their implementation, further description is provided in the accompanying drawings, detailed description and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of an image denoising method provided by an embodiment;

[0019] Figure 2 A schematic diagram of the structure of an image denoising model provided by an embodiment;

[0020] Figure 3 A schematic diagram of the structure of a first dense residual submodule in an image denoising model provided by an embodiment;

[0021] Figure 4 A flowchart of another image denoising method provided by an embodiment;

[0022] Figure 5A A flowchart of obtaining a first training noise image and a corresponding first training true value image provided by an embodiment;

[0023] Figure 5B A flowchart of generating a second training true value image provided by an embodiment;

[0024] Figure 6 A schematic diagram of the structure of a generative network in an adversarial network provided by an embodiment;

[0025] Figure 7 A schematic diagram of the structure of a discriminant network in an adversarial network provided by an embodiment;

[0026] Figure 8 A schematic diagram of the structure of an image denoising device provided by an embodiment;

[0027] Fig. 9 A schematic diagram of the structure of another image denoising device provided by an embodiment;

[0028] Fig.10 A schematic diagram of the structure of another image denoising device provided by an embodiment;

[0029] Fig.11 A schematic diagram of the structure of an image denoising device provided by an embodiment. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present application more clear, the embodiments of the present application will be described in detail below with reference to the accompanying drawings. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other arbitrarily without conflict.

[0031] Figure 1 The flowchart of an image denoising method provided by an embodiment is shown in FIG. This embodiment is applicable to a scenario where an image is denoised. This embodiment can be performed by an image denoising apparatus, which can be implemented by software and / or hardware, and can be integrated into an image denoising device. Figure 1 As shown, the image denoising method provided in this embodiment includes the following steps:

[0032] Step 101: Obtain an image to be processed.

[0033] Step 102: Input the image to be processed into the image denoising model to obtain the denoised image.

[0034] Among them, the image denoising model is a model formed by combining U-type network, residual network and dense network.

[0035] Specifically, the image to be processed in this embodiment can be an image captured by a front-end monitoring device, or an image frame in a video captured by a front-end monitoring device. Of course, the image to be processed in this embodiment can also be an image in other fields, such as a medical image.

[0036] The image denoising model in this embodiment is a pre-trained model. The model is a model formed by combining a U-type network, a residual network and a dense network. The U-type network in this embodiment, i.e., Unet, refers to a U-shaped network including feature extraction, i.e., the encoding part, and upsampling, i.e., the decoding part. The residual network in this embodiment refers to a network including a direct mapping part and a residual part, and the direct mapping part and the residual part perform an addition operation. The dense network in this embodiment refers to a network including a direct mapping part and a residual part, and the direct mapping part and the residual part perform a channel connection operation.

[0037] The image denoising model in this embodiment can combine the characteristics of the residual network and the dense network to better utilize the deep and shallow features of the image to be processed, and can achieve the goal of removing noise while retaining the details of the image to be processed as much as possible. Therefore, on the basis of achieving better denoising performance, the quality of the denoised image is taken into account.

[0038] Figure 2 FIG. 1 is a schematic diagram of the structure of an image denoising model provided by an embodiment. Figure 2 As shown, the image denoising model provided by this embodiment may include: an input layer, a first convolutional layer, at least one dense residual module, a dense residual block, at least one upsampling module, a second convolutional layer, a third convolutional layer and an output layer connected in sequence.

[0039] The output end of the input layer is also subtracted from the output end of the third convolutional layer, and the result of the subtraction operation is input to the input end of the output layer.

[0040] The output end of the first convolutional layer is also added to the output end of the second convolutional layer, and the result of the addition operation is input to the input end of the third convolutional layer.

[0041] The dense residual module includes a first dense residual submodule and a convolution submodule connected in sequence, and the upsampling module includes an upsampling submodule and a second dense residual submodule connected in sequence. The output end of the first dense residual submodule is also added to the input end of the upsampling submodule.

[0042] Optionally, the number of dense residual modules may be 4, and the number of upsampling modules may be 4.

[0043] When the number of dense residual modules and the number of upsampling modules are both multiple, a mapping relationship between the dense residual module and the upsampling module can be established. According to the mapping relationship, the output end of the first dense residual submodule in the dense residual module is added to the input end of the upsampling submodule in the corresponding upsampling module in the mapping relationship.

[0044] In the image denoising model, the resolution of the image to be processed input to the input layer is N*N, and the number of channels is 3. The convolution kernel of the first convolution layer is 3*3, and the number of channels becomes 128. The convolution kernel in the convolution submodule is also 3*3, with a span of 2. The convolution kernels of the second and third convolution layers are both 3*3. The number of channels in the second convolution layer is 128. The number of channels in the third convolution layer is 3.

[0045] Figure 3 FIG. 1 is a schematic diagram of the structure of the first dense residual submodule in the image denoising model provided by an embodiment. Figure 3As shown, more specifically, the first dense residual submodule includes: a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer and a seventh convolutional layer connected in sequence.

[0046] The input of the fourth convolutional layer is also added to the input of the seventh convolutional layer. The input of the fifth convolutional layer is fused with the input of the fourth convolutional layer. The input of the sixth convolutional layer is fused with the output of the fourth convolutional layer and the input of the fourth convolutional layer. The input of the seventh convolutional layer is fused with the input of the fourth convolutional layer, the output of the fifth convolutional layer, and the output of the fourth convolutional layer.

[0047] Optionally, the fusion operation in this embodiment refers to a channel connection operation in a dense network.

[0048] It should be noted that the structures of the second dense residual submodule and the dense residual block are the same as those of the first dense residual submodule, and will not be repeated here.

[0049] The convolution kernels of the fourth convolution layer, the fifth convolution layer, and the sixth convolution layer are all 3*3. The convolution kernel of the seventh convolution layer is 1*1. M represents the number of channels in the fourth convolution layer, the fifth convolution layer, the sixth convolution layer, and the seventh convolution layer.

[0050] The image denoising method provided in this embodiment includes: obtaining an image to be processed, inputting the image to be processed into an image denoising model, and obtaining a denoised image, wherein the image denoising model is a model formed by combining a U-shaped network, a residual network, and a dense network. The image denoising model in this embodiment can combine the characteristics of the residual network and the dense network to better utilize the deep and shallow features of the image to be processed, and can achieve the goal of retaining the details of the image to be processed as much as possible while removing the noise, thereby achieving better denoising performance while taking into account the quality of the denoised image.

[0051] Figure 4 FIG. 1 is a flowchart of another image denoising method provided by an embodiment. Figure 1 Based on the illustrated embodiment and various optional solutions, a detailed description of how to train the image denoising model is given. This embodiment only shows the steps of training the image denoising model. Figure 4 As shown, the image denoising method provided in this embodiment includes the following steps:

[0052] Step 401: Acquire a first training noise image and a corresponding first training true value image.

[0053] In one implementation, the first training noise image and the corresponding first training true value image may be acquired by actual shooting.

[0054] In another implementation, the first training noise image and the corresponding first training true value image can be obtained by actual shooting and adversarial network.

[0055] Figure 5A A flowchart of obtaining a first training noise image and a corresponding first training true value image is provided in an embodiment. Figure 5A As shown, the acquisition process includes the following steps:

[0056] Step 501: Generate a second training true value image based on a plurality of target images taken under a first preset light source brightness.

[0057] Optionally, the first preset light source brightness may be 200 lux. When shooting the target image under the first preset light source brightness, the camera module is enabled, and the denoising algorithm and bad pixel removal algorithm in the image signal processing module are all turned off. When shooting, the camera and the scene are kept absolutely still. The number of target images may be 200.

[0058] In one implementation, when generating the second training true value image based on multiple target images, an average image of the multiple target images may be determined based on the multiple target images, and the average image of the multiple target images is used as the second training true value image.

[0059] In another implementation, the multiple target images may be subjected to steps such as bad pixel removal, intensity alignment, and mean value acquisition to generate a final second training true value image. This implementation will be described in detail later.

[0060] Step 502: Use a plurality of images captured by randomly adjusting the light source brightness within the range of the second preset light source brightness and the third preset light source brightness as second training noise images.

[0061] The second preset light source brightness is smaller than the third preset light source brightness, and the third preset light source brightness is smaller than the first preset light source brightness.

[0062] In this embodiment, a plurality of images captured by randomly adjusting the light source brightness within the range of the second preset light source brightness and the third preset light source brightness are used as the second training noise images.

[0063] The second preset light source brightness may be 0.1 lux, and the third preset light source brightness may be 100 lux. The number of second training noise images may be 200. When shooting the second training noise images, the camera parameters when shooting each second training noise image may be recorded, for example, analog gain and digital gain.

[0064] It should be noted that, in this embodiment, the scene can be changed to capture multiple target images and second training noise images in different scenes. Multiple target images in the same scene are used to generate a second training true value image. For one scene, one second training true value image corresponds to multiple second training noise images.

[0065] Step 503: Train an initial adversarial network based on the second training true value image and the second training noise image to obtain an adversarial network formed by final training.

[0066] Specifically, the adversarial network in this embodiment includes a generative network and a discriminative network. Both the generative network and the discriminative network in the adversarial network can be U-type networks.

[0067] The initial generation network in the initial adversarial network is used to generate a noise image, and the initial decision network in the initial adversarial network is used to judge the similarity between the noise image output by the initial generation network and the actual second training noise image. According to the loss function of the initial generation network and the loss function of the initial decision network, the initial adversarial network is trained separately and iteratively until convergence. The specific process is as follows.

[0068] Input the normally distributed noise, the second training true value image, and the camera parameters corresponding to any second training noise image into the initial generation network in the initial adversarial network to obtain an output noise image;

[0069] Input the noise image, any one of the second training noise images, the camera parameters corresponding to the any one of the second training noise images, and the second training true value image into an initial discriminant network in the initial adversarial network to obtain an output probability of the initial discriminant network;

[0070] According to the output probability, the camera parameters corresponding to any second training noise image, the second training true value image, any second training noise image and the noise image, the loss function of the initial discriminant network and the loss function of the initial generative network are determined. When it is determined that the initial adversarial network does not converge according to the loss function of the initial discriminant network and the loss function of the initial generative network, the step of inputting the normally distributed noise, the second training true value image and the camera parameters corresponding to any second training noise image into the initial generative network in the initial adversarial network to obtain the output noise image is alternately executed, and the step of inputting the noise image, any second training noise image, the camera parameters corresponding to any second training noise image and the second training true value image into the initial discriminant network in the initial adversarial network to obtain the output probability of the initial discriminant network is returned, until it is determined that the adversarial network converges according to the loss function of the initial discriminant network and the loss function of the initial generative network, and the converged adversarial network is determined as the adversarial network formed by the final training.

[0071] The output probability of the initial discriminant network is a value between 0 and 1 to represent the probability of a true or false noise image. 0 represents the noise image generated by the generative network, and 1 represents the second training noise image actually collected.

[0072] Among them, the loss function of the generated network in the adversarial network is:

[0073] L G =E[(D(N c ,N r ,N f )) 2 ]+E[(D(N c ,N f ,N r )-1) 2 ]

[0074] The loss function of the decision network is:

[0075] L D =E[(D(N c ,N r ,N f )-1) 2 ]+E[(D(N c ,N f ,N r )) 2 ]

[0076] N c To generate the conditional signal of the noise image, including the analog gain, digital gain and the second training truth image of the camera, N f N is the noise image generated by the generative network. r is the second real noise image for training. D(*) represents the output of the discriminant network. E(*) represents the mean.

[0077] Step 504: input the pre-acquired third training true value image into the adversarial network to obtain the third training noise image output by the generative network of the adversarial network.

[0078] After the adversarial network is trained, the pre-acquired third training true value image can be input into the adversarial network to obtain the third training noise image output by the generative network of the adversarial network.

[0079] Step 505: Use the second training noise image and the third training noise image as first training images, and use the second training true value image and the third training true value image as corresponding first training true value images.

[0080] In this embodiment, the second training noise image actually collected and the third training noise image generated by the adversarial network can be used as the first training noise image of the training image denoising model, and the second training true value image and the third training true value image can be used as the corresponding first training true value image. The use of the adversarial network to generate training sample pairs greatly enriches the number of training samples. Based on these rich training samples, the image denoising model can be fully trained, thereby improving the denoising performance of the image denoising model.

[0081] Step 402: Input the first training noisy image into the initial image denoising model to obtain an output result.

[0082] Step 403: Determine a loss function according to the output result and the corresponding first training true value image.

[0083] Step 404: When the loss function is greater than a preset threshold, the network structure and network parameters of the initial image denoising model are adjusted according to the loss function, an updated image denoising model is determined, and the updated image denoising model is used as a new initial image denoising model. The process returns to the step of inputting the first training noisy image into the initial image denoising model to obtain an output result until the loss function is less than or equal to the preset threshold, and the image denoising model when the loss function is less than or equal to the preset threshold is determined as the image denoising model.

[0084] Step 402 to step 404 is a process of iteratively training the image denoising model.

[0085] Since the first training noisy images acquired in step 401 are relatively abundant, the denoising performance and generalization ability of the image denoising model can be significantly improved by training the image denoising model based on these first training noisy images.

[0086] Figure 5B A flowchart of generating a second training true value image is provided in an embodiment. Figure 5B As shown, the process of generating the second training true value image can be as follows:

[0087] Step 601: Generate an average image of multiple images based on multiple images taken in a dark environment.

[0088] The pixel value of each pixel in the average image is the average value of the pixel values ​​of the corresponding position of all images taken in the dark environment.

[0089] Step 602: For each target image, compare the first pixel value of each pixel of the target image with the second pixel value of the pixel at the corresponding position in the average image.

[0090] Step 603: If the absolute value of the difference between the first pixel value and the second pixel value is greater than a preset first pixel difference threshold, a bilinear interpolation method is used to determine an updated value of the first pixel value to form an updated target image.

[0091] The process of step 601-step 603 is a bad pixel removal process. Bad pixel pixels will affect the accuracy of the true value image estimation because they do not follow the random process of generating noise at normal pixel positions. The absolute value of the difference between the first pixel value and the second pixel value is greater than the preset first pixel difference threshold, indicating that the pixel corresponding to the first pixel value is a defective pixel. The bilinear interpolation method is used to determine the updated value of the first pixel value. After all bad pixels are corrected, an updated target image is formed. If in a target image, the absolute value of the difference between the first pixel value of all pixels and the second pixel value of the pixel at the corresponding position of the average image is less than or equal to the preset first pixel difference threshold, the target image is used as the updated target image.

[0092] Step 604: Generate a second training true value image according to the updated target image.

[0093] In one implementation, an average image of multiple updated target images may be used as the second training true value image.

[0094] In another implementation, step 604 may specifically be: for each updated target image, determining the average pixel value of all pixels in the updated target image; determining the average of the average pixel values ​​of multiple updated target images based on the average pixel value of each updated target image; deleting the updated target images whose absolute value of the difference between the corresponding average pixel value and the average of the average pixel values ​​is greater than a preset second pixel difference threshold to form a filtered updated target image; and generating a second training true value image based on the filtered updated target image.

[0095] This implementation achieves image intensity alignment. Due to the inaccuracy of the exposure algorithm, it is still necessary to consider the slight changes in the images collected under the same scene and the same brightness. The updated target image whose absolute value of the difference between the corresponding average pixel value and the average of the average pixel values ​​is greater than the preset second pixel difference threshold is deleted.

[0096] According to the average pixel value of each updated target image, the average value and variance of the average pixel values ​​of the multiple updated target images are determined, a confidence interval is obtained based on the mean value and the variance, and then the images outside the confidence interval are deleted.

[0097] More specifically, generating a second training true value image based on the filtered update target image includes: for each filtered update target image, if the pixel value at the first position in the filtered update target image is less than or equal to the preset first pixel threshold or greater than or equal to the preset second pixel threshold, then updating the pixel value at the first position to the pixel value with the largest number of occurrences among the pixel values ​​at the first position of all filtered update target images, generating a filtered secondary update target image, wherein the first pixel threshold is less than the second pixel threshold; and determining the average image of all filtered secondary update target images as the second training true value image. Exemplarily, the preset first pixel threshold may be 0, and the preset second pixel threshold may be 255.

[0098] Figure 6 FIG. 1 is a schematic diagram of a structure of a generative network in an adversarial network provided by an embodiment. Figure 6 As shown, the generated network may include: an input layer, 3 layers of first convolution modules, a first pooling layer, 3 layers of second convolution modules, a second pooling layer, 8 layers of third convolution modules, a first upsampling layer, 3 layers of fourth convolution modules, a second upsampling layer, 3 layers of fifth convolution modules and a convolution layer connected in sequence. The convolution modules each include a convolution layer, a normalization layer and an activation layer. The convolution kernels involved in the generated network are all 3*3. The resolution and number of channels of the image in each layer or each module are shown in the figure.

[0099] Figure 7 The schematic diagram of the structure of the discriminant network in the adversarial network provided by an embodiment. Figure 7 As shown in the figure, the decision network includes: an input layer, 3 layers of first convolution modules, a first pooling layer, 3 layers of second convolution modules, a second pooling layer, 3 layers of third convolution modules, a third pooling layer, 2 layers of fourth convolution modules, a fourth pooling layer, a fifth convolution module, a fully connected layer and an activation layer connected in sequence. Among them, the activation function of the activation layer can be a Sigmoid function. The convolution modules all include a convolution layer, a normalization layer and an activation layer. The convolution kernels involved in the decision network are all 3*3. The resolution and number of channels of the image in each layer or each module are shown in the figure.

[0100] In the image denoising method provided by this embodiment, the second training noise image actually collected and the third training noise image generated by the adversarial network can be used as the first training noise image of the training image denoising model, and the second training true value image and the third training true value image can be used as the corresponding first training true value image. The use of the adversarial network to generate training sample pairs greatly enriches the number of training samples. Based on these rich training samples, the image denoising model can be fully trained, thereby improving the denoising performance of the image denoising model.

[0101] Figure 8FIG. 1 is a schematic diagram of the structure of an image denoising device provided by an embodiment. Figure 8 As shown, the image denoising device provided in this embodiment includes the following modules: a first acquisition module 81 and a second acquisition module 82 .

[0102] The first acquisition module 81 is configured to acquire an image to be processed.

[0103] The second acquisition module 82 is configured to input the image to be processed into the image denoising model to obtain the denoised image.

[0104] Among them, the image denoising model is a model formed by combining a U-type network, a residual network and a dense network.

[0105] Optionally, the image denoising model includes: an input layer, a first convolutional layer, at least one dense residual module, a dense residual block, at least one upsampling module, a second convolutional layer, a third convolutional layer and an output layer connected in sequence.

[0106] The output end of the input layer is further subtracted from the output end of the third convolutional layer, and the result of the subtraction operation is input to the input end of the output layer.

[0107] The output end of the first convolutional layer is also added to the output end of the second convolutional layer, and the result of the addition operation is input to the input end of the third convolutional layer.

[0108] The dense residual module includes a first dense residual submodule and a convolution submodule connected in sequence, and the upsampling module includes an upsampling submodule and a second dense residual submodule connected in sequence. The output end of the first dense residual submodule is also added to the input end of the upsampling submodule.

[0109] Optionally, the first dense residual submodule includes: a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer and a seventh convolutional layer connected in sequence.

[0110] The input end of the fourth convolutional layer is also added to the input end of the seventh convolutional layer.

[0111] An input end of the fifth convolutional layer is fused with an input end of the fourth convolutional layer.

[0112] The input end of the sixth convolutional layer is fused with the output end of the fourth convolutional layer and the input end of the fourth convolutional layer.

[0113] The input end of the seventh convolutional layer is fused with the input end of the fourth convolutional layer, the output end of the fifth convolutional layer, and the output end of the fourth convolutional layer.

[0114] The image denoising device provided in this embodiment is used to execute the image denoising method of any of the above embodiments. The implementation principle and technical effect of the image denoising device provided in this embodiment are similar and will not be described in detail here.

[0115] Fig. 9 FIG. 1 is a schematic diagram of the structure of another image denoising device provided by an embodiment. Figure 8 Based on the embodiment shown, other modules included in the image denoising device are described in detail. Fig. 9 As shown, the image denoising device provided in this embodiment further includes the following modules: a third acquisition module 91 , a fourth acquisition module 92 , a first determination module 93 and a second determination module 94 .

[0116] The third acquisition module 91 is configured to acquire a first training noise image and a corresponding first training true value image.

[0117] The fourth acquisition module 92 is configured to input the first training noise image into the initial image denoising model to obtain an output result.

[0118] The first determination module 93 is configured to determine a loss function according to the output result and the corresponding first training true value image.

[0119] The second determination module 94 is configured to adjust the network structure and network parameters of the initial image denoising model according to the loss function when the loss function is greater than a preset threshold, determine an updated image denoising model, use the updated image denoising model as a new initial image denoising model, return to execute the step of inputting the first training noisy image into the initial image denoising model to obtain an output result, until the loss function is less than or equal to the preset threshold, and determine the image denoising model when the loss function is less than or equal to the preset threshold as the image denoising model.

[0120] The image denoising device provided in this embodiment is used to execute the image denoising method of any of the above embodiments. The implementation principle and technical effect of the image denoising device provided in this embodiment are similar and will not be described in detail here.

[0121] Fig.10 FIG. 1 is a schematic diagram of the structure of another image denoising device provided by an embodiment. Fig. 9 Based on the embodiment shown, the specific structure of the third acquisition module 91 is described in detail. Fig.10 As shown, the third acquisition module 91 includes: a generation submodule 911 , a first determination submodule 912 , a first acquisition submodule 913 , a second acquisition submodule 914 and a second determination submodule 915 .

[0122] The generating submodule 911 is configured to generate a second training true value image according to a plurality of target images taken under a first preset light source brightness.

[0123] Optionally, the generation submodule 911 is specifically used to: generate an average image of multiple images based on multiple images taken in a light-free environment; for each target image, compare the first pixel value of each pixel of the target image with the second pixel value of the pixel at the corresponding position in the average image; if the absolute value of the difference between the first pixel value and the second pixel value is greater than a preset first pixel difference threshold, use bilinear interpolation to determine the updated value of the first pixel value to form an updated target image; and generate the second training true value image based on the updated target image.

[0124] In one implementation, in terms of generating the second training true value image based on the updated target image, the generation submodule 911 is specifically used to: determine, for each updated target image, the average pixel value of all pixels in the updated target image; determine, based on the average pixel value of each updated target image, the average of the average pixel values ​​of multiple updated target images; delete the updated target images whose absolute value of the difference between the corresponding average pixel value and the average of the average pixel values ​​is greater than a preset second pixel difference threshold to form a filtered updated target image; and generate the second training true value image based on the filtered updated target image.

[0125] In one implementation, in terms of generating the second training true value image based on the filtered update target image, the generation submodule 911 is specifically used to: for each filtered update target image, if the pixel value at the first position in the filtered update target image is less than or equal to a preset first pixel threshold or greater than or equal to a preset second pixel threshold, then update the pixel value at the first position to the pixel value that appears the most times among the pixel values ​​at the first positions of all filtered update target images, to generate a filtered secondary update target image, wherein the first pixel threshold is less than the second pixel threshold; and determine the average image of all filtered secondary update target images as the second training true value image.

[0126] The first determination submodule 912 is configured to use a plurality of images captured by randomly adjusting the light source brightness within the range of the second preset light source brightness and the third preset light source brightness as second training noise images.

[0127] The brightness of the second preset light source is smaller than the brightness of the third preset light source, and the brightness of the third preset light source is smaller than the brightness of the first preset light source.

[0128] The first acquisition submodule 913 is configured to train an initial adversarial network according to the second training true value image and the second training noise image, and acquire the adversarial network formed by the final training.

[0129] Optionally, the first acquisition submodule 913 is specifically used for:

[0130] Input the normally distributed noise, the second training true value image, and the camera parameters corresponding to any second training noise image into the initial generation network in the initial adversarial network to obtain an output noise image;

[0131] Input the noise image, any one of the second training noise images, the camera parameters corresponding to any one of the second training noise images, and the second training true value image into an initial discriminant network in the initial adversarial network to obtain an output probability of the initial discriminant network;

[0132] According to the output probability, the camera parameters corresponding to any second training noise image, the second training true value image, any second training noise image and the noise image, determine the loss function of the initial discriminant network and the loss function of the initial generative network. When it is determined that the initial adversarial network does not converge according to the loss function of the initial discriminant network and the loss function of the initial generative network, alternately execute the step of returning to input the normally distributed noise, the second training true value image and the camera parameters corresponding to any second training noise image into the initial generative network in the initial adversarial network to obtain the output noise image, and return to the step of inputting the noise image, the any second training noise image, the camera parameters corresponding to any second training noise image and the second training true value image into the initial discriminant network in the initial adversarial network to obtain the output probability of the initial discriminant network, until it is determined that the adversarial network converges according to the loss function of the initial discriminant network and the loss function of the initial generative network, and the converged adversarial network is determined as the adversarial network formed by the final training.

[0133] Optionally, the generative network and the discriminative network in the adversarial network are both U-type networks.

[0134] The second acquisition submodule 914 is configured to input the pre-acquired third training true value image into the adversarial network, and acquire the third training noise image output by the generative network of the adversarial network.

[0135] The second determining submodule 915 is configured to use both the second training noise image and the third training noise image as the first training image, and use both the second training true value image and the third training true value image as the corresponding first training true value image.

[0136] The image denoising device provided in this embodiment is used to implement the image denoising method of any of the above embodiments. The implementation principle and technical effect of the image denoising device provided in this embodiment are similar and will not be repeated here.

[0137] Fig.11 FIG. 1 is a schematic diagram of the structure of an image denoising device provided by an embodiment. Fig.11 As shown, the image denoising device includes a processor 111 and a memory 112; the number of the processor 111 in the image denoising device may be one or more. Fig.11 In the figure, a processor 111 is taken as an example; the processor 111 and the memory 112 in the image denoising device can be connected by a bus or other means, Fig.11 The example of connecting through bus is taken in the following.

[0138] The memory 112 is a computer-readable storage medium that can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the image denoising method in the embodiment of the present application (for example, the first acquisition module 81 and the second acquisition module 82 in the image denoising device). The processor 111 runs the software programs, instructions and modules stored in the memory 112, thereby implementing various functional applications and data processing of the image denoising device, that is, realizing the above-mentioned image denoising method.

[0139] The memory 112 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the image denoising device, etc. In addition, the memory 112 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0140] The embodiment of the present application further provides a storage medium containing computer executable instructions, and when the computer executable instructions are executed by a computer processor, they are used to perform an image denoising method, the method comprising:

[0141] Get the image to be processed;

[0142] The image to be processed is input into an image denoising model to obtain a denoised image; wherein the image denoising model is a model formed by combining a U-type network, a residual network and a dense network.

[0143] Of course, the computer executable instructions of a storage medium containing computer executable instructions provided by the present application are not limited to the method operations described above, and can also execute related operations in the image denoising method provided by any embodiment of the present application.

[0144] The above description is merely an exemplary embodiment of the present application and is not intended to limit the protection scope of the present application.

[0145] In general, various embodiments of the present application can be implemented in hardware or dedicated circuits, software, logic or any combination thereof. For example, some aspects can be implemented in hardware, while other aspects can be implemented in firmware or software that can be executed by a controller, microprocessor or other computing device, although the present application is not limited thereto.

[0146] Embodiments of the present application may be implemented by executing computer program instructions by a data processor of a mobile device, for example in a processor entity, or by hardware, or by a combination of software and hardware. The computer program instructions may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages.

[0147] The block diagram of any logic flow in the accompanying drawings of the present application can represent program steps, or can represent interconnected logic circuits, modules and functions, or can represent a combination of program steps and logic circuits, modules and functions. The computer program can be stored on a memory. The memory can have any type suitable for the local technical environment and can be implemented using any suitable data storage technology, such as but not limited to read-only memory (ROM), random access memory (RAM), optical memory device and system (digital versatile disc DVD or CD disc), etc. Computer-readable media may include non-transient storage media. The data processor can be any type suitable for the local technical environment, such as but not limited to a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a programmable logic device (FGPA) and a processor based on a multi-core processor architecture.

[0148] By way of exemplary and non-limiting examples, a detailed description of exemplary embodiments of the present application has been provided above. However, various modifications and adjustments to the above embodiments will be apparent to those skilled in the art when considered in conjunction with the accompanying drawings and claims, but will not depart from the scope of the present invention. Therefore, the proper scope of the present invention will be determined according to the claims.

Claims

1. An image denoising method, characterized in that: The method comprises: Get the image to be processed; Input the image to be processed into an image denoising model to obtain a denoised image; wherein the image denoising model is a model formed by combining a U-type network, a residual network and a dense network, and the image denoising model includes: an input layer, a first convolutional layer, at least one dense residual module, a dense residual block, at least one upsampling module, a second convolutional layer, a third convolutional layer and an output layer connected in sequence; The output end of the input layer is further subtracted from the output end of the third convolutional layer, and the result of the subtraction operation is input to the input end of the output layer; The output end of the first convolutional layer is further added to the output end of the second convolutional layer, and the result of the addition operation is input to the input end of the third convolutional layer; The dense residual module includes a first dense residual submodule and a convolution submodule connected in sequence, and the upsampling module includes an upsampling submodule and a second dense residual submodule connected in sequence; the output end of the first dense residual submodule is also added to the input end of the upsampling submodule.

2. The method according to claim 1, characterized in that: The first dense residual submodule includes: a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer and a seventh convolutional layer connected in sequence; The input end of the fourth convolutional layer is also added to the input end of the seventh convolutional layer; An input end of the fifth convolutional layer is fused with an input end of the fourth convolutional layer; The input end of the sixth convolutional layer is fused with the output end of the fourth convolutional layer and the input end of the fourth convolutional layer; The input end of the seventh convolutional layer is fused with the input end of the fourth convolutional layer, the output end of the fifth convolutional layer, and the output end of the fourth convolutional layer.

3. The method according to claim 1 or 2, characterized in that: Before inputting the image to be processed into the image denoising model to obtain the denoised image, the method further includes: Acquire a first training noise image and a corresponding first training true value image; Inputting the first training noise image into an initial image denoising model to obtain an output result; Determine a loss function according to the output result and the corresponding first training true value image; When the loss function is greater than a preset threshold, the network structure and network parameters of the initial image denoising model are adjusted according to the loss function, an updated image denoising model is determined, the updated image denoising model is used as a new initial image denoising model, and the process returns to the step of inputting the first training noisy image into the initial image denoising model to obtain an output result until the loss function is less than or equal to the preset threshold, and the image denoising model when the loss function is less than or equal to the preset threshold is determined as the image denoising model.

4. The method according to claim 3, characterized in that The step of acquiring a first training noise image and a corresponding first training true value image includes: Generate a second training true value image according to a plurality of target images taken under a first preset light source brightness; Using a plurality of images captured by randomly adjusting the light source brightness within a range between a second preset light source brightness and a third preset light source brightness as second training noise images; wherein the second preset light source brightness is less than the third preset light source brightness, and the third preset light source brightness is less than the first preset light source brightness; Training an initial adversarial network according to the second training true value image and the second training noise image to obtain an adversarial network formed by final training; Inputting a third training true value image acquired in advance into the adversarial network, and acquiring a third training noise image output by a generative network of the adversarial network; The second training noise image and the third training noise image are both used as the first training noise image, and the second training true value image and the third training true value image are both used as the corresponding first training true value image.

5. The method according to claim 4, characterized in that The step of generating a second training true value image according to a plurality of target images captured under a first preset light source brightness comprises: generating an average image of the plurality of images according to the plurality of images captured in a light-free environment; For each target image, comparing a first pixel value of each pixel point of the target image with a second pixel value of a pixel point at a corresponding position in the average image; If the absolute value of the difference between the first pixel value and the second pixel value is greater than a preset first pixel difference threshold, a bilinear interpolation method is used to determine an updated value of the first pixel value to form an updated target image; The second training true value image is generated according to the updated target image.

6. The method according to claim 5, characterized in that The step of generating a second training true value image according to the updated target image comprises: For each updated target image, determining an average pixel value of all pixels in the updated target image; Determining an average of the average pixel values ​​of the plurality of updated target images according to the average pixel value of each updated target image; Deleting the updated target image whose absolute value of the difference between the corresponding average pixel value and the average value of the average pixel value is greater than a preset second pixel difference threshold value to form a filtered updated target image; The second training true value image is generated according to the filtered updated target image.

7. The method according to claim 6, characterized in that The step of generating the second training true value image according to the filtered updated target image comprises: For each filtered update target image, if the pixel value at the first position in the filtered update target image is less than or equal to a preset first pixel threshold or greater than or equal to a preset second pixel threshold, the pixel value at the first position is updated to the pixel value with the largest number of occurrences among the pixel values ​​at the first positions of all filtered update target images, to generate a filtered secondary update target image; wherein the first pixel threshold is less than the second pixel threshold; The average image of all filtered secondary updated target images is determined as the second training true value image.

8. The method according to claim 4, characterized in that Training an initial adversarial network according to the second training true value image and the second training noise image to obtain an adversarial network formed by final training includes: Input the normally distributed noise, the second training true value image, and the camera parameters corresponding to any second training noise image into the initial generation network in the initial adversarial network to obtain an output noise image; Input the noise image, any one of the second training noise images, the camera parameters corresponding to any one of the second training noise images, and the second training true value image into an initial discriminant network in the initial adversarial network to obtain an output probability of the initial discriminant network; According to the output probability, the camera parameters corresponding to any second training noise image, the second training true value image, any second training noise image and the noise image, determine the loss function of the initial discriminant network and the loss function of the initial generative network. When it is determined that the initial adversarial network does not converge according to the loss function of the initial discriminant network and the loss function of the initial generative network, alternately execute the step of returning to input the normally distributed noise, the second training true value image and the camera parameters corresponding to any second training noise image into the initial generative network in the initial adversarial network to obtain the output noise image, and return to the step of inputting the noise image, the any second training noise image, the camera parameters corresponding to any second training noise image and the second training true value image into the initial discriminant network in the initial adversarial network to obtain the output probability of the initial discriminant network, until it is determined that the adversarial network converges according to the loss function of the initial discriminant network and the loss function of the initial generative network, and the converged adversarial network is determined as the adversarial network formed by the final training.

9. The method according to any one of claims 4 to 8, characterized in that: The generative network and the discriminative network in the adversarial network are both U-type networks.

10. An image denoising device, characterized in that: The device comprises: A first acquisition module is configured to acquire an image to be processed; A second acquisition module is configured to input the image to be processed into an image denoising model to obtain a denoised image; wherein the image denoising model is a model formed by combining a U-type network, a residual network and a dense network, and the image denoising model includes: an input layer, a first convolutional layer, at least one dense residual module, a dense residual block, at least one upsampling module, a second convolutional layer, a third convolutional layer and an output layer connected in sequence; The output end of the input layer is further subtracted from the output end of the third convolutional layer, and the result of the subtraction operation is input to the input end of the output layer; The output end of the first convolutional layer is further added to the output end of the second convolutional layer, and the result of the addition operation is input to the input end of the third convolutional layer; The dense residual module includes a first dense residual submodule and a convolution submodule connected in sequence, and the upsampling module includes an upsampling submodule and a second dense residual submodule connected in sequence; the output end of the first dense residual submodule is also added to the input end of the upsampling submodule.

11. An image denoising device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the image denoising method as described in any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image denoising method as described in any one of claims 1 to 9 is implemented.

Images