Image Denoising Method, Device, Electronic Device and Computer Storage Medium

Abstract

The present application discloses an image denoising method, apparatus, electronic device, and computer storage medium, which relate to the technical field of image processing. The method includes: creating a bridged U-shaped denoising network model, constructing a first sample pair dataset and a second sample pair dataset based on an initial sample set, generating a first predicted image and a second predicted image respectively based on the bridged U-shaped denoising network, updating network parameters based on the first predicted image and the second predicted image, obtaining an output image sample set as the initial noise image sample set for the next round, determining the number of iteration rounds, using the first predicted image and the second predicted image of the current round for model training until the number of training rounds reaches the number of iteration rounds to obtain a target denoising network model, acquiring the current noisy image, and inputting the current noisy image into the target denoising network model to obtain a denoised image.

Description

Technical Field

[0001] This application belongs to the technical field of image processing, and more specifically, relates to an image denoising method, apparatus, electronic device, and computer storage medium. Background Art

[0002] Surface defects and internal defects of components are potential hazards for the safe and stable operation of mechanical equipment. The quality assessment of component products depends on accurate defect detection technologies. Common non-destructive testing methods for component defects include ultrasonic testing and X-ray testing. These methods can visually reflect defects in the form of images. However, the acquired images are often interfered by noise. Denoising the images to obtain clean images is an important guarantee for subsequent defect detection and product quality assessment. Especially in the trend of intelligent non-destructive testing, image denoising is crucial for high-precision automatic defect recognition.

[0003] In practice, obtaining a large number of clean images usually requires a great cost, and even in some special cases, clean images cannot be obtained. To address the problem of difficult acquisition of clean images, self-supervised denoising methods have been proposed, such as the IDR (Self-Supervised Image Denoising via Iterative Data Refinement) method and the Noise2Void (Learning Denoising from Single Noisy Images) method. The above methods only require a single noisy image in the same scenario. The IDR method adds noise to the noisy data to construct a noisier-noise paired dataset, and through iterative training, the dataset is refined while the model is updated. However, the IDR method does not fully utilize the correlation between images; the Noise2Void method uses a blind spot network or sets blind spots in the image and trains with noisy images to enable the model to learn the correlation between pixels, and infers the true value of the blind spot based on the pixel information in the receptive field of the blind spot. Although the Noise2Void method fully utilizes the correlation between pixels, it hides the pixel information of the blind spot by setting blind spots; therefore, the above methods all have their limitations, which restricts the denoising ability. Summary of the Invention

[0004] In view of this, the present invention provides an image denoising method, apparatus, electronic device, and computer storage medium, mainly aiming to solve the problem of limited denoising ability caused by using the IDR method and the Noise2Void method in the current image denoising process.

[0005] According to the first aspect of the present application, an image denoising method is provided, including:

[0006] Create a bridging U-shaped denoising network model, and construct a first sample pair dataset and a second sample pair dataset based on the initial noise image sample set. Each first sample pair in the first sample pair dataset is obtained by pairing an initial noise image with a pseudo-blind spot noise image, and each second sample pair in the second sample pair dataset is obtained by pairing the initial noise image with a global noise image;

[0007] Denoise each pseudo-blind spot noise image in the first sample pair dataset and each global noise image in the second sample pair dataset respectively based on the bridging U-shaped denoising network to generate a first predicted image and a second predicted image, and train the bridging U-shaped denoising network model based on the first predicted image and the second predicted image, update the network parameters, and obtain the trained denoising network model;

[0008] Input the initial noise image sample set into the trained denoising network model to obtain an output image sample set as the initial noise image sample set for the next round;

[0009] Determine the number of iteration rounds. In each round of training, use the output image sample set obtained in the previous round to construct the first sample pair dataset and the second sample pair dataset for the current round. Use the denoising network model obtained in the previous round as the initialization model for the current round, denoise each pseudo-blind spot noise image in the first sample pair dataset for the current round and each global noise image in the second sample pair dataset for the current round to generate the first predicted image and the second predicted image for the current round, and perform model training based on the first predicted image and the second predicted image for the current round until the number of training rounds reaches the iteration rounds to obtain the target denoising network model;

[0010] Obtain the current noise picture and input the current noise picture into the target denoising network model to obtain the denoised picture.

[0011] Optionally, the constructing the first sample pair dataset and the second sample pair dataset based on the initial noise image sample set includes:

[0012] Obtain the initial noise image dataset, where the initial noise image dataset includes multiple initial noise images;

[0013] For each initial noise image in the initial noise image dataset, perform a noise masking operation to obtain the pseudo-blind spot noise image, and construct the first sample pair based on the initial noise image and the pseudo-blind spot noise image;

[0014] Add global noise to the initial noise image to generate the global noise image, and construct the second sample pair based on the initial noise image and the global noise image;

[0015] Determine the first sample pair dataset based on multiple first sample pairs, and determine the second sample pair dataset based on multiple second sample pairs.

[0016] Optionally, for each initial noise image in the initial noise image dataset, obtaining the pseudo-blind spot noise image by using a noise mask operation includes:

[0017] Determine multiple pixel positions in each of the initial noise images;

[0018] Create a filter mask with a preset size at each pixel position, move the filter mask along a preset azimuth at a preset step length, and add the pixel indicated by the pixel position to the corresponding mask pixel to obtain the pseudo-blind spot noise map.

[0019] Optionally, the bridged U-shaped denoising network model is obtained by bridging two identical U-shaped networks based on residual dense units.

[0020] Optionally, denoising each pseudo-blind spot noise image in the first sample pair dataset and each global noise image in the second sample pair dataset respectively based on the bridged U-shaped denoising network to generate a first predicted image and a second predicted image includes:

[0021] Perform the following operations on each pseudo-blind spot noise image in the first sample pair dataset: input the pseudo-blind spot noise image into the bridged U-shaped denoising network to obtain a denoised image, and perform a blind spot sampling operation on the denoised image to obtain the first predicted image;

[0022] Perform the following operations on each global noise image in the second sample pair dataset: input the global noise image into the bridged U-shaped denoising network to obtain the second predicted image.

[0023] Optionally, the blind spot sampling operation on the denoised image to obtain the first predicted image includes:

[0024] Determine multiple noise pseudo-blind spot positions in the denoised image;

[0025] Perform identity sampling on the pixel points indicated by each noise pseudo-blind spot position among the multiple noise pseudo-blind spot positions based on a sampling function;

[0026] Stitch the sampled pixel points into a target denoised picture, and use the stitched target denoised picture as the first predicted image.

[0027] Optionally, training the bridging U-shaped denoising network based on the first predicted image and the second predicted image, and updating network parameters to obtain a trained denoising network model includes:

[0028] Calculating a first mean squared error loss based on the initial noise image and the corresponding first predicted image, and calculating a second mean squared error loss based on the initial noise image and the corresponding second predicted image, determining the product of a balance coefficient and the first mean squared error loss, and calculating the sum of the product and the second mean squared error loss to obtain a first result;

[0029] Calculating the sum of multiple first results to obtain a total mean squared error loss, determining to minimize the total mean squared error loss, and using the minimized total mean squared error loss to update the network parameters of the bridging U-shaped denoising network to obtain the trained denoising network model as the initialization model for the next round.

[0030] According to a second aspect of the present application, there is provided an image denoising device, including:

[0031] A construction module, configured to create a bridging U-shaped denoising network model, and construct a first sample pair data set and a second sample pair data set based on an initial noise image sample set, where each first sample pair in the first sample pair data set is obtained by pairing an initial noise image and a pseudo-blind spot noise image, and each second sample pair in the second sample pair data set is obtained by pairing the initial noise image and a global noise image;

[0032] An initial training module, configured to denoise each pseudo-blind spot noise image in the first sample pair data set and each global noise image in the second sample pair data set based on the bridging U-shaped denoising network, generate a first predicted image and a second predicted image, and train the bridging U-shaped denoising network model based on the first predicted image and the second predicted image, and update network parameters to obtain a trained denoising network model;

[0033] A data refinement module, configured to input the initial noise image sample set into the trained denoising network model to obtain an output image sample set as the initial noise image sample set for the next round;

[0034] The iterative training module is used to, in each round of training, utilize the output image sample set obtained in the previous round to construct the first sample pair dataset and the second sample pair dataset of the current round. Using the denoising network model obtained in the previous round as the initialization model of the current round, it denoises each pseudo-blind spot noise image in the first sample pair dataset of the current round and each global noise image in the second sample pair dataset of the current round, generates the first prediction image and the second prediction image of the current round, and performs model training based on the first prediction image and the second prediction image of the current round until the number of training rounds reaches the iterative round to obtain the target denoising network model;

[0035] The prediction module is used to obtain the current noise picture and input the current noise picture into the target denoising network model to obtain the denoised picture.

[0036] Optionally, the construction module is further used to obtain the initial noise image dataset, and the initial noise image dataset includes multiple initial noise images; for each initial noise image in the initial noise image dataset, a pseudo-blind spot noise image is obtained by performing a noise masking operation, and the first sample pair is constructed based on the initial noise image and the pseudo-blind spot noise image; global noise is added to the initial noise image to generate the global noise image, and the second sample pair is constructed based on the initial noise image and the global noise image; the first sample pair dataset is determined based on multiple first sample pairs, and the second sample pair dataset is determined based on multiple second sample pairs.

[0037] Optionally, the construction module is further used to determine multiple pixel positions in each initial noise image; create a filter mask with a preset size at each pixel position, move the filter mask along a preset azimuth at a preset step length, and add the pixel indicated by the pixel position to the corresponding mask pixel to obtain the pseudo-blind spot noise map.

[0038] Optionally, the bridged U-shaped denoising network model is obtained by bridging two identical U-shaped networks based on residual dense units.

[0039] Optionally, the initial training module is further used to perform the following operations on each pseudo-blind spot noise image in the first sample pair dataset: input the pseudo-blind spot noise image into the bridged U-shaped denoising network to obtain a denoised image, and perform a blind spot sampling operation on the denoised image to obtain the first prediction image; perform the following operations on each global noise image in the second sample pair dataset: input the global noise image into the bridged U-shaped denoising network to obtain the second prediction image.

[0040] Optionally, the initial training module is further configured to determine multiple noise pseudo-blind spot positions in the denoised image; perform identity sampling on the pixel points indicated by each noise pseudo-blind spot position among the multiple noise pseudo-blind spot positions based on a sampling function; splice the sampled pixel points into a target denoised picture, and use the spliced target denoised picture as the first predicted image.

[0041] Optionally, the initial training module is further configured to calculate a first mean square error loss based on the initial noise image and the corresponding first predicted image, and calculate a second mean square error loss based on the initial noise image and the corresponding second predicted image, determine the product of the balance coefficient and the first mean square error loss, calculate the sum of the product and the second mean square error loss to obtain a first result; calculate the sum of multiple first results to obtain a total mean square error loss, determine to minimize the total mean square error loss, and update the network parameters of the bridge U-shaped denoising network using the minimized total mean square error loss to obtain the trained denoising network model as the initialization model for the next round.

[0042] According to a third aspect of the present application, there is provided an electronic device, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, where when the processor executes the computer program, the steps of the method according to any one of the first aspects are implemented.

[0043] According to a fourth aspect of the present application, there is provided a computer storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method according to any one of the first aspects are implemented.

[0044] With the above technical solution, the present application provides an image denoising method. First, a bridged U-shaped denoising network model is created, and based on an initial noise image sample set, a first sample pair data set and a second sample pair data set are constructed. Subsequently, based on the bridged U-shaped denoising network, denoising is performed on each pseudo-blind spot noise image in the first sample pair data set and each global noise image in the second sample pair data set to generate a first predicted image and a second predicted image. And based on the first predicted image and the second predicted image, the network parameters are updated to obtain a trained denoising network model, and an output image sample set is obtained as the initial noise image sample set for the next round. Then, the number of iteration rounds is determined, and based on the denoising network model obtained in the previous round, the first predicted image and the second predicted image of the current round are generated, and model training is performed using the first predicted image and the second predicted image of the current round until the number of training rounds reaches the number of iteration rounds to obtain a target denoising network model. Finally, the current noise picture is obtained and input into the target denoising network model to obtain a denoised picture. This method only needs to use the noise picture to train the model, encourages the model to learn the correlation between pixels, and makes full use of the blind spot information, overcoming the problems in the prior art that the IDR method does not fully utilize the correlation between pixels, and the Noise2Void method hides the pixel information of the blind spot, resulting in limited denoising ability.

[0045] The above description is only an overview of the technical solution of the present invention. In order to be able to understand the technical means of the present invention more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features and advantages of the present invention more obvious and understandable, the following specifically gives the specific implementation manners of the present invention. Brief Description of the Drawings

[0046] By reading the detailed description of the preferred embodiments below, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention. And throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0047] Figure 1 Shows a flowchart of an image denoising method provided by an embodiment of the present application;

[0048] Figure 2 Shows an iterative flowchart of model iterative training of an image denoising method provided by an embodiment of the present application;

[0049] Figure 3 Shows a framework diagram of model iterative training of an image denoising method provided by an embodiment of the present application;

[0050] Figure 4Shows a framework diagram of the noise mask operation / blind spot sampling operation of an image denoising method provided by an embodiment of the present application;

[0051] Figure 5 Shows an architecture diagram of a denoising network model of an image denoising method provided by an embodiment of the present application;

[0052] Figure 6 Shows a schematic structural diagram of an RDB module used in a denoising network of an image denoising method provided by an embodiment of the present application;

[0053] Figure 7 Shows a schematic structural diagram of an image denoising device provided by an embodiment of the present application;

[0054] Figure 8 Shows a schematic structural diagram of a device of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0055] Reference is made herein to the various aspects and features of the present application with reference to the accompanying drawings.

[0056] It should be understood that various modifications can be made to the embodiments applied herein. Therefore, the above description should not be construed as limiting, but merely as an example of the embodiments. Those skilled in the art will think of other modifications within the scope and spirit of the present application.

[0057] The accompanying drawings, which are included in and constitute a part of this specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.

[0058] These and other features of the present application will become apparent from the following description of the preferred forms of the embodiments given as non - limiting examples with reference to the accompanying drawings.

[0059] It should also be understood that, although the present application has been described with reference to some specific examples, those skilled in the art can surely implement many other equivalent forms of the present application.

[0060] When combined with the accompanying drawings, the above and other aspects, features and advantages of the present application will become more apparent in view of the following detailed description.

[0061] Specific embodiments of the present application will be described hereinafter with reference to the accompanying drawings; however, it should be understood that the claimed embodiments are merely examples of the present application, which can be implemented in various ways. Well-known and / or repetitive functions and structures are not described in detail to avoid obscuring the present application with unnecessary or redundant details. Therefore, the specific structural and functional details claimed herein are not intended to be limiting, but are merely a basis and representative basis for the claims to teach those skilled in the art to use the present application in a substantially appropriate detailed structure in various ways.

[0062] This specification may use the phrases "in one embodiment", "in another embodiment", "in yet another embodiment", or "in other embodiments", each of which may refer to one or more of the same or different embodiments according to the present application.

[0063] Embodiment 1

[0064] The embodiment of the present application provides an image denoising method, as Figure 1 shown, including:

[0065] 101. Create a bridged U-shaped denoising network model, and based on the initial noise image sample set, construct a first sample pair data set and a second sample pair data set.

[0066] In the embodiment of the present application, each first sample pair in the first sample pair data set is obtained by pairing an initial noise image and a pseudo-blind spot noise image, and each second sample pair in the second sample pair data set is obtained by pairing the initial noise image and a global noise image.

[0067] 102. Denoise each pseudo-blind spot noise image in the first sample pair data set and each global noise image in the second sample pair data set respectively based on the bridged U-shaped denoising network to generate a first predicted image and a second predicted image, and train the bridged U-shaped denoising network model based on the first predicted image and the second predicted image to update the network parameters and obtain a trained denoising network model.

[0068] 103. Input the initial noise image sample set into the trained denoising network model to obtain an output image sample set as the initial noise image sample set for the next round.

[0069] 104. Determine the number of iterative rounds. In each round of training, use the output image sample set obtained in the previous round to construct the first sample pair data set and the second sample pair data set of the current round. Use the denoising network model obtained in the previous round as the initialization model of the current round, and denoise each pseudo-blind spot noise image in the first sample pair data set of the current round and each global noise image in the second sample pair data set of the current round to generate the first predicted image and the second predicted image of the current round, and perform model training based on the first predicted image and the second predicted image of the current round until the number of training rounds reaches the iterative round to obtain the target denoising network model.

[0070] 105. Obtain the current noise picture and input the current noise picture into the target denoising network model to obtain a denoised picture.

[0071] The method provided in the embodiment of the present application first creates a bridged U-shaped denoising network model and constructs a first sample pair data set and a second sample pair data set based on the initial noise image sample set. Subsequently, denoise each pseudo-blind spot noise image in the first sample pair data set and each global noise image in the second sample pair data set based on the bridged U-shaped denoising network to generate a first predicted image and a second predicted image, and update the network parameters based on the first predicted image and the second predicted image to obtain a trained denoising network model, and obtain an output image sample set as the initial noise image sample set for the next round. Then determine the number of iterative rounds, and generate the first predicted image and the second predicted image of the current round based on the denoising network model obtained in the previous round, and perform model training using the first predicted image and the second predicted image of the current round until the number of training rounds reaches the iterative round to obtain the target denoising network model. Finally, obtain the current noise picture and input the current noise picture into the target denoising network model to obtain a denoised picture. This method only needs to train the model using noise pictures, encourages the model to learn the correlation between pixels, and makes full use of the blind spot information, overcoming the problems that the IDR method used in the prior art does not fully utilize the correlation between pixels, and the Noise2Void method hides the pixel information of the blind spot, resulting in limited denoising ability.

[0072] Furthermore, an image denoising method proposed in the present application includes a model training stage and a model prediction stage.

[0073] I. Model training stage

[0074] The model training is carried out in an iterative manner, and the data set is refined simultaneously during the iteration, such as Figure 2As shown, given the number of iterations, in each round of training, the images in the original noise dataset are denoised using the model trained in the previous round to obtain a refined dataset, and then the refined dataset is used to train the model. Repeat this process until the iteration terminates. The specific steps are as follows:

[0075] 201. Set the number of iteration rounds n.

[0076] 202. Conduct the first round of training. Given the initial noise dataset Train to obtain the first F1 of the model.

[0077] 203. Conduct the second round of training. Before training, use the model F1 of the previous round to denoise the initial noise dataset to obtain the second-round refined dataset At the same time, initialize the model parameters with the model F1 of the previous round. Use the dataset Train the model to obtain F2.

[0078] 204. Repeat step 203 to conduct the next round of training. When conducting the m-th round of training, use the model F m-1 To denoise the initial noise dataset to obtain the refined dataset required for the m-th round At the same time, use the model F m-1 Initialize the model parameters. Use the dataset F m-1 Train the model to obtain F m .

[0079] 205. When m = n, the iteration terminates, and the target denoising network model F n is obtained.

[0080] In the above iterative training, for each round of training, the method framework used is as Figure 3 shown, and the specific steps can be described as:

[0081] 301. Given the component defect image dataset Determine the sample batch size. For each input image x in a batch of samples i , generate a pseudo-blind spot noise map M(x i ) through the noise mask M(·), and construct the sample pair (M(x i ), x i ).

[0082] 302. For the same input image x i , add global noise to generate the global noise image x i +n, and construct the sample pair (x i +n, x i ).

[0083] 303. For the sample pair (M(x i ), xi ) and input M(x i ) into the denoising network F(·) to obtain the output F(M(x i )), and then generate the predicted image m(F(M(x i ))) through blind spot sampling m(·). The predicted image and the original noisy image x i are used to calculate the mean square error pixel by pixel to obtain the mean square error loss

[0084] 304. For the sample pair (x i +n, x i ), input x i +n into the denoising network F(·) to obtain the output F(x i +n) as the predicted image, and calculate the mean square error pixel by pixel with the original noisy image x i to obtain the mean square error loss

[0085] 305. For the input image x i , the loss is the sum of the two losses, that is where β is the balance coefficient. The model parameters are updated by minimizing the total loss of each batch of samples

[0086] Among them, the noise mask M(·) operation in step 301 and the blind spot sampling m(·) operation in step 303 are as Figure 4 shown, and the implementation details of the two are as follows

[0087] Noise mask M(·) operation

[0088] Select 2×2 pixels as the size of the filter mask, and the 4 pixel positions are respectively marked as a, b, c, and d; based on this, create 4 filter masks, namely M a , M b , M c and M d ; the pixel values at the a, b, c, and d positions in the masks M a , M b , M c and M d are noises, which are called noise pixel points, and the pixel values at the remaining positions are 0; each mask slides up and down, left and right along the input picture, and the sliding step is 2; during the sliding process, the pixels in the picture and the mask pixels are added correspondingly, and finally 4 pseudo-blind spot noise maps are obtained; in each pseudo-blind spot noise map, the pixel points added to the mask noise pixel points are called noise pseudo-blind spots

[0089] Blind spot sampling m(·) operation

[0090] For each input image, after the corresponding 4 pseudo-blind spot noise maps are input into the denoising network F(·), 4 denoised images with the same size as the input image are output. The blind spot sampling function m(·) performs identity sampling on the pixel points at the positions of the noise pseudo-blind spots in the 4 output images, and stitches together all the sampled pixel points to form a denoised image. The position coordinates of each pixel point in the image are the same as its original position coordinates.

[0091] II. Model Prediction Phase

[0092] In the model prediction phase, when a noisy image is input into the target denoising network model, the corresponding denoised image can be output.

[0093] Furthermore, the denoising network model adopted by the method of this application is a newly designed deep denoising network. This denoising network is bridged by two U-shaped networks based on residual dense units, RDB-Unet, and the network structure is as Figure 5 shown. The specific structure of the network is as follows:

[0094] The network is bridged by two RDB-Unets (RDB-Unet1 and RDB-Unet2) with the same structure. In each RDB-Unet, the encoding process includes two downsamplings, and the decoding process includes two upsamplings. Downsampling and upsampling are inverse operations. Compared with the original U-shaped network Unet, each RDB-Unet has two fewer downsamplings and upsamplings. At the same time, the modules before and after the sampling layer of the original Unet are convolutional modules, while in RDB-Unet, they are residual dense RDB modules. The encoding and decoding structures are symmetric. The first layer of encoding contains 3 RDB modules, the second layer contains 2 RDB modules, and the third layer contains 1 RDB module, which also belongs to the first layer of decoding. The outputs of the first and second layers of RDB-Unet1 encoding are respectively skip-connected to the outputs of the first and second layers of RDB-Unet2 encoding. The outputs of the first and second layers of RDB-Unet1 decoding are respectively concatenated with the inputs of the second and third layers of RDB-Unet2 encoding. A large number of concatenations and skip connections are adopted in the whole network, making the network easy to train and having a relatively fast convergence speed.

[0095] The structure of the RDB module adopted in the network is as Figure 6As shown in the figure, the RDB module, whose full name is Residual Dense Block, is formed by connecting multiple convolutional layers in series. The last layer is a 1×1 convolution layer, and the layer before it is a Concat (used for splicing input data) splicing layer. Except for the last convolutional layer, a ReLU (Rectified Linear Unit) activation function follows each convolutional layer. In addition, the input of each convolutional layer is the splicing of the output results of all previous layers. Therefore, the RDB module belongs to the dense convolutional module. At the same time, a skip connection is made between the input and output of the RDB module. Therefore, the RDB module also has the advantages of the residual module.

[0096] Embodiment 2

[0097] An embodiment of the present application provides an image denoising device, as Figure 7 shown, including: a construction module 701, an initial training module 702, a data refinement module 703, an iterative training module 704, and a prediction module 705.

[0098] The construction module 701 is used to create a bridged U-shaped denoising network model, and based on the initial noise image sample set, construct a first sample pair data set and a second sample pair data set. Each first sample pair in the first sample pair data set is obtained by pairing an initial noise image and a pseudo-blind spot noise image, and each second sample pair in the second sample pair data set is obtained by pairing the initial noise image and a global noise image.

[0099] The initial training module 702 is used to denoise each pseudo-blind spot noise image in the first sample pair data set and each global noise image in the second sample pair data set based on the bridged U-shaped denoising network, generate a first prediction image and a second prediction image, and train the bridged U-shaped denoising network model based on the first prediction image and the second prediction image, update the network parameters, and obtain a trained denoising network model.

[0100] The data refinement module 703 is used to input the initial noise image sample set into the trained denoising network model to obtain an output image sample set as the initial noise image sample set for the next round.

[0101] The iterative training module 704 is used to determine the number of iterative rounds. In each round of training, the output image sample set obtained in the previous round is used to construct the first sample pair data set and the second sample pair data set for the current round. The denoising network model obtained in the previous round is used as the initialization model for the current round. Each pseudo-blind spot noise image in the first sample pair data set for the current round and each global noise image in the second sample pair data set for the current round are denoised to generate the first prediction image and the second prediction image for the current round, and model training is performed based on the first prediction image and the second prediction image for the current round until the number of training rounds reaches the number of iterative rounds to obtain the target denoising network model.

[0102] The prediction module 705 is configured to obtain a current noisy image and input the current noisy image into the target denoising network model to obtain a denoised image.

[0103] In a specific application scenario, the construction module 701 is further configured to obtain an initial noisy image dataset, where the initial noisy image dataset includes a plurality of initial noisy images; for each initial noisy image in the initial noisy image dataset, perform a noise mask operation to obtain a pseudo-blind spot noisy image, and construct a first sample pair based on the initial noisy image and the pseudo-blind spot noisy image; add global noise to the initial noisy image to generate a global noisy image, and construct a second sample pair based on the initial noisy image and the global noisy image; determine a first sample pair dataset based on a plurality of first sample pairs, and determine a second sample pair dataset based on a plurality of second sample pairs.

[0104] In a specific application scenario, the construction module 701 is further configured to determine a plurality of pixel positions in each initial noisy image; create a filter mask with a preset size at each pixel position, move the filter mask along a preset direction at a preset step size, and add the pixel indicated by the pixel position to the corresponding masked pixel to obtain a pseudo-blind spot noisy map.

[0105] In a specific application scenario, the bridged U-shaped denoising network model is obtained by bridging two identical U-shaped networks based on residual dense units.

[0106] In a specific application scenario, the initial training module 702 is further configured to perform the following operations on each pseudo-blind spot noisy image in the first sample pair dataset: input the pseudo-blind spot noisy image into the bridged U-shaped denoising network to obtain a denoised image, and perform a blind spot sampling operation on the denoised image to obtain a first prediction image; perform the following operations on each global noisy image in the second sample pair dataset: input the global noisy image into the bridged U-shaped denoising network to obtain a second prediction image.

[0107] In a specific application scenario, the initial training module 702 is further configured to determine a plurality of noise pseudo-blind spot positions in the denoised image; perform identity sampling on each pixel point indicated by the plurality of noise pseudo-blind spot positions based on a sampling function; splice the sampled pixel points into a target denoised image, and use the spliced target denoised image as the first prediction image.

[0108] In a specific application scenario, the initial training module 702 is further configured to calculate a first mean squared error loss based on the initial noise image and the corresponding first predicted image, and calculate a second mean squared error loss based on the initial noise image and the corresponding second predicted image, determine the product of the balance coefficient and the first mean squared error loss, calculate the sum of the product and the second mean squared error loss to obtain a first result; calculate the sum of multiple first results to obtain a total mean squared error loss, determine to minimize the total mean squared error loss, and update the network parameters of the bridged U-shaped denoising network by minimizing the total mean squared error loss to obtain a trained denoising network model as the initialization model for the next round.

[0109] The device provided in the embodiment of the present application first creates a bridged U-shaped denoising network model through a construction module, and constructs a first sample pair data set and a second sample pair data set based on an initial noise image sample set. Subsequently, the initial training module denoises each pseudo-blind spot noise image in the first sample pair data set and each global noise image in the second sample pair data set based on the bridged U-shaped denoising network to generate a first predicted image and a second predicted image, and updates the network parameters based on the first predicted image and the second predicted image to obtain a trained denoising network model. And obtain an output image sample set as the initial noise image sample set for the next round through a data refinement module. Then, the iterative training module determines the number of iterative rounds, and generates the first predicted image and the second predicted image of the current round based on the denoising network model obtained in the previous round, and uses the first predicted image and the second predicted image of the current round for model training until the number of training rounds reaches the iterative rounds to obtain a target denoising network model. Finally, the prediction module obtains the current noise picture and inputs the current noise picture into the target denoising network model to obtain a denoised picture. This method only needs to train the model using noise pictures, encourages the model to learn the correlation between pixels, and makes full use of the blind spot information, overcoming the problems that the IDR method used in the prior art does not fully utilize the correlation between pixels, and the Noise2Void method hides the pixel information of the blind spots, resulting in limited denoising ability.

[0110] Embodiment 3

[0111] The embodiment of the present application also provides an electronic device, as Figure 8 shown. The electronic device includes a bus, a processor, a memory, and a communication interface, and may further include an input / output interface and a display device. Among them, each functional unit can complete mutual communication through the bus. The memory stores a computer program, and the processor is configured to execute the program stored on the memory to execute the image denoising method in the above embodiment.

[0112] Embodiment 4

[0113] The embodiment of the present application also provides a storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the image denoising method are implemented.

[0114] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by hardware or by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, and the software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various implementation scenarios of the present application.

[0115] The above embodiments are only exemplary embodiments of the present application and are not used to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art can make various modifications or equivalent replacements within the essence and protection scope of the present application, and such modifications or equivalent replacements should also be regarded as falling within the protection scope of the present application.

Claims

1. An image denoising method, characterized in that, Including: Create a bridging U-shaped denoising network model, and construct a first sample pair dataset and a second sample pair dataset based on an initial noise image sample set, where each first sample pair in the first sample pair dataset is obtained by pairing an initial noise image and a pseudo-blind spot noise image, and each second sample pair in the second sample pair dataset is obtained by pairing the initial noise image and a global noise image; Denoise each pseudo-blind spot noise image in the first sample pair dataset and each global noise image in the second sample pair dataset based on the bridging U-shaped denoising network, generate a first predicted image and a second predicted image, and train the bridging U-shaped denoising network model based on the first predicted image and the second predicted image, update the network parameters, and obtain a trained denoising network model; Input the initial noise image sample set into the trained denoising network model to obtain an output image sample set as the initial noise image sample set for the next round; Determine the number of iteration rounds. In each round of training, use the output image sample set obtained in the previous round to construct the first sample pair dataset and the second sample pair dataset for the current round, use the denoising network model obtained in the previous round as the initialization model for the current round, denoise each pseudo-blind spot noise image in the first sample pair dataset for the current round and each global noise image in the second sample pair dataset for the current round, generate the first predicted image and the second predicted image for the current round, and perform model training based on the first predicted image and the second predicted image for the current round until the number of training rounds reaches the number of iteration rounds to obtain a target denoising network model; Obtain the current noise picture and input the current noise picture into the target denoising network model to obtain a denoised picture.

2. The image denoising method according to claim 1, characterized in that, The constructing the first sample pair dataset and the second sample pair dataset based on the initial noise image sample set includes: Obtain the initial noise image sample set, where the initial noise image sample set includes multiple initial noise images; For each initial noise image in the initial noise image sample set, perform a noise masking operation to obtain the pseudo-blind spot noise image, and construct the first sample pair based on the initial noise image and the pseudo-blind spot noise image; Add global noise to the initial noise image to generate the global noise image, and construct the second sample pair based on the initial noise image and the global noise image; Determine the first sample pair dataset based on multiple first sample pairs, and determine the second sample pair dataset based on multiple second sample pairs.

3. The image denoising method according to claim 2, wherein The performing a noise masking operation on each initial noise image in the initial noise image sample set to obtain the pseudo-blind spot noise image includes: Determine multiple pixel positions in each initial noise image; Create a filter mask with a preset size at each pixel position, move the filter mask along a preset direction at a preset step size, and add the pixel indicated by the pixel position to the corresponding mask pixel to obtain the pseudo-blind spot noise map.

4. The image denoising method according to claim 1, wherein, The bridged U-shaped denoising network model is obtained by bridging two identical U-shaped networks based on residual dense units.

5. The image denoising method according to claim 1, wherein Denoising each pseudo-blind spot noise image in the first sample pair dataset and each global noise image in the second sample pair dataset respectively based on the bridged U-shaped denoising network to generate a first predicted image and a second predicted image, including: Performing the following operations on each pseudo-blind spot noise image in the first sample pair dataset: inputting the pseudo-blind spot noise image into the bridged U-shaped denoising network to obtain a denoised image, and performing a blind spot sampling operation on the denoised image to obtain the first predicted image; Performing the following operations on each global noise image in the second sample pair dataset: inputting the global noise image into the bridged U-shaped denoising network to obtain the second predicted image.

6. The image denoising method according to claim 5, wherein The performing a blind spot sampling operation on the denoised image to obtain the first predicted image includes: Determining a plurality of noise pseudo-blind spot positions in the denoised image; Performing identity sampling on the pixel points indicated by each noise pseudo-blind spot position among the plurality of noise pseudo-blind spot positions based on a sampling function; Splicing the sampled pixel points into a target denoised picture, and using the spliced target denoised picture as the first predicted image.

7. The image denoising method according to claim 1, wherein Training the bridged U-shaped denoising network based on the first predicted image and the second predicted image, updating network parameters, and obtaining a trained denoising network model, including: Calculating a first mean square error loss based on the initial noise image and the corresponding first predicted image, and calculating a second mean square error loss based on the initial noise image and the corresponding second predicted image, determining the product of a balance coefficient and the first mean square error loss, calculating the sum of the product and the second mean square error loss to obtain a first result; Calculating the sum of multiple first results to obtain a total mean square error loss, determining to minimize the total mean square error loss, and updating the network parameters of the bridged U-shaped denoising network using the minimized total mean square error loss to obtain the trained denoising network model as the initialization model for the next round.

8. An image denoising device, characterized in that, Including: A construction module for creating a bridged U-shaped denoising network model and constructing a first sample pair dataset and a second sample pair dataset based on an initial noise image sample set, wherein each first sample pair in the first sample pair dataset is obtained by pairing an initial noise image and a pseudo-blind spot noise image, and each second sample pair in the second sample pair dataset is obtained by pairing the initial noise image and a global noise image; An initial training module for denoising each pseudo-blind spot noise image in the first sample pair dataset and each global noise image in the second sample pair dataset respectively based on the bridged U-shaped denoising network to generate a first predicted image and a second predicted image, and training the bridged U-shaped denoising network model based on the first predicted image and the second predicted image, updating network parameters, and obtaining a trained denoising network model; A data refinement module that inputs the initial noisy image sample set into the trained denoising network model to obtain an output image sample set as the initial noisy image sample set for the next round; An iterative training module that, in each round of training, uses the output image sample set obtained in the previous round to construct the first sample pair data set and the second sample pair data set for the current round. Using the denoising network model obtained in the previous round as the initialization model for the current round, it denoises each pseudo-blind spot noisy image in the first sample pair data set of the current round and each global noisy image in the second sample pair data set of the current round to generate the first prediction image and the second prediction image for the current round, and performs model training based on the first prediction image and the second prediction image for the current round until the number of training rounds reaches the iterative round to obtain the target denoising network model; A prediction module that is used to obtain the current noisy picture and input the current noisy picture into the target denoising network model to obtain a denoised picture.

9. An electronic device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 7.

10. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method described in any one of claims 1 to 7.

Images