Image denoising model training method, image denoising method and device

Abstract

The embodiments of the present application provide a training method, an image denoising method, and an apparatus for an image denoising model, wherein the training method comprises: obtaining a plurality of training sample images; inputting the training sample images into a model to be trained for training processing to obtain an image denoising model; wherein the model to be trained comprises an image denoising network and an image quality assessment network, and the training processing comprises: denoising the training sample images through the image denoising network to obtain a denoised image, and generating a generated image of the training sample images; performing quality assessment processing on the training sample images through the image quality assessment network to obtain a first evaluation score for the training sample images; determining a target loss value based on the training sample images, the denoised images, the generated images, and the first evaluation score; and adjusting the model parameters of the model to be trained based on the target loss value. Through the embodiments of the present application, the denoising effect of the image denoising model on a local part of the image is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a training method for an image denoising model, an image denoising method and a device. Background Art

[0002] With the popularization of various digital instruments and digital products, images and videos have become the most commonly used information carriers in human activities. They contain a large amount of information about objects and have become the main way for people to obtain original information from the outside world. However, during the acquisition, transmission, and storage of images, they are often interfered with and affected by various noises, which degrades the image quality. The quality of image preprocessing algorithms is directly related to the effectiveness of subsequent image processing, such as image segmentation, object recognition, and edge extraction. Therefore, in order to obtain high-quality digital images, it is necessary to perform noise reduction on the images to maintain the integrity of the original information (i.e., the main features) while removing useless information from the signal.

[0003] Currently, the commonly used denoising methods mainly use the noise estimation map as input to balance the suppression of uniformly distributed noise and the preservation of details. However, the denoising effect of locally blurred images is poor. Summary of the Invention

[0004] The present application provides an image denoising model training method, an image denoising method and an image denoising device to improve the denoising effect of locally blurred images.

[0005] In a first aspect, an embodiment of the present application provides a method for training an image denoising model, comprising:

[0006] Obtain multiple training sample images;

[0007] Inputting the training sample image into the model to be trained for training processing to obtain an image denoising model;

[0008] In which, the model to be trained includes an image denoising network and an image quality evaluation network, and the training process includes: performing denoising processing on the training sample image through the image denoising network to obtain a denoised image, and generating a generated image of the training sample image; performing quality evaluation processing on the training sample image through the image quality evaluation network to obtain a first evaluation score of the training sample image; determining a target loss value based on the training sample image, the denoised image, the generated image and the first evaluation score; and adjusting the model parameters of the model to be trained based on the target loss value.

[0009] It can be seen that in the embodiment of the present application, the model to be trained includes an image denoising network and an image quality evaluation network. During the training process of the model to be trained, the training sample image is denoised by the image denoising network to obtain a denoised image, and a generated image of the training sample image is generated. The first evaluation score of the training sample image is obtained by the image quality evaluation network, and the target loss value is determined according to the training sample image, the denoised image, the generated image and the first evaluation score, and the model parameters of the model to be trained are adjusted based on the target loss value. On the one hand, the first evaluation score can be used to provide the image denoising network with the overall blur degree of the training sample image, guide the image denoising network to perform denoising, and improve the denoising effect. On the other hand, since the generated image has a high image quality, the pixel-level representation of the image can be brought closer based on the generated image, further improving the denoising effect. Therefore, training the model to be trained can obtain an image denoising model with better denoising effect, and denoising the image by the image denoising model can improve the local denoising effect while ensuring the overall quality of the image.

[0010] In a second aspect, an embodiment of the present application provides an image denoising method, comprising:

[0011] Obtaining a noise image to be processed;

[0012] The noisy image is input into a target image denoising network of an image denoising model for denoising to obtain a target denoised image; the image denoising model is trained according to the training method of the image denoising model provided in the above aspect.

[0013] It can be seen that in the embodiment of the present application, the image denoising model used for denoising the noisy image is obtained by training a to-be-trained model including an image denoising network and an image quality evaluation network, and in the training process of the to-be-trained model, the training sample image is denoised by the image denoising network to obtain a denoised image, and a generated image of the training sample image is generated; a first evaluation score of the training sample image is obtained by the image quality evaluation network; and a target loss value is determined based on the training sample image, the denoised image, the generated image and the first evaluation score, and the model parameters of the to-be-trained model are adjusted based on the target loss value; on the one hand, the first evaluation score can be used to provide the image denoising network with the overall blur of the training sample image, guide the image denoising network to perform denoising, and improve the denoising effect; on the other hand, since the generated image has a high image quality, the pixel-level representation of the image can be brought closer based on the generated image, further improving the denoising effect. In other words, the image denoising model obtained by training the to-be-trained model has a better denoising effect. Therefore, denoising the noisy image by using the image denoising model can improve the local denoising effect while ensuring the overall quality of the image.

[0014] In a third aspect, an embodiment of the present application provides a training device for an image denoising model, comprising:

[0015] An acquisition module, used for acquiring a plurality of training sample images to be trained;

[0016] A training module, configured to input the training sample images into the model to be trained for training processing to obtain an image denoising model;

[0017] In which, the model to be trained includes an image denoising network and an image quality evaluation network, and the training process includes: performing denoising processing on the training sample image through the image denoising network to obtain a denoised image, and generating a generated image of the training sample image; performing quality evaluation processing on the training sample image through the image quality evaluation network to obtain a first evaluation score of the training sample image; determining a target loss value based on the training sample image, the denoised image, the generated image and the first evaluation score; and adjusting the model parameters of the model to be trained based on the target loss value.

[0018] In a fourth aspect, an embodiment of the present application provides an image denoising device, comprising:

[0019] An acquisition module, used for acquiring a noise image to be processed;

[0020] A denoising module is used to input the noisy image into the target image denoising network of the image denoising model for denoising processing to obtain a target denoised image; the image denoising model is trained according to the training method of the image denoising model provided in the first aspect of this application.

[0021] In a fifth aspect, an embodiment of the present application provides an electronic device, including:

[0022] A processor; and a memory arranged to store computer-executable instructions, wherein the executable instructions are configured to be executed by the processor, the executable instructions including steps for executing the training method of the image denoising model provided in the first aspect, or the executable instructions including steps for executing the image denoising method provided in the second aspect.

[0023] In a sixth aspect, an embodiment of the present application provides a storage medium for storing computer-executable instructions, wherein the executable instructions enable the computer to execute the steps in the training method of the image denoising model provided in the first aspect above, or the executable instructions enable the computer to execute the steps in the image denoising method provided in the second aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate one or more embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0025] Figure 1 A flowchart of a method for training an image denoising model provided in an embodiment of the present application;

[0026] Figure 2 A schematic diagram of the structure of an image denoising network provided in an embodiment of the present application;

[0027] Figure 3 A schematic diagram of the structure of an image quality evaluation network provided in an embodiment of the present application;

[0028] Figure 4 A flowchart of an image denoising method provided in an embodiment of the present application;

[0029] Figure 5 A schematic diagram of the module composition of a training device for an image denoising model provided in an embodiment of the present application;

[0030] Figure 6 A schematic diagram of the module composition of an image denoising device provided in an embodiment of the present application;

[0031] Figure 7 A schematic diagram of the structure of an electronic device provided in one or more embodiments of the present application. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of the present application, the technical solutions in one or more embodiments of the present application will be clearly and completely described below in conjunction with the drawings in one or more embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on one or more embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of this document.

[0033] The embodiment of the present application provides a training method for an image denoising model, an image denoising method and an apparatus. Considering that the existing image denoising method mainly uses a noise estimation map as input to balance the suppression of uniformly distributed noise and the preservation of details, the denoising effect of this denoising method is poor for locally blurred images. Based on this, the embodiment of the present application proposes a to-be-trained model including an image denoising network and an image quality evaluation network, and in the training process of the to-be-trained model, the training sample image is denoised by the image denoising network to obtain a denoised image, and a generated image of the training sample image is generated; the first evaluation score of the training sample image is obtained by the image quality evaluation network; and the target loss value is determined according to the training sample image, the denoised image, the generated image and the first evaluation score, and the model parameters of the to-be-trained model are adjusted based on the target loss value; on the one hand, the first evaluation score can be used to provide the image denoising network with the overall blur degree of the training sample image, guide the image denoising network to perform denoising, and improve the denoising effect; on the other hand, since the generated image has a high image quality, the pixel-level representation of the image can be brought closer based on the generated image, further improving the denoising effect. Therefore, training the training model can obtain an image denoising model with better denoising effect, and denoising the image through the image denoising model can improve the local denoising effect while ensuring the overall quality of the image.

[0034] Specifically, Figure 1 A flowchart of a method for training an image denoising model according to one or more embodiments of the present application is provided. Figure 1 The method can be executed by a training device for an image denoising model, which can be provided in a terminal device or a server. The terminal device can be a mobile phone, tablet computer, desktop computer, portable notebook, etc. The server can be a standalone server or a server cluster consisting of multiple servers.

[0035] like Figure 1 As shown, the method includes the following steps:

[0036] Step S102, obtaining a plurality of training sample images;

[0037] In order to effectively complete the training of the model to be trained, each training sample image in the embodiment of the present application includes a noise image and a standard denoised image corresponding to the noise image. The noise image is marked with a label, and the label includes a second evaluation score of the noise image. The second evaluation score is used to evaluate the quality of the noise image. The higher the score, the higher the image quality and the clearer the image. Conversely, the worse the image quality and the blurrier the image. The second evaluation score can be any score within a preset score range. For example, the preset score range is 0 to 100 points. The second evaluation score can be any score between 0 and 100 points, and 0 points represents the worst image quality, and 100 points represents a high-definition image. The training sample images may include faces.

[0038] It should be noted that the method for obtaining the training sample images is not specifically limited in this application and can be set as needed in actual applications.

[0039] Step S104: input the training sample image into the model to be trained for training processing to obtain an image denoising model; wherein the model to be trained includes an image denoising network and an image quality evaluation network, and the training processing includes: denoising the training sample image through the image denoising network to obtain a denoised image, and generating a generated image of the training sample image; performing quality evaluation processing on the training sample image through the image quality evaluation network to obtain a first evaluation score of the training sample image; determining a target loss value based on the training sample image, the denoised image, the generated image and the first evaluation score; and adjusting the model parameters of the model to be trained based on the target loss value.

[0040] The image quality assessment network can be a pre-trained network. During the training of the model to be trained, the parameters of the image quality assessment network remain unchanged. The resulting image denoising model includes a target image denoising network and an image quality assessment network. The target image denoising network is the trained image denoising network. In other words, the target image denoising network in the image denoising model can be used to denoise the noisy image to be processed, yielding a target denoised image.

[0041] In an embodiment of the present application, the model to be trained includes an image denoising network and an image quality evaluation network. During the training process of the model to be trained, the training sample image is denoised by the image denoising network to obtain a denoised image, and a generated image of the training sample image is generated. The first evaluation score of the training sample image is obtained by the image quality evaluation network, and the target loss value is determined according to the training sample image, the denoised image, the generated image and the first evaluation score, and the model parameters of the model to be trained are adjusted based on the target loss value. On the one hand, the first evaluation score can be used to provide the image denoising network with the overall blur degree of the training sample image, guide the image denoising network to perform denoising, and improve the denoising effect. On the other hand, since the generated image has a high image quality, the pixel-level representation of the image can be brought closer based on the generated image, further improving the denoising effect. Therefore, training the model to be trained can obtain an image denoising model with better denoising effect, and denoising the image by the image denoising model can improve the local denoising effect while ensuring the overall quality of the image.

[0042] In order to improve the local denoising effect of the image, such as Figure 2 As shown, the image denoising network in the embodiment of the present application may include a first feature extraction module, an image generation module, a post-processing module and a key point detection module. Among them, the first feature extraction module is used to perform feature extraction processing on the noise image in the training sample image to obtain a first feature map, and perform first image processing on the first feature map to obtain a second feature map, and perform second image processing on the first feature map to obtain a third feature map; the image generation module is used to generate a generated image corresponding to the noise image based on the first feature map and the second feature map; the post-processing module is used to perform smoothing processing on the third feature map to obtain a denoised image; the key point detection module is used to perform key point detection processing on the standard denoised image in the training sample image to obtain a first Gram matrix, and perform key point detection processing on the generated image to obtain a second Gram matrix.

[0043] Correspondingly, the aforementioned quality evaluation processing of the training sample image is performed through the image quality evaluation network to obtain a first evaluation score of the training sample image; the target loss value is determined based on the training sample image, the denoised image, the generated image and the first evaluation score, including: performing quality evaluation processing on the noisy image through the image quality evaluation network to obtain the first evaluation score of the noisy image; determining the target loss value based on the standard denoised image, the denoised image, the generated image, the first Gram matrix, the second Gram matrix and the first evaluation score.

[0044] Furthermore, the first feature extraction module may include N sampling units connected in sequence; the i-th sampling unit among the N sampling units includes a downsampling layer, a first fully connected layer (first FC layer), a first transformation layer (first Reshape layer), a first addition layer, an upsampling layer and a first convolution layer (for example, a 3*3 convolution) connected in sequence.

[0045] Where N is a positive integer and i is a positive integer less than or equal to N. When i is greater than 1, the input of the downsampling layer in the i-th sampling unit is connected to the output of the downsampling layer in the i-1th sampling unit. When i is less than N, the input of the first additive layer in the i-th sampling unit is also connected to the output of the upsampling layer in the i+1th sampling unit. The output of the downsampling layer is also connected to the input of the first additive layer. The first fully connected layer outputs the first feature map, the first convolutional layer outputs the second feature map, and the upsampling layer in the first sampling unit outputs the third feature map.

[0046] Furthermore, the sequential connection can be understood as the output of the previous sampling unit serving as the input of the next sampling unit, and the output of the previous layer serving as the input of the next layer. For example, if N is 4 and the four sampling units are numbered sequentially from left to right, the structure of sampling unit 4 (i.e., the fourth sampling unit) is as follows: Figure 2 As shown, it can be seen that the output of the downsampling layer 4 is connected to the input of the first fully connected layer 4, the output of the first fully connected layer 4 is connected to the input of the first transformation layer 4, the output of the first transformation layer 4 is connected to the first addition layer 4, the output of the first addition layer 4 is connected to the input of the upsampling layer 4, and the output of the upsampling layer 4 is connected to the input of the first convolutional layer 4. It can be understood that the structures of sampling unit 1 (i.e., the first sampling unit), sampling unit 2 (i.e., the second sampling unit), and sampling unit 3 (i.e., the third sampling unit) are the same as that of sampling unit 4; for example, for sampling unit 2, the input of its first addition layer 2 is connected to the output of the downsampling layer 2, the output of the first transformation layer 2, and the output of the upsampling layer 3, respectively. Figure 2 Not all of them are shown, but simply connected with dotted lines.

[0047] Furthermore, corresponding to the structure of the sampling unit, the above-mentioned step of performing feature extraction processing on the training sample image to obtain the first feature map, and performing first image processing on the first feature map to obtain the second feature map, may include:

[0048] Downsampling the first input data through a downsampling layer to obtain a first sub-feature map;

[0049] Performing feature extraction processing on the first sub-feature map through the first fully connected layer to obtain a first feature map;

[0050] Performing dimension transformation on the first feature map through a first transformation layer to obtain a second sub-feature map;

[0051] Performing a first merging process on the first sub-feature map and the second sub-feature map through a first adding layer to obtain a third sub-feature map;

[0052] Upsampling the second input data through an upsampling layer to obtain a fourth sub-feature map;

[0053] Performing dimensionality reduction and channel transformation on the fourth sub-feature map through the first convolution layer to obtain the second feature map;

[0054] Among them, N sampling units correspond to N permutation numbers; the first input data of the downsampling layer in the sampling unit with the smallest permutation number is the training sample image, and the first input data of the downsampling layer in the sampling unit with a non-small permutation number is the first sub-feature map obtained by the previous downsampling layer; the second input data of the upsampling layer in the sampling unit with the largest permutation number is the third sub-feature map, and the second input data of the upsampling layer in the sampling unit with a non-large permutation number is the fifth sub-feature map; the fifth sub-feature map is obtained by performing a first merging process on the third sub-feature map obtained by the corresponding first addition layer and the fourth sub-feature map obtained by the subsequent upsampling layer. The first merging process can be an addition process of pixel values ​​of the same pixel point. It can be understood that each sampling unit performs the above process based on the input first input data.

[0055] Furthermore, after the downsampling layer in each sampling unit downsamples the first input data, it outputs a first sub-feature map with a dimension twice as large as that of the first sub-feature map. After the upsampling layer in each sampling unit upsamples the second input data, it outputs a fourth sub-feature map with a dimension twice as large as that of the second sub-feature map. As an example, the size of the noise image in the training sample image is 480*460 and has 3 channels, which is represented as 3*480*460. The first feature extraction module includes 4 sampling units. The data processing process of the 4th sampling unit is used as an example for explanation. The input of its downsampling layer 4 is the first input data of 3*60*80. The downsampling layer 4 downsamples the input first input data of 3*60*80 and outputs a first sub-feature map of 3*30*40. The first fully connected layer 4 performs feature extraction on the input first sub-feature map of 3*30*40 and outputs a first feature map of 3*1200*1. The first transformation layer 4 extracts the 3*30*40 first sub-feature map and outputs a first feature map of 3*1200*1. The first feature map of size *1200*1 is dimensional transformed and a second sub-feature map of size 3*30*40 is output; the first addition layer 4 performs a first merging process on the input first sub-feature map of size 3*30*40 and the second sub-feature map of size 3*30*40, and outputs a third sub-feature map of size 3*30*40; the upsampling layer 4 performs an upsampling process on the input third sub-feature map of size 3*30*40 (i.e., the second input data), and outputs a fourth sub-feature map of size 3*60*80; the first convolutional layer 4 performs a dimensionality reduction process and a channel transformation process on the input fourth sub-feature map of size 3*60*80 to obtain a second feature map of size 64*30*40, i.e., the second feature map has 64 channels.

[0056] Thus, the first feature extraction module generates the first feature map and the second feature map based on the noise image in the input training sample image. In order to better improve the pixel-level representation and enhance the denoising effect of the image, in the embodiment of the present application, the image generation module generates the generated image corresponding to the noise image based on the first feature map and the second feature map. Specifically, Figure 2 As shown, the image generation module includes a first pooling layer, and a regularization processing layer, a second transformation layer (second Reshape layer), a branch unit, a merging layer (Concat layer), a fourth convolutional layer (for example, 3*3 convolution) and a DCGAN network (i.e., a deep convolutional generative adversarial network) connected in sequence; the branch unit includes a second convolutional layer (for example, 3*3 convolution), and a third convolutional layer (for example, 3*3 convolution), a multiplication layer and a second addition layer connected in sequence.

[0057] The input of the first pooling layer is connected to the output of each first convolutional layer, the output of the first pooling layer is connected to the input of the multiplication layer, and the first pooling layer can be a global maximum pooling layer. The input of the regularization processing layer is connected to the output of each first fully connected layer. The output of the second transformation layer is connected to the input of the second convolutional layer and the input of the third convolutional layer. The input of the merging layer is connected to the output of the second addition layer and the output of the second convolutional layer. The DCGAN network output generates an image. The DCGAN network can be a pre-trained network. Its training process can refer to existing methods, and this application does not make specific limitations on this.

[0058] Corresponding to the structure of the image generation module, the above-mentioned generating an image corresponding to the noise image according to the first feature map and the second feature map may include:

[0059] Regularizing the first feature map through a regularization processing layer to obtain a fourth feature map;

[0060] Transforming the fourth feature map through a second transformation layer to obtain a fifth feature map;

[0061] Performing feature extraction processing on the fifth feature map through the third convolutional layer to obtain a sixth feature map;

[0062] The second feature map of the M channel is compressed by the first pooling layer to obtain M weights; M is a positive integer;

[0063] The sixth feature map and M weights are multiplied by a multiplication layer to obtain a seventh feature map;

[0064] Performing a first merging process on the seventh feature map and the second feature map through a second addition layer to obtain an eighth feature map;

[0065] Performing feature extraction and channel transformation on the fifth feature map through the second convolutional layer to obtain a ninth feature map;

[0066] Performing a second merging process on the eighth feature map and the ninth feature map through a merging layer to obtain a tenth feature map;

[0067] Perform channel transformation on the tenth feature map through the fourth convolution layer to obtain the eleventh feature map;

[0068] A generated image corresponding to the noise image is generated according to the eleventh feature map through the DCGAN network, and the generated image and a third evaluation score of the generated image are output.

[0069] The first pooling layer compresses the second feature maps of the M channels and outputs M 1*1 vectors, i.e., M numerical values, which are then determined as M weights. The DCGAN network includes a discriminator that can generate a third evaluation score that represents the probability that the generated image is a standard denoised image; a higher third evaluation score indicates a greater probability that the generated image is a standard denoised image, and vice versa. The second merging process can be a channel splicing process.

[0070] Continuing with the above example, the regularization layer performs regularization on the input 3*1200*1 first feature map and outputs a 3*1200*1 fourth feature map, the pixel values ​​of which are different from those of the first feature map; the second transformation layer performs transformation processing (i.e., dimension transformation processing) on ​​the input 3*1200*1 fourth feature map and outputs a 3*30*40 fifth feature map; the third convolution layer performs feature extraction processing on the input 3*30*40 fifth feature map and outputs a 3*30*40 sixth feature map; the first pooling layer compresses the input 64-channel 30*40 second feature map and outputs 64 weights; the multiplication layer multiplies the input 3*30*40 sixth feature map and 64 weights and outputs a 64*30*40 seventh feature map; the second addition layer performs feature extraction processing on the input 64*3 The 0*40 seventh feature map and the 64*30*40 second feature map are first merged (i.e., the pixel values ​​of the same pixels are added), and the 64*30*40 eighth feature map is output; the second convolutional layer performs feature extraction and channel transformation on the input 3*30*40 fifth feature map, and outputs a 64*30*40 ninth feature map; the merging layer performs a second merging process (i.e., channel splicing) on ​​the input 64*30*40 eighth feature map and the 64*30*40 ninth feature map, and outputs a 128*30*40 tenth feature map; the fourth convolutional layer performs channel transformation on the input 128*30*40 tenth feature map, and outputs a 64*30*40 eleventh feature map; the DCGAN network generates a 3*480*460 generated image based on the input 64*30*40 eleventh feature map.

[0071] Thus, the generated image corresponding to the noise image in the training sample image is generated by the image generation module, so that key point detection can be performed based on the generated image, and the target loss value can be determined based on the detection result, thereby optimizing the model parameters. Specifically, Figure 2 As shown, the key point detection module includes a detection layer; the generated image and the standard denoised image include a face, and the key point detection processing of the standard denoised image is performed to obtain a first Gram matrix, and the key point detection processing of the generated image is performed to obtain a second Gram matrix. The following operations can be performed by the detection layer:

[0072] Perform key point detection on the standard denoised image according to a preset key point detection algorithm to obtain the left eye area, right eye area, nose area and mouth area;

[0073] Perform key point detection on the generated image according to a preset key point detection algorithm to obtain a left eye image, a right eye image, a nose image, and a mouth image;

[0074] The left eye area, right eye area, nose area and mouth area of ​​the standard denoised image are transformed to obtain the first Gram matrix;

[0075] The left eye area, right eye area, nose area, and mouth area of ​​the generated image are transformed to obtain a second Gram matrix.

[0076] The conversion process may refer to the existing conversion method, which will not be described in detail in this application.

[0077] By obtaining the first Gram matrix and the second Gram matrix, the corresponding loss value can be calculated based on them, so that the training sample image corresponding to the generated image is closer in texture to its standard denoised image.

[0078] Further, such as Figure 2 As shown, the post-processing module includes a ninth convolutional layer, which smoothes the third feature map output by the upsampling layer in the first sampling unit and outputs a denoised image. The ninth convolutional layer can be a 3*3 convolution.

[0079] The above is the structure of the image denoising network and its image processing process. The following describes the structure of the image quality evaluation network and its image processing process. Figure 3 As shown, the image quality assessment network includes a fifth convolutional layer, a sixth convolutional layer, a third addition layer, a second feature extraction module, a second pooling layer, a second fully connected layer, and a third fully connected layer, which are connected in sequence. The input of the fifth convolutional layer includes the training sample image, and the output of the fifth convolutional layer is also connected to the input of the third addition layer. The third fully connected layer outputs the first evaluation score. The fifth and sixth convolutional layers can both be 3*3 convolutions with 64 channels, and the second pooling layer can be a global pooling layer.

[0080] Correspondingly, the aforementioned quality evaluation process for the noisy image to obtain the first evaluation score of the noisy image may include:

[0081] Perform channel transformation processing on the noise image in the training sample image through the fifth convolution layer to obtain the twelfth feature map;

[0082] Perform feature extraction processing on the twelfth feature map through the sixth convolutional layer to obtain a thirteenth feature map;

[0083] Performing a first merging process on the twelfth feature map and the thirteenth feature map through a third adding layer to obtain a fourteenth feature map;

[0084] Performing feature extraction processing on the fourteenth feature map by a second feature extraction module to obtain a fifteenth feature map;

[0085] The fifteenth feature map is converted by the second pooling layer to obtain a sixteenth feature map;

[0086] The sixteenth feature map is converted by the second fully connected layer to obtain a seventeenth feature map;

[0087] The seventeenth feature map is converted through the third fully connected layer to obtain a first evaluation score.

[0088] Taking the second fully connected layer including 64 neurons and the third fully connected layer including 1 neuron as an example, the fifth convolutional layer performs channel transformation processing on the input 3*480*640 noise image and outputs the twelfth feature map of 64*480*640; the sixth convolutional layer performs feature extraction processing on the input twelfth feature map of 64*480*640 and outputs the thirteenth feature map of 64*480*640; the third addition layer performs the first merging processing on the input twelfth feature map of 64*480*640 and the thirteenth feature map of 64*480*640 , outputs the 14th feature map of 64*480*640; the second feature extraction module performs feature extraction processing on the input 14th feature map of 64*480*640, and outputs the 15th feature map of 64*480*640; the second pooling layer converts the input 15th feature map of 64*480*640, and outputs the 16th feature map of 1*1; the second fully connected layer converts the input 16th feature map of 1*1, and outputs the 17th feature map of 64*1; the third fully connected layer converts the input 17th feature map of 64*1, and outputs the first evaluation score. It can be understood that the third fully connected layer converts the input 17th feature map of 64*1, and outputs a 1*1 vector, that is, a numerical value, which is the first evaluation score.

[0089] Further, such as Figure 3 As shown, the second feature extraction module includes P extraction units connected in sequence; the j-th extraction unit in the P extraction units includes a third pooling layer, a seventh convolutional layer, an eighth convolutional layer and a fourth addition layer connected in sequence;

[0090] Where P is a positive integer, and j is a positive integer less than or equal to P. The output of the seventh convolutional layer is also connected to the input of the fourth additive layer. When j is 1, the input of the third pooling layer of the jth extraction unit is connected to the output of the third additive layer. When j is greater than 1, the input of the third pooling layer of the jth extraction unit is connected to the output of the j-1th extraction unit. In one embodiment, P can be 4.

[0091] Corresponding to the structure of the second feature extraction module, the aforementioned feature extraction processing of the fourteenth feature map by the second feature extraction module to obtain the fifteenth feature map may include:

[0092] Performing dimensionality reduction processing on the third input data through the third pooling layer to obtain a sixth sub-feature map;

[0093] Performing dimension transformation on the sixth sub-feature map through the seventh convolutional layer to obtain the seventh sub-feature map;

[0094] Performing feature extraction processing on the seventh sub-feature map through the eighth convolutional layer to obtain an eighth sub-feature map;

[0095] Performing a first merging process on the seventh sub-feature map and the eighth sub-feature map through a fourth adding layer to obtain a ninth sub-feature map or a fifteenth sub-feature map;

[0096] Among them, P extraction units correspond to P arrangement numbers, the third input data of the third pooling layer in the extraction unit with the smallest arrangement number is the fourteenth feature map, and the third input data of the third pooling layer in the extraction unit whose arrangement number is not the smallest is the ninth sub-feature map output by the previous extraction unit; the extraction unit with the largest arrangement number outputs the fifteenth feature map.

[0097] Taking extraction unit 1 (i.e., the first extraction unit) as an example, following the aforementioned example, the third pooling layer of extraction unit 1 performs dimensionality reduction processing on the input 14th feature map (i.e., the third input data) of 64*480*640, and outputs the sixth sub-feature map of 64*240*320; the seventh convolutional layer of extraction unit 1 performs dimensionality transformation processing on the input sixth sub-feature map of 64*240*320, and outputs the seventh sub-feature map of 64*480*640; the eighth convolutional layer of extraction unit 1 performs feature extraction processing on the input seventh sub-feature map of 64*480*640, and obtains the eighth sub-feature map of 64*480*640; the fourth addition layer of extraction unit 1 performs a first merging processing on the input seventh sub-feature map of 64*480*640 and the eighth sub-feature map of 64*480*640, and outputs the ninth sub-feature map of 64*480*640.

[0098] After obtaining the denoised image, the generated image, the first Gram matrix, and the second Gram matrix through the image denoising network, and obtaining the first evaluation score through the image quality assessment network, the target loss value can be calculated based on the determined target loss function. Specifically, before determining the target loss value based on the standard denoised image, the denoised image, the generated image, the first Gram matrix, the second Gram matrix, and the first evaluation score, the method may further include:

[0099] Determine a first loss function based on the denoised image and the standard denoised image; determine a second loss function based on the standard denoised image and the generated image; determine a third loss function based on the third evaluation score; determine a fourth loss function based on the first Gram matrix and the second Gram matrix; determine a fifth loss function based on the first evaluation score and the second evaluation score in the label of the training sample image; determine a target loss function based on the first loss function, the second loss function, the third loss function, the fourth loss function and the fifth loss function;

[0100] Correspondingly, determining the target loss value according to the standard denoised image, the denoised image, the generated image, the first Gram matrix, the second Gram matrix and the first evaluation score may include: based on the target loss function, determining the target loss value according to the standard denoised image, the denoised image, the generated image, the first Gram matrix, the second Gram matrix and the first evaluation score.

[0101] Among them, based on the target loss function, the target loss value is determined according to the standard denoised image, the denoised image, the generated image, the first Gram matrix, the second Gram matrix and the first evaluation score. The first loss value can be first calculated based on the denoised image and the standard denoised image according to the first loss function; the second loss value is calculated based on the standard denoised image and the generated image according to the second loss function; the third loss value is calculated based on the third evaluation score according to the third loss function; the fourth loss value is calculated based on the first Gram matrix and the second Gram matrix according to the fourth loss function; the fifth loss value is calculated based on the first evaluation score and the second evaluation score in the label of the noisy image according to the fifth loss function; and then the target loss value is calculated based on the first loss value, the second loss value, the third loss value, the fourth loss value and the fifth loss value according to the target loss function.

[0102] In one embodiment, the above-mentioned first loss function can be expressed as: Loss1 = ||HR-LR||1; wherein Loss1 represents the first loss value, HR represents the standard denoised image in the label of the training sample image, LR represents the denoised image obtained by denoising the training sample image, and ||HR-LR||1 represents the calculation of the Euclidean distance between the same pixel points of the standard denoised image and the denoised image.

[0103] In one embodiment, the above-mentioned second loss function can be expressed as: Loss2 = ||HR-DR||1; wherein Loss2 represents the second loss value, HR represents the standard denoised image in the label of the training sample image, DR represents the generated image corresponding to the training sample image generated by the image generation module, and ||HR-DR||1 represents the Euclidean distance between the same pixel points of the standard denoised image and the generated image.

[0104] In one embodiment, the third loss function can be expressed as: Loss3 = -log(e GR +1); where Loss3 represents the third loss value, GR represents the third evaluation score of the generated image, and e is a natural constant. The third loss function can make the faces in the generated images more realistic.

[0105] In one embodiment, the fourth loss function can be expressed as: Here, Loss4 represents the fourth loss value, gram(HR) represents the first Gram matrix of the standard denoised image HR, gram(DR) represents the second Gram matrix of the generated image DR, and e is a natural constant. (gram(HR)-gram(DR))1 means that the first Gram matrix is ​​first subtracted from the second Gram matrix to obtain the target matrix, and then the average of each element in the target matrix is ​​taken. Because the Gram matrix can capture the general style of the entire image, the fourth loss function can measure the difference in the general style of the two images, making the face in the generated image DR closer to the style of real facial features.

[0106] In one embodiment, the fifth loss function can be expressed as: Loss5 = (IQA - mos) 2 ; Among them, Loss5 represents the fifth loss value, IQA represents the second evaluation score in the label of the training sample image, and mos represents the first evaluation score obtained by the image quality assessment network.

[0107] In one embodiment, the above target loss function can be expressed as: Loss6=Loss1+Loss2+Loss3+Loss4+Loss5; wherein Loss6 represents the target loss value.

[0108] It should be noted that the target loss value calculation operation can be performed within the model to be trained or outside the model to be trained. When the target loss value calculation operation is performed within the model to be trained, the model to be trained also includes a loss calculation layer, the input of which is connected to the output of the image processing network and the output of the quality assessment network respectively. The loss calculation layer is used to calculate the target loss value in the above manner and output the target loss value.

[0109] The above-mentioned loss functions in the embodiments of the present application can effectively supervise the cyclic training of the model to be trained, and improve the local denoising effect of the model while ensuring the overall image quality. It should be noted that the above-mentioned loss functions are only used for illustration and not for limitation, and can be adjusted as needed in actual applications. For example, Loss6 can be a linear weighted sum of Loss1 to Loss5.

[0110] In one or more embodiments of the present application, the model to be trained includes an image denoising network and an image quality evaluation network. During the training process of the model to be trained, the training sample image is denoised by the image denoising network to obtain a denoised image, and a generated image of the training sample image is generated; a first evaluation score of the training sample image is obtained by the image quality evaluation network; and a target loss value is determined based on the training sample image, the denoised image, the generated image and the first evaluation score, and the model parameters of the model to be trained are adjusted based on the target loss value; on the one hand, the first evaluation score can be used to provide the image denoising network with the overall blur degree of the training sample image, guide the image denoising network to perform denoising, and improve the denoising effect; on the other hand, since the generated image has a high image quality, the pixel-level representation of the image can be brought closer based on the generated image, further improving the denoising effect. Therefore, training the model to be trained can obtain an image denoising model with better denoising effect, and denoising the image by the image denoising model can improve the local denoising effect while ensuring the overall quality of the image.

[0111] Corresponding to the training method of the image denoising model described above, based on the same technical concept, one or more embodiments of the present application further provide an image denoising method. Figure 4 A flowchart of an image denoising method provided in one or more embodiments of the present application is provided. Figure 4 The method can be performed by an image denoising device; the image denoising device can be set in a terminal device or in a server. The terminal device can be a mobile phone, a tablet computer, a desktop computer, a portable notebook, etc. The server can be an independent server or a server cluster composed of multiple servers. Figure 4 As shown, the method includes the following steps:

[0112] Step S202, obtaining a noise image to be processed;

[0113] Step S204: input the noisy image into the target image denoising network of the image denoising model for denoising, thereby obtaining a target denoised image.

[0114] The image denoising model is trained according to the training method of the image denoising model provided in the above embodiment. The target image denoising network of the image denoising model may include a first feature extraction module and a post-processing module; accordingly, step S204 may include:

[0115] The noise image is input into the first feature extraction module for feature extraction processing to obtain a third feature map; the third feature map is input into the post-processing module for smoothing processing to obtain a target denoised image.

[0116] In the embodiment of the present application, the image denoising model used for denoising a noisy image is obtained by training a to-be-trained model including an image denoising network and an image quality evaluation network, and during the training process of the to-be-trained model, the training sample image is denoised by the image denoising network to obtain a denoised image, and a generated image of the training sample image is generated; a first evaluation score of the training sample image is obtained by the image quality evaluation network; and a target loss value is determined based on the training sample image, the denoised image, the generated image, and the first evaluation score, and the model parameters of the to-be-trained model are adjusted based on the target loss value; on the one hand, the first evaluation score can be used to provide the image denoising network with the overall blur of the training sample image, guiding the image denoising network to perform denoising and improving the denoising effect; on the other hand, since the generated image has a high image quality, the pixel-level representation of the image can be brought closer based on the generated image, further improving the denoising effect. In other words, the image denoising model obtained by training the to-be-trained model has a better denoising effect, so denoising a noisy image by using the image denoising model can improve the local denoising effect while ensuring the overall quality of the image.

[0117] Corresponding to the above-described method for training an image denoising model, based on the same technical concept, one or more embodiments of the present application also provide a device for training an image denoising model. Figure 5 A schematic diagram of the module composition of a training device for an image denoising model provided in one or more embodiments of the present application, such as Figure 5 As shown, the device includes:

[0118] An acquisition module 301 is used to acquire a plurality of training sample images to be trained;

[0119] A training module 302 is configured to input the training sample image into the model to be trained for training, thereby obtaining an image denoising model;

[0120] In which, the model to be trained includes an image denoising network and an image quality evaluation network, and the training process includes: performing denoising processing on the training sample image through the image denoising network to obtain a denoised image, and generating a generated image of the training sample image; performing quality evaluation processing on the training sample image through the image quality evaluation network to obtain a first evaluation score of the training sample image; determining a target loss value based on the training sample image, the denoised image, the generated image and the first evaluation score; and adjusting the model parameters of the model to be trained based on the target loss value.

[0121] The training device of the image denoising model provided in the embodiment of the present application inputs the acquired training sample image into the model to be trained for training processing to obtain the image denoising model; wherein the model to be trained includes an image denoising network and an image quality evaluation network, and in the training process of the model to be trained, the training sample image is denoised by the image denoising network to obtain a denoised image, and a generated image of the training sample image is generated; a first evaluation score of the training sample image is obtained by the image quality evaluation network; and a target loss value is determined according to the training sample image, the denoised image, the generated image and the first evaluation score, and the model parameters of the model to be trained are adjusted based on the target loss value; on the one hand, the first evaluation score can be used to provide the image denoising network with the overall blur degree of the training sample image, guide the image denoising network to perform denoising processing, and improve the denoising effect; on the other hand, since the generated image has a high image quality, the pixel-level representation of the image can be brought closer based on the generated image, further improving the denoising effect. Therefore, training the model to be trained can obtain an image denoising model with better denoising effect, and denoising the image by the image denoising model can improve the local denoising effect while ensuring the overall quality of the image.

[0122] It should be noted that the embodiment of the training device for the image denoising model in this application and the embodiment of the training method for the image denoising model in this application are based on the same inventive concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the corresponding training method for the image denoising model mentioned above, and the repeated parts will not be repeated.

[0123] Furthermore, corresponding to the image denoising method described above, based on the same technical concept, one or more embodiments of the present application also provide an image denoising device. Figure 6 A schematic diagram of the module composition of an image denoising device provided in one or more embodiments of the present application is shown in FIG. Figure 6 As shown, the device includes:

[0124] An acquisition module 401 is used to acquire a noise image to be processed;

[0125] The denoising module 402 is used to input the noisy image into the target image denoising network of the image denoising model for denoising processing to obtain a target denoised image; the image denoising model is trained according to the training method of the image denoising model provided in the previous embodiment of the present application.

[0126] The image denoising device provided in the embodiment of the present application is a device that uses an image denoising model used when denoising a noisy image to obtain a denoised image by training a to-be-trained model including an image denoising network and an image quality assessment network. In the training process of the to-be-trained model, the image denoising network denoises the training sample image to obtain a denoised image and generates a generated image of the training sample image; the image quality assessment network obtains a first evaluation score of the training sample image; and the target loss value is determined based on the training sample image, the denoised image, the generated image, and the first evaluation score, and the model parameters of the to-be-trained model are adjusted based on the target loss value. On the one hand, the first evaluation score can be used to provide the image denoising network with the overall blur of the training sample image, guiding the image denoising network to perform denoising and improving the denoising effect; on the other hand, since the generated image has a high image quality, the pixel-level representation of the image can be brought closer based on the generated image, further improving the denoising effect. In other words, the image denoising model obtained by training the to-be-trained model has a better denoising effect. Therefore, denoising the noisy image using the image denoising model can improve the local denoising effect while ensuring the overall image quality.

[0127] It should be noted that the embodiment of the image denoising device in this application and the embodiment of the image denoising method in this application are based on the same inventive concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the corresponding image denoising method mentioned above, and the repeated parts will not be repeated.

[0128] Furthermore, corresponding to the above-described image denoising model training method, based on the same technical concept, one or more embodiments of the present application further provide an electronic device, which is used to execute the above-described image denoising model training method. Figure 7 A schematic diagram of the structure of an electronic device provided in one or more embodiments of the present application.

[0129] like Figure 7As shown, electronic devices may have relatively large differences due to different configurations or performances, and may include one or more processors 501 and memory 502, and the memory 502 may store one or more storage applications or data. Among them, the memory 502 can be a temporary storage or a persistent storage. The application stored in the memory 502 may include one or more modules (not shown in the figure), each module may include a series of computer executable instructions in the electronic device. Furthermore, the processor 501 can be configured to communicate with the memory 502 to execute a series of computer executable instructions in the memory 502 on the electronic device. The electronic device may also include one or more power supplies 503, one or more wired or wireless network interfaces 504, one or more input and output interfaces 505, one or more keyboards 506, etc.

[0130] In a specific embodiment, the electronic device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the electronic device, and the one or more programs are configured to be executed by one or more processors, including computer-executable instructions for performing the following:

[0131] Obtaining multiple training sample images to be trained;

[0132] Inputting the training sample image into the model to be trained for training processing to obtain an image denoising model;

[0133] In which, the model to be trained includes an image denoising network and an image quality evaluation network, and the training process includes: performing denoising processing on the training sample image through the image denoising network to obtain a denoised image, and generating a generated image of the training sample image; performing quality evaluation processing on the training sample image through the image quality evaluation network to obtain a first evaluation score of the training sample image; determining a target loss value based on the training sample image, the denoised image, the generated image and the first evaluation score; and adjusting the model parameters of the model to be trained based on the target loss value.

[0134] One or more embodiments of the present application provide an electronic device that inputs an acquired training sample image into a to-be-trained model and iteratively trains the to-be-trained model using a target loss value to obtain an image denoising model; wherein the to-be-trained model includes an image denoising network and an image quality assessment network. During the training process of the to-be-trained model, the image denoising network denoises the training sample image to obtain a denoised image and generates a generated image of the training sample image; the image quality assessment network obtains a first evaluation score for the training sample image; and the target loss value is determined based on the training sample image, the denoised image, the generated image, and the first evaluation score, and the model parameters of the to-be-trained model are adjusted based on the target loss value; on the one hand, the first evaluation score can be used to provide the image denoising network with an overall blur degree of the training sample image, guiding the image denoising network to perform denoising and improving the denoising effect; on the other hand, since the generated image has a high image quality, the pixel-level representation of the image can be brought closer based on the generated image, further improving the denoising effect. Thus, training the to-be-trained model can obtain an image denoising model with better denoising effect, and denoising the image using the image denoising model can improve the local denoising effect while ensuring the overall image quality.

[0135] In another specific embodiment, the electronic device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the electronic device, and the one or more programs are configured to be executed by one or more processors, including computer-executable instructions for performing the following:

[0136] Obtaining a noise image to be processed;

[0137] The noisy image is input into a target image denoising network of an image denoising model for denoising to obtain a target denoised image; the image denoising model is trained according to the training method of the image denoising model provided in the aforementioned embodiment.

[0138] The electronic device provided in the embodiment of the present application is characterized in that the image denoising model used for denoising a noisy image is obtained by training a to-be-trained model including an image denoising network and an image quality evaluation network, and in the training process of the to-be-trained model, the image denoising network is used to denoise the training sample image to obtain a denoised image, and a generated image of the training sample image is generated; a first evaluation score of the training sample image is obtained by the image quality evaluation network; and a target loss value is determined based on the training sample image, the denoised image, the generated image, and the first evaluation score, and the model parameters of the to-be-trained model are adjusted based on the target loss value; on the one hand, the first evaluation score can be used to provide the image denoising network with the overall blur degree of the training sample image, guiding the image denoising network to perform denoising and improving the denoising effect; on the other hand, since the generated image has a high image quality, the pixel-level representation of the image can be brought closer based on the generated image, further improving the denoising effect. In other words, the image denoising model obtained by training the to-be-trained model has a better denoising effect, so denoising the noisy image by using the image denoising model can improve the local denoising effect while ensuring the overall image quality.

[0139] It should be noted that the embodiment of the electronic device in this application and the embodiment of the training method of the image denoising model and the image denoising method in this application are based on the same inventive concept. Therefore, the specific implementation of this embodiment can refer to the aforementioned corresponding training method of the image denoising model and the implementation of the image denoising method, and the repeated parts will not be repeated.

[0140] Furthermore, corresponding to the training method of the image denoising model described above, based on the same technical concept, one or more embodiments of the present application further provide a storage medium for storing computer-executable instructions. In a specific embodiment, the storage medium may be a USB flash drive, an optical disk, a hard disk, etc. When the computer-executable instructions stored in the storage medium are executed by the processor, the following process can be implemented:

[0141] Obtaining multiple training sample images to be trained;

[0142] Inputting the training sample image into the model to be trained for training processing to obtain an image denoising model;

[0143] In which, the model to be trained includes an image denoising network and an image quality evaluation network, and the training process includes: performing denoising processing on the training sample image through the image denoising network to obtain a denoised image, and generating a generated image of the training sample image; performing quality evaluation processing on the training sample image through the image quality evaluation network to obtain a first evaluation score of the training sample image; determining a target loss value based on the training sample image, the denoised image, the generated image and the first evaluation score; and adjusting the model parameters of the model to be trained based on the target loss value.

[0144] When the computer executable instructions stored in the storage medium provided by one or more embodiments of the present application are executed by the processor, the acquired training sample image is input into the model to be trained for training processing to obtain an image denoising model; wherein, the model to be trained includes an image denoising network and an image quality evaluation network, and in the training process of the model to be trained, the training sample image is denoised by the image denoising network to obtain a denoised image, and a generated image of the training sample image is generated; a first evaluation score of the training sample image is obtained by the image quality evaluation network; and a target loss value is determined according to the training sample image, the denoised image, the generated image and the first evaluation score, and the model parameters of the model to be trained are adjusted based on the target loss value; on the one hand, the first evaluation score can be used to provide the image denoising network with the overall blur degree of the training sample image, guide the image denoising network to perform denoising processing, and improve the denoising effect; on the other hand, since the generated image has a higher image quality, the pixel-level representation of the image can be zoomed in based on the generated image, thereby further improving the denoising effect. Therefore, training the training model can obtain an image denoising model with better denoising effect, and denoising the image through the image denoising model can improve the local denoising effect while ensuring the overall quality of the image.

[0145] In another specific embodiment, the storage medium may be a USB flash drive, an optical disk, a hard disk, etc., and the computer executable instructions stored in the storage medium, when executed by the processor, can implement the following process:

[0146] Obtaining a noise image to be processed;

[0147] The noisy image is input into a target image denoising network of an image denoising model for denoising to obtain a target denoised image; the image denoising model is trained according to the training method of the image denoising model provided in the aforementioned embodiment.

[0148] When the computer-executable instructions stored in the storage medium provided by one or more embodiments of the present application are executed by a processor, the image denoising model used for denoising a noisy image is obtained by training a to-be-trained model including an image denoising network and an image quality assessment network, and during the training process of the to-be-trained model, the training sample image is denoised by the image denoising network to obtain a denoised image, and a generated image of the training sample image is generated; a first evaluation score of the training sample image is obtained by the image quality assessment network; and a target loss value is determined based on the training sample image, the denoised image, the generated image, and the first evaluation score, and the model parameters of the to-be-trained model are adjusted based on the target loss value; on the one hand, the first evaluation score can be used to provide the image denoising network with an overall blur degree of the training sample image, guiding the image denoising network to perform denoising and improving the denoising effect; on the other hand, since the generated image has a higher image quality, the pixel-level representation of the image can be brought closer based on the generated image, further improving the denoising effect. In other words, the image denoising model obtained by training the to-be-trained model has a better denoising effect, so denoising the noisy image by using the image denoising model can improve the local denoising effect while ensuring the overall image quality.

[0149] It should be noted that the embodiment of the storage medium in this application and the embodiment of the training method of the image denoising model and the image denoising method in this application are based on the same inventive concept. Therefore, the specific implementation of this embodiment can refer to the aforementioned corresponding training method of the image denoising model and the implementation of the image denoising method, and the repeated parts will not be repeated.

[0150] The foregoing description describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0151] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures such as diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to hire a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0152] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the means for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the means for implementing various functions can be considered as both a software module implementing the method and a structure within the hardware component.

[0153] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0154] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing the embodiments of the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0155] Those skilled in the art will appreciate that one or more embodiments of the present application may be provided as a method, system, or computer program product. Therefore, one or more embodiments of the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0156] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0157] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0158] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0159] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0160] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0161] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0162] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0163] One or more embodiments of the present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0164] The various embodiments in this application are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiment is generally similar to the method embodiment, so the description is relatively simple. For relevant parts, refer to the partial description of the method embodiment.

[0165] The foregoing description is merely an example of the present invention and is not intended to limit the present invention. Persons skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of the claims herein.

Claims

1. A training method for an image denoising model, characterized in that: include: Obtain multiple training sample images; Inputting the training sample image into the model to be trained for training processing to obtain an image denoising model; In which, the model to be trained includes an image denoising network and an image quality evaluation network, and the training process includes: denoising the training sample image through the image denoising network to obtain a denoised image, and generating a generated image of the training sample image; the generated image is generated based on a first feature map and a second feature map, the first feature map is obtained by performing feature extraction processing on the noise image included in the training sample image, and the second feature map is obtained by performing first image processing on the first feature map; quality evaluation processing is performed on the training sample image through the image quality evaluation network to obtain a first evaluation score of the training sample image; a target loss value is determined based on the training sample image, the denoised image, the generated image and the first evaluation score; and the model parameters of the model to be trained are adjusted based on the target loss value.

2. The method according to claim 1, characterized in that The training sample image includes a noisy image and a standard denoised image corresponding to the noisy image; the image denoising network includes a first feature extraction module, an image generation module, a post-processing module and a key point detection module; The first feature extraction module is configured to perform feature extraction processing on the noise image to obtain a first feature map, perform first image processing on the first feature map to obtain a second feature map, and perform second image processing on the first feature map to obtain a third feature map; the image generation module is configured to generate the generated image based on the first feature map and the second feature map; the post-processing module is configured to perform smoothing processing on the third feature map to obtain the denoised image; the key point detection module is configured to perform key point detection processing on the standard denoised image to obtain a first Gram matrix, and perform key point detection processing on the generated image to obtain a second Gram matrix; The method comprises: performing quality evaluation processing on the training sample image through the image quality evaluation network to obtain a first evaluation score of the training sample image; determining a target loss value based on the training sample image, the denoised image, the generated image and the first evaluation score, including: performing quality evaluation processing on the noisy image through the image quality evaluation network to obtain a first evaluation score of the noisy image; and determining a target loss value based on the standard denoised image, the denoised image, the generated image, the first Gram matrix, the second Gram matrix and the first evaluation score.

3. The method according to claim 2, characterized in that The first feature extraction module includes N sampling units connected in sequence; the i-th sampling unit among the N sampling units includes a downsampling layer, a first fully connected layer, a first transformation layer, a first addition layer, an upsampling layer and a first convolutional layer connected in sequence; N is a positive integer, and i is a positive integer less than or equal to N; When i is greater than 1, the input of the downsampling layer in the i-th sampling unit is connected to the output of the downsampling layer in the i-1-th sampling unit; When i is less than N, the input of the first addition layer in the i-th sampling unit is also connected to the output of the upsampling layer in the i+1-th sampling unit; The output of the downsampling layer is also connected to the input of the first addition layer, the first fully connected layer outputs the first feature map, the first convolutional layer outputs the second feature map, and the upsampling layer in the first sampling unit outputs the third feature map.

4. The method according to claim 3, characterized in that The image generation module includes a first pooling layer, and a regularization processing layer, a second transformation layer, a branch unit, a merging layer, a fourth convolutional layer and a DCGAN network connected in sequence; the branch unit includes a second convolutional layer, and a third convolutional layer, a multiplication layer and a second addition layer connected in sequence; The input of the first pooling layer is connected to the output of each of the first convolutional layers, and the output of the first pooling layer is connected to the input of the multiplication layer; the input of the regularization layer is connected to the output of each of the first fully connected layers; the output of the second transformation layer is connected to the input of the second convolutional layer and the input of the third convolutional layer; the input of the merging layer is connected to the output of the second addition layer and the output of the second convolutional layer; The DCGAN network outputs the generated image.

5. The method according to claim 2, characterized in that The standard denoised image and the generated image include a human face; performing key point detection processing on the standard denoised image to obtain a first Gram matrix, and performing key point detection processing on the generated image to obtain a second Gram matrix, include: Performing key point detection processing on the standard denoised image according to a preset key point detection algorithm to obtain a left eye area, a right eye area, a nose area, and a mouth area; Performing key point detection processing on the generated image according to the preset key point detection algorithm to obtain a left eye image, a right eye image, a nose image, and a mouth image; Performing conversion processing on the left eye area, the right eye area, the nose area, and the mouth area of ​​the standard denoised image to obtain a first Gram matrix; The left eye area, the right eye area, the nose area, and the mouth area of ​​the generated image are transformed to obtain a second Gram matrix.

6. The method according to claim 1, wherein The image quality assessment network includes a fifth convolutional layer, a sixth convolutional layer, a third addition layer, a second feature extraction module, a second pooling layer, a second fully connected layer and a third fully connected layer connected in sequence; The input of the fifth convolutional layer includes the training sample image, and the output of the fifth convolutional layer is also connected to the input of the third addition layer; the third fully connected layer outputs the first evaluation score.

7. The method according to claim 6, characterized in that The second feature extraction module includes P extraction units connected in sequence; the j-th extraction unit among the P extraction units includes a third pooling layer, a seventh convolutional layer, an eighth convolutional layer and a fourth addition layer connected in sequence; P is a positive integer, and j is a positive integer less than or equal to P; The output of the seventh convolutional layer is also connected to the input of the fourth addition layer; When j is equal to 1, the input of the third pooling layer of the j-th extraction unit is connected to the output of the third addition layer; When j is greater than 1, the input of the third pooling layer of the j-th extraction unit is connected to the output of the j-1-th extraction unit.

8. The method according to claim 2, characterized in that The label of the noisy image includes a second evaluation score of the noisy image; the image generation module further outputs a third evaluation score of the generated image; before determining the target loss value based on the standard denoised image, the denoised image, the generated image, the first Gram matrix, the second Gram matrix, and the first evaluation score, the method further includes: Determining a first loss function according to the denoised image and the standard denoised image; Determining a second loss function based on the standard denoised image and the generated image; Determining a third loss function according to the third evaluation score; Determine a fourth loss function according to the first Gram matrix and the second Gram matrix; Determining a fifth loss function according to the first evaluation score and the second evaluation score; Determine a target loss function according to the first loss function, the second loss function, the third loss function, the fourth loss function, and the fifth loss function; Determining the target loss value according to the standard denoised image, the denoised image, the generated image, the first Gram matrix, the second Gram matrix and the first evaluation score includes: determining the target loss value according to the standard denoised image, the denoised image, the generated image, the first Gram matrix, the second Gram matrix and the first evaluation score based on the target loss function.

9. An image denoising method, characterized in that: include: Obtaining a noise image to be processed; Inputting the noisy image into a target image denoising network of an image denoising model for denoising to obtain a target denoised image; The image denoising model is trained according to the image denoising model training method according to any one of claims 1 to 8.

10. The method according to claim 9, characterized in that The target image denoising network includes a first feature extraction module and a post-processing module; the step of inputting the noisy image into the target image denoising network of the image denoising model for denoising to obtain a target denoised image includes: Inputting the noise image into the first feature extraction module for feature extraction processing to obtain a third feature map; The third feature map is input into the post-processing module for smoothing to obtain a target denoised image.

11. A training device for an image denoising model, characterized in that: include: An acquisition module, used for acquiring a plurality of training sample images to be trained; A training module, configured to input the training sample images into the model to be trained for training processing to obtain an image denoising model; In which, the model to be trained includes an image denoising network and an image quality evaluation network, and the training process includes: denoising the training sample image through the image denoising network to obtain a denoised image, and generating a generated image of the training sample image; the generated image is generated based on a first feature map and a second feature map, the first feature map is obtained by performing feature extraction processing on the noise image included in the training sample image, and the second feature map is obtained by performing first image processing on the first feature map; quality evaluation processing is performed on the training sample image through the image quality evaluation network to obtain a first evaluation score of the training sample image; a target loss value is determined based on the training sample image, the denoised image, the generated image and the first evaluation score; and the model parameters of the model to be trained are adjusted based on the target loss value.

12. An image denoising device, characterized in that: include: An acquisition module, used for acquiring a noise image to be processed; A denoising module is used to input the noisy image into a target image denoising network of an image denoising model for denoising to obtain a target denoised image; the image denoising model is trained according to the training method of the image denoising model according to any one of claims 1-8.

13. An electronic device, characterized in that: include: processor; as well as, A memory arranged to store computer-executable instructions, wherein the executable instructions are configured to be executed by the processor, the executable instructions including steps for executing the method for training an image denoising model as described in any one of claims 1 to 8, or the executable instructions including steps for executing the method for image denoising as described in any one of claims 9 to 10.

14. A storage medium, characterized in that The storage medium is used to store computer-executable instructions, which enable the computer to execute the training method of the image denoising model as described in any one of claims 1 to 8, or the executable instructions enable the computer to execute the image denoising method as described in any one of claims 9 to 10.

Images