Method for training image denoising model, image denoising method, apparatus, and device

Abstract

The application discloses a kind of training method of image denoising model, image denoising method, device and equipment, it is related to artificial intelligence technical field.Method includes: obtaining the training sample of image denoising model;Through encoder, the characteristic information of noisy sample image is extracted;Through first decoder, the characteristic information of noisy sample image is processed, and the predicted noise image corresponding to noisy sample image is obtained;According to predicted noise image and noisy sample image, the first predicted denoising image corresponding to noisy sample image is generated;Through second decoder, the characteristic information of noisy sample image is processed, and the second predicted denoising image corresponding to noisy sample image is obtained;According to first predicted denoising image, second predicted denoising image and noiseless sample image, the training loss of image denoising model is determined, and the parameter of image denoising model is adjusted based on training loss.The application scheme can be applied in the scene of image shadow removal of document, PPT (slide) etc..

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to training methods for image denoising models, image denoising methods, apparatus and devices. Background Technology

[0002] When taking pictures, people inevitably encounter some shooting problems, resulting in noise such as shadows and blur in the pictures.

[0003] Taking image shadow removal as an example, a deep learning-based image shadow removal method is provided in related technologies. The shadowed image and its corresponding shadow mask image are input into a trained image shadow removal model, which then outputs a shadow-removed image.

[0004] However, in practical applications, it is usually only possible to obtain the image with shadows, but the shadow mask image corresponding to the image with shadows cannot be obtained. This results in the above method failing to achieve a good shadow removal effect. Summary of the Invention

[0005] This application provides a training method for an image denoising model, an image denoising method, an apparatus, and a device. The technical solution is as follows:

[0006] According to one aspect of the embodiments of this application, a method for training an image denoising model is provided, the image denoising model including an encoder, a first decoder, and a second decoder; the method includes:

[0007] The training samples of the image denoising model are obtained, including noise-free sample images, pure noise images, and noisy sample images generated based on the noise-free sample images and the pure noise images.

[0008] The encoder extracts feature information from the noisy sample image;

[0009] The first decoder processes the feature information of the noisy sample image to obtain a predicted noise image corresponding to the noisy sample image; based on the predicted noise image and the noisy sample image, a first predicted denoised image corresponding to the noisy sample image is generated.

[0010] The second decoder processes the feature information of the noisy sample image to obtain the second predicted denoised image corresponding to the noisy sample image;

[0011] Based on the first predicted denoised image, the second predicted denoised image, and the noise-free sample image, the training loss of the image denoising model is determined, and the parameters of the image denoising model are adjusted based on the training loss.

[0012] According to one aspect of the embodiments of this application, an image denoising method is provided, the method comprising:

[0013] A noisy image to be processed by an image denoising model is obtained, wherein the image denoising model includes an encoder and a first decoder;

[0014] The encoder extracts feature information from the noisy image;

[0015] The first decoder processes the feature information of the noisy image to obtain the noise image corresponding to the noisy image;

[0016] Based on the noisy image and the noisy image, a denoised image corresponding to the noisy image is generated.

[0017] According to one aspect of the embodiments of this application, a training apparatus for an image denoising model is provided, the image denoising model including an encoder, a first decoder, and a second decoder; the apparatus includes:

[0018] The sample acquisition module is used to acquire training samples of the image denoising model. The training samples include noise-free sample images, pure noise images, and noisy sample images generated based on the noise-free sample images and the pure noise images.

[0019] The information extraction module is used to extract feature information from the noisy sample image through the encoder;

[0020] The first processing module is configured to process the feature information of the noisy sample image through the first decoder to obtain a predicted noise image corresponding to the noisy sample image; and generate a first predicted denoising image corresponding to the noisy sample image based on the predicted noise image and the noisy sample image.

[0021] The second processing module is used to process the feature information of the noisy sample image through the second decoder to obtain the second predicted denoised image corresponding to the noisy sample image;

[0022] The parameter adjustment module is used to determine the training loss of the image denoising model based on the first predicted denoised image, the second predicted denoised image, and the noise-free sample image, and to adjust the parameters of the image denoising model based on the training loss.

[0023] According to one aspect of the embodiments of this application, an image denoising apparatus is provided, the apparatus comprising:

[0024] An image acquisition module is used to acquire a noisy image to be processed by an image denoising model, wherein the image denoising model includes an encoder and a first decoder;

[0025] The information extraction module is used to extract feature information of the noisy image through the encoder;

[0026] The information processing module is used to process the feature information of the noisy image through the first decoder to obtain the noise image corresponding to the noisy image;

[0027] The image generation module is used to generate a denoised image corresponding to the noisy image based on the noisy image and the noisy image.

[0028] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the above-described image denoising model training method, or to implement the above-described image denoising method.

[0029] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored in the storage medium, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the above-described image denoising model training method, or to implement the above-described image denoising method.

[0030] According to one aspect of the embodiments of this application, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described image denoising model training method, or to perform the above-described image denoising method.

[0031] The technical solutions provided in this application have at least the following beneficial effects:

[0032] This application provides a training method for an image denoising model based on multi-task learning. On one hand, the image denoising model learns the predicted noisy image corresponding to a noisy sample image; on the other hand, it learns the predicted denoised image corresponding to a noisy sample image. These two sub-tasks mutually promote each other's performance, thereby helping to improve the denoising effect of the finally trained image denoising model. Moreover, the input data of this image denoising model only needs to include the noisy sample image, and does not need to include pure noise images. Therefore, it can overcome the limitations on model input data in related technologies, improving the versatility and practicality of the solution. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a schematic diagram of the implementation environment of a solution provided in one embodiment of this application;

[0035] Figure 2 This is a flowchart of a training method for an image denoising model provided in one embodiment of this application;

[0036] Figure 3 This is a schematic diagram of a shadow decoder provided in one embodiment of this application;

[0037] Figure 4 This is a schematic diagram of a restoration decoder provided in one embodiment of this application;

[0038] Figure 5 This is a flowchart of a training method for an image denoising model provided in another embodiment of this application;

[0039] Figure 6 This is a flowchart of a classifier provided in one embodiment of this application;

[0040] Figure 7 This is a flowchart of a discriminator provided in one embodiment of this application;

[0041] Figure 8 This is a schematic diagram of the components of a discriminator provided in one embodiment of this application;

[0042] Figure 9 This is a schematic diagram of the structure of a discriminator provided in one embodiment of this application;

[0043] Figure 10 This is a flowchart of an image denoising method provided in one embodiment of this application;

[0044] Figure 11 This is a schematic diagram of an image denoising method provided in one embodiment of this application;

[0045] Figure 12 This is a block diagram of a training apparatus for an image denoising model provided in one embodiment of this application;

[0046] Figure 13 This is a block diagram of a training apparatus for an image denoising model provided in another embodiment of this application;

[0047] Figure 14 This is a block diagram of an image denoising apparatus provided in one embodiment of this application;

[0048] Figure 15 This is a block diagram of an image denoising apparatus provided in another embodiment of this application;

[0049] Figure 16 This is a schematic diagram of the structure of a computer device provided in one embodiment of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0051] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0052] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0053] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0054] This application relates to machine learning technology in artificial intelligence learning. It trains an image denoising model through machine learning and uses the image denoising model to denoise the image. The technical solution of this application will be introduced through several embodiments below.

[0055] Please refer to Figure 1 This diagram illustrates an implementation environment for a solution provided in one embodiment of this application. The implementation environment may include a model training device 10 and a model usage device 20.

[0056] The model training device 10 can be an electronic device such as a computer, server, or intelligent robot, or other electronic devices with strong computing power. The model training device 10 is used to train the image denoising model 30. In this embodiment, the image denoising model 30 is a neural network model for image denoising, and the model training device 10 can train the image denoising model 30 to achieve better denoising performance.

[0057] The trained image denoising model 30 can be deployed on the model-using device 20 to provide image denoising results. The model-using device 20 can be a terminal device such as a mobile phone, computer, smart TV, multimedia playback device, wearable device, medical device, or a server; this application does not limit this.

[0058] In some embodiments, such as Figure 1 As shown, the image denoising model 30 may include an encoder 31, a first decoder 32, and a second decoder 33. The encoder 31 extracts feature information from the noisy sample image. The first decoder 32 processes the feature information of the noisy sample image to obtain a predicted noise image corresponding to the noisy sample image. Subsequently, a first predicted denoised image corresponding to the noisy sample image can be generated based on this predicted noise image and the noisy sample image. The second decoder 33 processes the feature information of the noisy sample image to obtain a second predicted denoised image corresponding to the noisy sample image. Then, based on the first predicted denoised image, the second predicted denoised image, and the noise-free sample image, the training loss of the image denoising model 30 is determined, and the parameters of the image denoising model 30 are adjusted based on this training loss.

[0059] The image denoising model training method provided in this application adopts the concept of multi-task learning. Multi-task learning refers to unifying multiple sub-tasks into a single machine learning model for learning. During training, the model can learn multiple sub-tasks simultaneously, and these sub-tasks can share some information from the model and mutually promote each other. In this application embodiment, on one hand, the image denoising model learns the predicted noisy image corresponding to the noisy sample image; on the other hand, the image denoising model learns the predicted denoised image (i.e., the second predicted denoised image hereinafter) corresponding to the noisy sample image. These two sub-tasks mutually promote each other in performance, thereby helping to improve the denoising effect of the finally trained image denoising model.

[0060] Optionally, such as Figure 1As shown, the image denoising model 30 also includes a classifier 34. The classifier 34 processes the feature information of noisy or noise-free sample images to obtain noise prediction results for the noisy or noise-free sample images. These noise prediction results indicate whether the noisy or noise-free sample images contain noise. Based on the two subtasks mentioned above, a further subtask can be added, performing all three subtasks simultaneously during model training. This allows the three subtasks to mutually enhance each other's performance, which helps to further improve the denoising effect of the image denoising model.

[0061] Optionally, in the training process of the image denoising model, embodiments of this application may also introduce the concept of adversarial learning. Adversarial learning introduces a discriminator and a generator. The discriminator distinguishes whether the input sample comes from a sample generated by the generator or a labeled sample. The generator deceives the discriminator into believing that its generated sample is a labeled sample. Through the adversarial learning process between the generator and the discriminator, the generator eventually evolves step by step until the distribution of the generated data approximates the distribution of the labeled data. Optionally, as... Figure 1 As shown, with the introduction of adversarial learning, the image denoising model 30 also includes a discriminator 35. The discriminator 35 is used to perform pixel-level discrimination on the first predicted denoised image and the noise-free sample image respectively, to obtain the pixel-level discrimination results corresponding to the first predicted denoised image and the noise-free sample image respectively. Correspondingly, the generator can be regarded as a combination of the encoder 31 and the first decoder 32. By using adversarial learning for pixel-level discrimination to optimize the details of the denoising results, the denoising effect of the image denoising model can be further improved, and its detail display quality can be enhanced.

[0062] The image denoising model provided in this application embodiment can be applied to the removal of noise information in images such as pictures or videos. Optionally, the noise information includes, but is not limited to, at least one of the following: shadows, rain, snow, fog, blemishes, etc.

[0063] The technical solution of this application will be described below through several method embodiments.

[0064] Please refer to Figure 2 The diagram illustrates a flowchart of a training method for an image denoising model according to an embodiment of this application. The execution entity for each step of this method can be the model training device described above. The method may include the following steps (210-250):

[0065] Step 210: Obtain training samples for the image denoising model. The training samples include noise-free sample images, pure noise images, and noisy sample images generated based on noise-free sample images and pure noise images.

[0066] Image denoising models are neural network models used to remove noise information from images. Noise information in images includes, but is not limited to, at least one of the following: shadows, rain, snow, fog, blemishes, etc. Image denoising models can achieve effects such as removing shadows, rain, snow, fog, and blemishes. Considering the performance of image denoising models, they typically only have the function of removing one type of noise information, such as a certain image denoising model only having the function of removing shadows. However, this application does not limit an image denoising model to simultaneously having the function of removing multiple types of noise information; for example, an image denoising model may have multiple functions such as removing rain, snow, and fog.

[0067] Optionally, the image denoising model includes an encoder, a first decoder (or "noise decoder"), and a second decoder (or "restoration decoder"). The encoder and the first decoder are combined to process the noisy sample image to obtain a predicted noisy image. The encoder and the second decoder are combined to process the noisy sample image to obtain a predicted denoised image (i.e., the "second predicted denoised image" hereinafter).

[0068] Training samples are used to train the image denoising model. The training samples include noise-free sample images, purely noisy images, and noisy sample images generated based on noise-free sample images and purely noisy images. Among them, noise-free sample images refer to images that do not contain noise information, purely noisy images refer to images that contain only noise information, and noisy sample images can be images generated by fusing noise-free sample images and purely noisy images.

[0069] In the case of noise information representing shadows, a noise-free sample image is an image without shadows. A noise-free sample image is obtained by photographing a target object under specific conditions, such as specific angles, lighting, and distances. The target object can be objects such as books, paper, or PowerPoint presentations; this application does not limit this. Optionally, different angles, lighting, and distances, or different target objects, can be adjusted to obtain multiple different noise-free sample images.

[0070] When the noise information is shadow, a pure noise image is an image containing only shadows. A pure noise image is obtained by photographing a blank object under specific conditions, such as a specific angle, lighting, and distance, and generating shadows through natural or constructed occlusion. The blank object can be a blank sheet of paper, a blank wall, a blank curtain, etc., and this application does not limit this. Optionally, different angles, lighting, and distances, or different blank objects, can be adjusted to obtain multiple different pure noise images.

[0071] Optionally, a noisy sample image can be generated by synthesizing any noise-free sample image and any purely noisy image. For example, multiplying the pixel values ​​at corresponding positions in the noise-free sample image and the purely noisy image yields the noisy sample image. Alternatively, adding the pixel values ​​at corresponding positions in the noise-free sample image and the purely noisy image also yields the noisy sample image. In this embodiment, the method of generating the noisy sample image is not limited. By selecting different noise-free sample images and different purely noisy images for synthesis, a large number of noisy sample images can be generated as training samples for training the model.

[0072] Step 220: Extract feature information from noisy sample images using an encoder.

[0073] An encoder is used to extract feature information from noisy sample images. The encoder can be a neural network capable of extracting high-level semantic features as feature information from noisy sample images. Optionally, the encoder includes a downsampling module, a normalization module, and an activation function module. After the noisy sample image is input into the encoder, it is first downsampled by the downsampling module to obtain a downsampling result; then, the normalization module normalizes the downsampling result to obtain a normalized result; finally, the activation function module processes the normalized result, ultimately outputting the feature information of the noisy sample image. The feature information of the noisy sample image can be output in the form of a feature map.

[0074] In some embodiments, such as Figure 3 As shown, Figure 3A schematic diagram of an encoder is illustrated, comprising a downsampling module 31, a normalization module 32, and an activation function module 33. Exemplarily, the encoder section includes eight downsampling modules that extract high-level semantic features from the input noisy sample image in ascending order of quality. Each downsampling module halves the spatial resolution of the input features. After eight downsampling modules, the spatial resolution of the input noisy sample image in terms of high-level semantic features is reduced to 1 / 256 of its original value. Therefore, the spatial resolution of the input noisy sample image is required to be an integer multiple of 256. Of course, the number of downsampling modules can be flexibly adjusted according to the actual task, and this application does not limit this. Figure 3 As shown, the downsampling process of downsampling module 31 is implemented using a convolutional neural network, for example, using convolution with a stride of 2 instead of the traditional pooling process with a kernel size of 2. After convolution, the downsampling result output by downsampling module 31 is processed by batch normalization (BN) by normalization module 32 to obtain the normalized result. Then, the normalized result is processed by activation function module 33, such as using Leaky ReLU with a parameter of 0.2 as the activation function to obtain the feature information of the noisy sample image. In the encoder, using convolution operation instead of pooling operation for downsampling can reduce information loss during the downsampling process, consider image information at each position as comprehensively as possible, and help improve the accuracy of the final feature information.

[0075] Step 230: The feature information of the noisy sample image is processed by the first decoder to obtain the predicted noise image corresponding to the noisy sample image; based on the predicted noise image and the noisy sample image, the first predicted denoising image corresponding to the noisy sample image is generated.

[0076] The first decoder, also known as the noise decoder, is used to predict the noise information contained in the input image, that is, to predict the noise image corresponding to the noisy sample image. Then, based on the noisy sample image and the predicted noise image, a first predicted denoised image can be obtained, wherein the resolution of the first predicted denoised image is the same as the resolution of the noisy sample image.

[0077] The first decoder can be a neural network. Optionally, the first decoder includes an upsampling module, a normalization module, and an activation function module. After the feature information of the noisy sample image is input into the first decoder, the upsampling module upsamples the feature information of the noisy sample image to obtain an upsampling result; then, the normalization module normalizes the upsampling result to obtain a normalized result; finally, the activation function module processes the normalized result to finally output the predicted noise image corresponding to the noisy sample image.

[0078] In some embodiments, such as Figure 4 As shown, Figure 4 An exemplary schematic diagram of a noise decoder is shown, which includes an upsampling module 41, a normalization module 42, and an activation function module 43. Exemplarily, the noise decoder comprises eight upsampling modules that progressively restore the spatial resolution loss caused by the encoder, ultimately obtaining an output with the same spatial resolution as the input noisy sample image. Figure 4 As shown, the upsampling process of the upsampling module 41 is implemented using a deconvolutional neural network, for example, using learnable parameter deconvolution instead of the traditional bilinear interpolation method. After deconvolution, the normalization module 42 performs Batch Normalization (BN) on the upsampling result output by the upsampling module 41 to obtain the normalized result. Then, the activation function module 43 processes the normalized result, such as using ReLU as the activation function, to obtain the predicted noise image corresponding to the noisy sample image. In the decoder, learning parameter deconvolution is used instead of the traditional bilinear interpolation method. On the one hand, learning parameter deconvolution has stronger representation capabilities, thus improving the representation performance of the decoder. On the other hand, deconvolution processing has lower complexity than bilinear interpolation, thus reducing computational complexity.

[0079] After obtaining the predicted noise image corresponding to the noisy sample image, a first predicted denoised image is obtained based on the predicted noise image and the noisy sample image. Optionally, the generation method of the first predicted denoised image corresponds to the generation method of the noisy sample image. For example, if the noisy sample image is obtained by multiplying the pixel values ​​at corresponding positions in the noise-free sample image and the pure noise image, then the noisy sample image needs to be divided by the predicted noise image to obtain the first predicted denoised image. As another example, if the noisy sample image is obtained by adding the pixel values ​​at corresponding positions in the noise-free sample image and the pure noise image, then the predicted noise image needs to be subtracted from the noisy sample image to obtain the first predicted denoised image.

[0080] Step 240: The feature information of the noisy sample image is processed by the second decoder to obtain the second predicted denoised image corresponding to the noisy sample image.

[0081] The second decoder, also known as the restoration decoder, is used to remove noise information from the noisy sample image to obtain a second predicted denoised image. Optionally, the structure of the second decoder is the same as that of the first decoder. The resolution of the second predicted denoised image is the same as that of the noisy sample image.

[0082] The second decoder can be a neural network. Optionally, the second decoder includes an upsampling module, a normalization module, and an activation function module. After the feature information of the noisy sample image is input into the second decoder, the upsampling module upsamples the feature information of the noisy sample image to obtain an upsampling result; then, the normalization module normalizes the upsampling result to obtain a normalized result; finally, the activation function module processes the normalized result to finally output the second predicted denoised image corresponding to the noisy sample image.

[0083] In some embodiments, the structure of the restoration decoder is the same as that of the noise decoder, so as to... Figure 4 Taking the structure as an example, Figure 4 A schematic diagram of a restoration decoder is shown as an example. The restoration decoder includes an upsampling module 41, a normalization module 42, and an activation function module 43. For example, the restoration decoder contains eight upsampling modules that progressively restore the spatial resolution loss caused by the encoder, ultimately obtaining an output with the same spatial resolution as the input noisy sample image. Figure 4 As shown, the upsampling process of the upsampling module 41 is implemented using a deconvolutional neural network, for example, using learnable deconvolution parameters instead of the traditional bilinear interpolation method. After deconvolution, the normalization module 42 performs Batch Normalization (BN) on the upsampling result output by the upsampling module 41 to obtain the normalized result. Then, the activation function module 43 processes the normalized result, such as using ReLU as the activation function, to obtain the second predicted denoised image corresponding to the noisy sample image. Furthermore, although the restoration decoder and the noise decoder have the same structure, they do not share parameters and learn independently. The restoration decoder is an end-to-end neural network capable of directly outputting the denoised image.

[0084] Step 250: Based on the first predicted denoised image, the second predicted denoised image, and the noise-free sample image, determine the training loss of the image denoising model, and adjust the parameters of the image denoising model based on the training loss.

[0085] Optionally, training losses for the encoder and the first decoder are determined based on the first predicted denoised image and the noise-free sample image, and the parameters of the encoder and the first decoder are adjusted based on the training losses. Optionally, training losses for the encoder and the second decoder are determined based on the second predicted denoised image and the noise-free sample image, and the parameters of the encoder and the second decoder are adjusted based on the training losses. The training loss is used to measure the difference between the predicted denoised image obtained by the image denoising model and the noise-free sample image.

[0086] This application provides a training method for an image denoising model based on multi-task learning. On one hand, the image denoising model learns the predicted noisy image corresponding to a noisy sample image; on the other hand, it learns the predicted denoised image corresponding to a noisy sample image. These two sub-tasks mutually promote each other's performance, thereby helping to improve the denoising effect of the finally trained image denoising model. Moreover, the input data of this image denoising model only needs to include the noisy sample image, and does not need to include pure noise images. Therefore, it can overcome the limitations on model input data in related technologies, improving the versatility and practicality of the solution.

[0087] Please refer to Figure 5 This document illustrates a flowchart of a training method for an image denoising model according to another embodiment of this application. The execution entity for each step of this method can be the model training device described above. The method may include the following steps (510-580). Some of these steps in this embodiment have already been described above and will not be repeated here.

[0088] Step 510: Obtain training samples for the image denoising model. The training samples include noise-free sample images, pure noise images, and noisy sample images generated based on noise-free sample images and pure noise images.

[0089] Step 520: Extract feature information from the noisy sample image using the encoder.

[0090] Step 530: The feature information of the noisy sample image is processed by the first decoder to obtain the predicted noise image corresponding to the noisy sample image; based on the predicted noise image and the noisy sample image, the first predicted denoising image corresponding to the noisy sample image is generated.

[0091] Optionally, generating the first predicted denoised image corresponding to the noisy sample image includes: dividing the values ​​of the corresponding pixels in the noisy sample image and the predicted noise image to obtain the first predicted denoised image; or, subtracting the values ​​of the corresponding pixels in the noisy sample image and the predicted noise image to obtain the first predicted denoised image.

[0092] In practice, the generation method of the first predicted denoised image is determined based on how the noisy sample image was generated. For example, if the noisy sample image is obtained by multiplying the pixel values ​​at corresponding positions in the noise-free sample image and the pure noise image, the pixel values ​​at corresponding positions in the noisy sample image and the predicted noise image are divided to obtain the first predicted denoised image. If the noisy sample image is obtained by adding the pixel values ​​at corresponding positions in the noise-free sample image and the pure noise image, the pixel values ​​at corresponding positions in the noisy sample image and the predicted noise image are subtracted to obtain the first predicted denoised image.

[0093] In some embodiments, when the predicted noisy sample image is obtained by multiplying the values ​​of corresponding pixels in the noise-free sample image and the pure noise image, the first predicted denoised image is obtained by dividing the value of a pixel in the predicted noisy sample image by the value of a pixel in the predicted noise image. The formula for the first predicted denoised image can be expressed as follows:

[0094]

[0095] Here, Input is the input to the entire network model, which is the predicted noisy sample image. Noise is the output of the noise decoder, which is the predicted noisy image. ε is a very small constant to avoid the denominator being zero in the above formula. By adding a very small constant ε, the feasibility of the solution is ensured without affecting the result.

[0096] In some embodiments, when the predicted noisy sample image is obtained by adding the values ​​of corresponding pixels in the noise-free sample image and the pure noise image, the first predicted denoised image is obtained by subtracting the values ​​of corresponding pixels in the predicted noise image from the values ​​of pixels in the predicted noisy sample image. The formula for the first predicted denoised image can be expressed as follows:

[0097] Input-Noise

[0098] Here, Input is the input to the entire network model, which is the predicted noisy sample image. Noise is the output of the noise decoder, which is the predicted noisy image.

[0099] Step 540: Determine the first training loss based on the first predicted denoised image and the noise-free sample image.

[0100] The first training loss is used to measure the difference between the first predicted denoised image and the noise-free sample image.

[0101] Optionally, step 540 includes the following sub-steps (1-2):

[0102] 1. Based on the first predicted denoised image and the noise-free sample image, determine the first pixel-level loss, the first semantic feature loss, and the first image gradient loss.

[0103] The first pixel-level loss is used to measure the difference between pixels in the first predicted denoised image and pixels in the noise-free sample image. For example, for corresponding pixels in the first predicted denoised image and the noise-free sample image, the distance between the values ​​of these two pixels is calculated, and then the distance values ​​of all corresponding pixels are summed to obtain the first pixel-level loss. The first pixel-level loss can be calculated using L1 distance, cosine distance, Euclidean distance, etc., and this application does not limit it to this method.

[0104] The first semantic feature loss measures the difference between the semantic features of the first predicted denoised image and the semantic features of the noise-free sample image. These semantic features refer to high-level semantic features in the image. For example, a pre-trained feature extractor can be used to extract the semantic features of both the first predicted denoised image and the noise-free sample image, and then the distance between these two semantic features can be calculated to obtain the first semantic feature loss. The semantic feature loss can also be called perceptual loss.

[0105] The first image gradient loss measures the difference between the image gradient of the first predicted denoised image and the image gradient of the noise-free sample image. It is calculated by separately calculating the image gradients of the first predicted denoised image and the noise-free sample image, and then calculating the difference between these two gradients. Image gradient loss can also be called gradient loss.

[0106] 2. Determine the first training loss based on the first pixel-level loss, the first semantic feature loss, and the first image gradient loss.

[0107] Optionally, the first pixel-level loss, the first semantic feature loss, and the first image gradient loss are weighted and summed to obtain the first training loss. In this embodiment, the first training loss is calculated based on the first pixel-level loss, the first semantic feature loss, and the first image gradient loss, taking into account various differences between images, such as pixel differences, semantic feature differences, and gradient differences, making the result of the first training loss more accurate, and further making the obtained image denoising model more accurate.

[0108] Step 550: Adjust the parameters of the encoder and the first decoder based on the first training loss.

[0109] Optionally, a gradient descent algorithm based on backpropagation is used to adjust the parameters of the encoder and the first decoder based on the first training loss.

[0110] Step 560: The feature information of the noisy sample image is processed by the second decoder to obtain the second predicted denoised image corresponding to the noisy sample image.

[0111] Step 570: Determine the second training loss based on the second predicted denoised image and the noise-free sample image.

[0112] The second training loss is used to measure the difference between the second predicted denoised image and the noise-free sample image.

[0113] Optionally, step 570 includes the following sub-steps (1-2):

[0114] 1. Based on the second predicted denoised image and the noise-free sample image, determine the second pixel-level loss, the second semantic feature loss, and the second image gradient loss.

[0115] Specifically, the second pixel-level loss is used to measure the difference between the pixels of the second predicted denoised image and the pixels of the noise-free sample image; the second semantic feature loss is used to measure the difference between the semantic features of the second predicted denoised image and the semantic features of the noise-free sample image; and the second image gradient loss is used to measure the difference between the image gradient of the second predicted denoised image and the image gradient of the noise-free sample image.

[0116] In addition, the calculation methods for the second pixel-level loss, the second semantic feature loss, and the second image gradient loss can refer to the calculation methods for the first pixel-level loss, the first semantic feature loss, and the first image gradient loss introduced above, and will not be repeated here.

[0117] 2. Determine the second training loss based on the second pixel-level loss, the second semantic feature loss, and the second image gradient loss.

[0118] Optionally, the second pixel-level loss, the second semantic feature loss, and the second image gradient loss are weighted and summed to obtain the second training loss. In this embodiment, the second training loss is calculated based on the second pixel-level loss, the second semantic feature loss, and the second image gradient loss, taking into account various differences between images, such as pixel differences, semantic feature differences, and gradient differences, making the result of the second training loss more accurate, and further making the obtained image denoising model more accurate.

[0119] Step 580: Adjust the parameters of the encoder and the second decoder based on the second training loss.

[0120] In addition, in some exemplary embodiments, the same data augmentation processing can be performed on both noisy sample images and noise-free sample images. This data augmentation processing includes, but is not limited to, at least one of the following: random scaling, random cropping, random horizontal flipping, random vertical flipping, random rotation, affine transformation, translation, scaling, and rotation, and perspective transformation. The noisy sample images after the above data augmentation processing are input into the encoder for feature extraction. By performing data augmentation processing on the images, the model's ability to learn features from different images can be improved, making the finally trained model more robust to different images.

[0121] This embodiment separates the process of adjusting the parameters of the image denoising model using the first training loss and the second training loss, thus making the training process of the noise decoder and the training process of the restoration decoder independent, achieving simultaneous operation of multiple tasks and saving training time.

[0122] Meanwhile, the training loss is obtained through multiple metrics (including pixel-level loss, semantic feature loss, and image gradient loss), and the parameters of the image denoising model are adjusted using this training loss. By generating the training loss through multiple metrics, the parameter adjustment of the image denoising model becomes more accurate.

[0123] Based on the above Figure 5 Based on the training method of the image denoising model provided in the embodiment, such as Figure 6 As shown, the training method provided in another exemplary embodiment of this application may further include the following steps (610-630):

[0124] Step 610: The feature information of the noisy sample image or the noise-free sample image is processed by the classifier to obtain the noise prediction result of the noisy sample image or the noise-free sample image. The noise prediction result is used to indicate whether the noisy sample image or the noise-free sample image contains noise.

[0125] A classifier is used to determine whether an input image contains noise. The input image can be a noisy sample image or a noise-free sample image. The classifier's output is the probability that the input image contains noise, which is the noise prediction result. For example, when a noisy sample image is input, the classifier outputs a noise prediction result of 90%; when a noise-free sample image is input, the classifier outputs a noise prediction result of 10%. The value of the classifier's output result can be a value within the closed interval [0,1].

[0126] Step 620: Determine the third training loss based on the noise prediction results and the actual noise results. The third training loss is used to measure the difference between the noise prediction results and the actual noise results.

[0127] The third training loss is derived based on the difference between the noise prediction result output by the classifier and the actual noise result of the input image.

[0128] Step 630: Adjust the parameters of the encoder and classifier based on the third training loss.

[0129] In this embodiment, a classifier is further introduced on the basis of the first decoder and the second decoder, so that three sub-tasks are performed simultaneously during the model training process, allowing the three sub-tasks to promote each other in performance, which helps to further improve the denoising effect of the image denoising model.

[0130] Meanwhile, by introducing a classifier, the system determines whether the input image is noisy based on the classifier's assessment of noise levels, thus deciding whether denoising is required through the first decoder branch. This eliminates the unnecessary operation of denoising noise-free images by the image denoising model, saving time and cost in image denoising.

[0131] Based on the training method of the image denoising model provided in any of the above embodiments, adversarial learning can be further introduced to optimize model performance on the basis of multi-task learning. For example... Figure 7 As shown, the training method provided in another exemplary embodiment of this application may further include the following steps (710-720):

[0132] Step 710: The first predicted denoised image and the noise-free sample image are distinguished at the pixel level by the discriminator to obtain the pixel-level discrimination results corresponding to the first predicted denoised image and the noise-free sample image respectively.

[0133] The discriminator performs pixel-level judgment on whether the input image has undergone model denoising. This pixel-level judgment assesses each pixel in the input image. The input image includes a first predicted denoised image and a noise-free sample image. The output is the probability of whether the input image has undergone model denoising. For example, when the input is a first predicted denoised image containing 10,000 pixels, the discriminator outputs the prediction results for those 10,000 pixels, where the prediction result for each pixel indicates whether it was obtained after model denoising. For example, the prediction result can be any probability value between [0,1].

[0134] Optionally, the discriminator includes a feature mapping layer, n cascaded downsampling layers, n cascaded upsampling layers, and a feature extraction layer, where n is an integer greater than 1. Step 610 includes the following steps (1-4):

[0135] 1. The first predicted denoised image or noise-free sample image is processed by the feature mapping layer to obtain the initial feature map of the first predicted denoised image or noise-free sample image.

[0136] The feature mapping layer first enlarges the spatial resolution of the input image to obtain the initial feature map of the input image.

[0137] In some embodiments, such as Figure 8 As shown, Figure 8 A schematic diagram of each layer in the discriminator is illustrated. The feature mapping layer 81 includes a convolution module 81a and an activation function module 81b. The input image first passes through the convolution module 81a to obtain a convolution result map, and then the convolution result map is input into the activation function module 82b to obtain the initial feature map of the input image.

[0138] 2. The initial feature map is downsampled in n steps by n cascaded downsampling layers to obtain the downsampled feature map.

[0139] The initial feature map is downsampled multiple times through multiple downsampling layers to obtain a downsampled feature map with reduced spatial resolution.

[0140] In some embodiments, such as Figure 8 As shown. The downsampling layer 82 includes a downsampling module 82a, a normalization module 82b, and an activation function module 82c. The initial feature map of the input image is first input into the downsampling module 82a to obtain the downsampling result map. Then, the downsampling result map is input into the normalization module 82b to obtain the normalized result map. Finally, the normalized result map is input into the activation function module 82c to obtain the downsampling feature map of the input image output by the encoder. Optionally, the downsampling process in the downsampling module uses a convolution with a stride of 2, uses Spectral Normalization (SN) as the normalization method, and uses Leaky ReLU as the activation function.

[0141] 3. The downsampled feature map is processed through n cascaded upsampling layers to obtain an upsampled feature map; the downsampled layers and the upsampled layers are connected by skip connections.

[0142] The obtained downsampled feature map is upsampled multiple times through multiple upsampling layers to obtain an upsampled feature map with increased spatial resolution.

[0143] In some embodiments, such as Figure 8 As shown, the upsampling layer 83 includes: a type adjustment module 83a, an upsampling module 83b, a normalization module 83c, and an activation function module 83d. The downsampled feature map of the input image first passes through the type adjustment module 83a to obtain a type-adjusted result map. Then, the type-adjusted result map is input into the upsampling module 83b to obtain an upsampled result map. Next, the upsampled result map is input into the normalization module 83c to obtain a normalized result map. Finally, the normalized result map is input into the activation function module 83d to obtain the upsampled feature map of the input image output by the encoder. Optionally, the upsampling process in the upsampling module uses a convolution with a stride of 1, spectral normalization as the normalization method, and LeakyReLU as the activation function.

[0144] Optionally, the above-described upsampling and downsampling layers may include multiple layers. In some embodiments, such as... Figure 8 As shown, Figure 9 A schematic diagram of the overall structure of the discriminator is shown. Optionally, in Figure 9In the diagram, the discriminator consists of a feature mapping layer, three downsampling layers, three upsampling layers, and a feature extraction layer. It can be seen that the downsampling and upsampling layers are connected via skip connections, i.e., as shown... Figure 9 As shown, for upsampling layer 2 91, it is controlled not only by upsampling layer 3 92, but also by downsampling layer 2 93. Similarly, for upsampling layer 1 94, it is controlled not only by upsampling layer 2 91, but also by downsampling layer 1 95. Likewise, during the image denoising process using the shadow decoder or restoration decoder in steps 220 to 240, the upsampling and downsampling layers are also connected in a skip connection.

[0145] 4. The upsampled feature map is processed by the feature extraction layer to obtain the pixel-level discrimination result of the first predicted denoised image or noise-free sample image.

[0146] The pixel values ​​of the upsampled feature map are judged by the feature extraction layer to obtain pixel-level discrimination results: the image obtained after denoising and the image obtained without denoising.

[0147] In some embodiments, such as Figure 8 As shown. The feature extraction layer 84 includes two downsampling layers with a stride of 1 and a downsampling module 84a with a stride of 1. The upsampled feature map of the input image first passes through the downsampling module 84a to obtain the downsampled result map. Then, the downsampled result map is input into the normalization module 84b to obtain the normalized result map. Finally, the normalized result map is input into the activation function module 84c to obtain the secondary downsampled feature map of the input image output by the encoder. After performing the above downsampling layer with a stride of 1 again, the obtained image is input into the downsampling module 84d with a stride of 1 to obtain the pixel-level discrimination result of the image. Optionally, the downsampling process in the downsampling module uses a convolution with a stride of 1, uses spectral normalization as the normalization method, and uses Leaky ReLU as the activation function.

[0148] Step 720: Based on the pixel-level discrimination results, determine the discrimination loss corresponding to the discriminator. The discrimination loss is used to measure the accuracy of the pixel-level discrimination results output by the discriminator.

[0149] The discriminant loss is used to adjust the parameters of the discriminator, and also to adjust the parameters of the encoder and the first decoder. Optionally, the parameters of the encoder and the first decoder are adjusted based on a combination of the discriminant loss and the aforementioned first training loss.

[0150] Based on the pixel-level discrimination results and the actual discrimination results of the image, the discrimination results of the discriminator are obtained: the correct probability and / or the incorrect probability. The correct probability is the accuracy of the discriminator's discrimination results, and the incorrect probability is the error rate of the discriminator's discrimination results. The sum of the correct probability and the incorrect probability of the same discriminator is 1.

[0151] This embodiment introduces the concept of generative adversarial learning on the basis of multi-task learning to improve the denoising effect of the image denoising generation model.

[0152] The discrimination methods mentioned above are all pixel-level discrimination methods. Optionally, global discrimination methods can also be used. Compared with global discrimination methods, pixel-level discrimination methods yield more accurate results and can train a more accurate discriminator. Furthermore, since the discriminator and the image denoising model employ adversarial learning to optimize model performance, a more accurate discrimination will result in a more accurate image denoising model.

[0153] The training process of the image denoising model has been introduced above. The following section describes how to use the same model. It should be noted that the training and usage processes are corresponding; for details not explained in one section, please refer to the description in the other.

[0154] Please refer to Figure 9 The diagram illustrates a flowchart of an image denoising method according to an embodiment of this application. The execution entity for each step of this method can be the model-using device described above. The method may include the following steps (910-940):

[0155] Step 910: Obtain the noisy image to be processed by the image denoising model. The image denoising model includes an encoder and a first decoder.

[0156] Step 920: Extract feature information of the noisy image using the encoder.

[0157] Optionally, the feature information of the noisy image is processed by a classifier to obtain a noise prediction result for the noisy image. The noise prediction result indicates whether the noisy image contains noise. If the noise prediction result indicates that the noisy image contains noise, the step of processing the feature information of the noisy image through a first decoder to obtain the corresponding noise image is performed, i.e., step 930 below. If the noise prediction result indicates that the noisy image does not contain noise, the image denoising model does not denoise the input noisy image (or input image) and directly outputs it or performs subsequent processing. Optionally, if the noise prediction result indicates that the noisy image contains noise, the image denoising model performs the step of processing the feature information of the noisy image through a first decoder to obtain the corresponding noise image.

[0158] Step 930: The feature information of the noisy image is processed by the first decoder to obtain the noise image corresponding to the noisy image.

[0159] The trained image denoising model uses the first decoder, also known as the denoising decoder, to denoise the image. Optionally, the trained image denoising model can also use the second decoder, also known as the restoration decoder, to denoise the image.

[0160] Optionally, the encoder includes k cascaded downsampling layers, and the first decoder includes k cascaded upsampling layers, where k is an integer greater than 1; step 930 includes: performing k-step upsampling processing on the feature information of the noisy image through the k cascaded upsampling layers to obtain the noisy image; wherein, the downsampling layer and the upsampling layer are connected by skip connections.

[0161] Step 940: Generate a denoised image corresponding to the noisy image based on the noisy image and the noisy image.

[0162] Optionally, step 940 includes: dividing the values ​​of corresponding pixels in the noisy image and the noise image to obtain a first predicted denoised image; or, subtracting the values ​​of corresponding pixels in the noisy image and the noise image to obtain a first predicted denoised image.

[0163] Optionally, the noisy image undergoes a first size adjustment process to obtain an adjusted noisy image; wherein the adjusted noisy image is input to an encoder for feature extraction; the noisy image undergoes a second size adjustment process to obtain an adjusted noisy image; wherein the second size adjustment process is the inverse adjustment process corresponding to the first size adjustment process. Generating a denoised image corresponding to the noisy image based on the noisy image and the noisy image includes: generating a denoised image corresponding to the noisy image based on the adjusted noisy image and the noisy image.

[0164] This embodiment uses the same image denoising method as the shadow decoder part of the image denoising model, and will not be described again here.

[0165] In this embodiment, the trained first decoder is used for image denoising, while the trained second decoder is not. This is because the first decoder outputs a predicted noisy image, while the second decoder outputs a second predicted denoised image. In the previous steps, resizing the decoder output image is necessary to obtain the final predicted denoised image, which significantly impacts the accuracy of the output image. The first decoder, however, outputs a predicted noisy image, which is not resized when processed with the noisy image. Therefore, in the image denoising process using the trained first decoder, the final result is calculated from the predicted noisy image and the original noisy image. In this case, only the predicted noisy image is resized; the original noisy image is not. Therefore, the denoised image portion in the final result is unresized. In contrast, in the image denoising model using the second decoder, the output is the second predicted denoised image, which is resized (using an interpolation algorithm) to obtain the final result. Therefore, compared to the second decoder, the first decoder generates denoised images that retain better details and is more suitable for use in image denoising models to denoise images.

[0166] Meanwhile, by introducing a classifier, the system determines whether the input image is noisy based on the classifier's assessment of noise levels, thus eliminating the unnecessary operation of denoising noise-free images by eliminating the need for the first decoder branch. This saves time and cost in image denoising.

[0167] The following are embodiments of the apparatus of this application. For details not described in the embodiments of the apparatus of this application, please refer to the embodiments of the method of this application.

[0168] Please refer to Figure 12 This diagram illustrates a block diagram of a training apparatus for an image denoising model according to an embodiment of this application. The apparatus has the function of implementing the training method for the aforementioned image denoising model. This function can be implemented in hardware or by hardware executing corresponding software. The apparatus 1200 can be the model training device described above, or it can be located within a model training device. The apparatus 1200 may include: a sample acquisition module 1210, an information extraction module 1220, a first processing module 1230, a second processing module 1240, and a parameter adjustment module 1250.

[0169] The sample acquisition module 1210 is used to acquire training samples of the image denoising model. The training samples include noise-free sample images, pure noise images, and noisy sample images generated based on the noise-free sample images and the pure noise images.

[0170] The information extraction module 1220 is used to extract feature information of the noisy sample image through the encoder.

[0171] The first processing module 1230 is used to process the feature information of the noisy sample image through the first decoder to obtain a predicted noise image corresponding to the noisy sample image; and to generate a first predicted denoised image corresponding to the noisy sample image based on the predicted noise image and the noisy sample image.

[0172] The second processing module 1240 is used to process the feature information of the noisy sample image through the second decoder to obtain the second predicted denoised image corresponding to the noisy sample image.

[0173] The parameter adjustment module 1250 is used to determine the training loss of the image denoising model based on the first predicted denoised image, the second predicted denoised image and the noise-free sample image, and to adjust the parameters of the image denoising model based on the training loss.

[0174] In some embodiments, the parameter adjustment module 1250 is used for:

[0175] Based on the first predicted denoised image and the noise-free sample image, a first training loss is determined, which is used to measure the difference between the first predicted denoised image and the noise-free sample image.

[0176] Based on the first training loss, adjust the parameters of the encoder and the first decoder.

[0177] In some embodiments, the parameter adjustment module 1250 is used for:

[0178] Based on the first predicted denoised image and the noise-free sample image, a first pixel-level loss, a first semantic feature loss, and a first image gradient loss are determined; wherein, the first pixel-level loss is used to measure the difference between the pixels of the first predicted denoised image and the pixels of the noise-free sample image, the first semantic feature loss is used to measure the difference between the semantic features of the first predicted denoised image and the semantic features of the noise-free sample image, and the first image gradient loss is used to measure the difference between the image gradient of the first predicted denoised image and the image gradient of the noise-free sample image;

[0179] The first training loss is determined based on the first pixel-level loss, the first semantic feature loss, and the first image gradient loss.

[0180] In some embodiments, the parameter adjustment module 1250 is used for:

[0181] A second training loss is determined based on the second predicted denoised image and the noise-free sample image, the second training loss being used to measure the difference between the second predicted denoised image and the noise-free sample image;

[0182] Based on the second training loss, the parameters of the encoder and the second decoder are adjusted.

[0183] In some embodiments, the parameter adjustment module 1250 is used for:

[0184] Based on the second predicted denoised image and the noise-free sample image, a second pixel-level loss, a second semantic feature loss, and a second image gradient loss are determined; wherein, the second pixel-level loss is used to measure the difference between the pixels of the second predicted denoised image and the pixels of the noise-free sample image, the second semantic feature loss is used to measure the difference between the semantic features of the second predicted denoised image and the semantic features of the noise-free sample image, and the second image gradient loss is used to measure the difference between the image gradient of the second predicted denoised image and the image gradient of the noise-free sample image;

[0185] The second training loss is determined based on the second pixel-level loss, the second semantic feature loss, and the second image gradient loss.

[0186] In some embodiments, the image denoising model further includes a discriminator. For example... Figure 13 As shown, the device 1200 further includes a result acquisition module 1260.

[0187] The result acquisition module 1260 is used to perform pixel-level discrimination on the first predicted denoised image and the noise-free sample image respectively through the discriminator, and obtain the pixel-level discrimination results corresponding to the first predicted denoised image and the noise-free sample image respectively.

[0188] The parameter adjustment module 1250 is further configured to determine the discrimination loss corresponding to the discriminator based on the pixel-level discrimination result, wherein the discrimination loss is used to measure the accuracy of the pixel-level discrimination result output by the discriminator; wherein the discrimination loss is used to adjust the parameters of the discriminator, and is also used to adjust the parameters of the encoder and the first decoder.

[0189] In some embodiments, the discriminator includes a feature mapping layer, n cascaded downsampling layers, n cascaded upsampling layers, and a feature extraction layer, where n is an integer greater than 1. The result acquisition module 1260 is used for:

[0190] The first predicted denoised image or the noise-free sample image is processed by the feature mapping layer to obtain the initial feature map of the first predicted denoised image or the noise-free sample image.

[0191] The initial feature map is downsampled through n steps by the n cascaded downsampling layers to obtain a downsampled feature map;

[0192] The downsampled feature map is processed through n upsampling steps by the n cascaded upsampling layers to obtain the upsampled feature map; wherein, the downsampling layer and the upsampling layer are connected by skip connections;

[0193] The upsampled feature map is processed by the feature extraction layer to obtain the pixel-level discrimination result of the first predicted denoised image or the noise-free sample image.

[0194] In some embodiments, the image denoising model further includes a classifier. For example... Figure 13 As shown, the device 1200 also includes a noise prediction module 1270.

[0195] The noise prediction module 1270 is used to process the feature information of the noisy sample image or the noise-free sample image through the classifier to obtain the noise prediction result of the noisy sample image or the noise-free sample image. The noise prediction result is used to indicate whether the noisy sample image or the noise-free sample image contains noise.

[0196] The parameter adjustment module 1250 is further configured to determine a third training loss based on the noise prediction result and the actual noise result, the third training loss being used to measure the difference between the noise prediction result and the actual noise result; and to adjust the parameters of the encoder and the classifier based on the third training loss.

[0197] In some embodiments, the first processing module 1230 is configured to: divide the values ​​of corresponding pixels in the noisy sample image and the predicted noise image to obtain the first predicted denoised image; or, subtract the values ​​of corresponding pixels in the noisy sample image and the predicted noise image to obtain the first predicted denoised image.

[0198] In some embodiments, such as Figure 13 As shown, the device 1200 also includes a data enhancement module (not shown in the figure).

[0199] The data augmentation module is used to perform the same data augmentation processing on the noisy sample image and the noise-free sample image. The data augmentation processing includes at least one of the following: random scaling, random cropping, random horizontal flipping, random vertical flipping, random rotation, affine transformation, translation, scaling and rotation, and perspective transformation. The noisy sample image or the noise-free sample image after the data augmentation processing is input to the encoder for feature extraction.

[0200] This application provides a training method for an image denoising model based on multi-task learning. On one hand, the image denoising model learns the predicted noisy image corresponding to a noisy sample image; on the other hand, it learns the predicted denoised image corresponding to a noisy sample image. These two sub-tasks mutually promote each other's performance, thereby helping to improve the denoising effect of the finally trained image denoising model. Moreover, the input data of this image denoising model only needs to include the noisy sample image, and does not need to include pure noise images. Therefore, it can overcome the limitations on model input data in related technologies, improving the versatility and practicality of the solution.

[0201] Please refer to Figure 14 This diagram illustrates a block diagram of an image denoising apparatus according to an embodiment of this application. The apparatus has the function of implementing the image denoising method described above; this function can be implemented in hardware or by hardware executing corresponding software. The apparatus can be the model-using device described above, or it can be installed within a model-using device. The apparatus 1400 may include: an image acquisition module 1410, an information extraction module 1420, an information processing module 1430, and an image generation module 1440.

[0202] The image acquisition module 1410 is used to acquire a noisy image to be processed by the image denoising model, wherein the image denoising model includes an encoder and a first decoder.

[0203] The information extraction module 1420 is used to extract feature information of the noisy image through the encoder.

[0204] The information processing module 1430 is used to process the feature information of the noisy image through the first decoder to obtain the noise image corresponding to the noisy image.

[0205] The image generation module 1440 is used to generate a denoised image corresponding to the noisy image based on the noisy image and the noisy image.

[0206] In some embodiments, the image generation module 1440 is configured to: divide the values ​​of corresponding pixels in the noisy image and the noise image to obtain the first predicted denoised image; or, subtract the values ​​of corresponding pixels in the noisy image and the noise image to obtain the first predicted denoised image.

[0207] In some embodiments, the encoder includes k cascaded downsampling layers, and the first decoder includes k cascaded upsampling layers, where k is an integer greater than 1. The information processing module 1430 is configured to: perform k-step upsampling processing on the feature information of the noisy image through the k cascaded upsampling layers to obtain the noisy image; wherein the downsampling layers and the upsampling layers are connected by skip connections.

[0208] In some embodiments, the image denoising model further includes a classifier. For example... Figure 15 As shown, the device 1400 further includes a noise prediction module 1450, which is used to process the feature information of the noisy image through the classifier to obtain a noise prediction result of the noisy image. The noise prediction result is used to indicate whether the noisy image contains noise. If the noise prediction result indicates that the noisy image contains noise, the step of processing the feature information of the noisy image through the first decoder to obtain the noise image corresponding to the noisy image is performed.

[0209] In some embodiments, such as Figure 15 As shown, the device 1400 further includes a size adjustment module 1460, which is used to perform a first size adjustment process on the noisy image to obtain an adjusted noisy image; wherein, the adjusted noisy image is input to the encoder for feature extraction; and to perform a second size adjustment process on the noisy image to obtain an adjusted noisy image; wherein, the second size adjustment process is the reverse adjustment process corresponding to the first size adjustment process.

[0210] The image generation module 1440 is further configured to generate a denoised image corresponding to the noisy image based on the adjusted noisy image and the noisy image.

[0211] In this embodiment, the trained first decoder is used for image denoising, while the trained second decoder is not. This is because the first decoder outputs a predicted noisy image, while the second decoder outputs a second predicted denoised image. In the previous steps, resizing the decoder output image is necessary to obtain the final predicted denoised image, which significantly impacts the accuracy of the output image. The first decoder, however, outputs a predicted noisy image, which is not resized when processed with the noisy image. Therefore, in the image denoising process using the trained first decoder, the final result is calculated from the predicted noisy image and the original noisy image. In this case, only the predicted noisy image is resized; the original noisy image is not. Therefore, the denoised image portion in the final result is unresized. In contrast, in the image denoising model using the second decoder, the output is the second predicted denoised image, which is resized (using an interpolation algorithm) to obtain the final result. Therefore, compared to the second decoder, the first decoder generates denoised images that retain better details and is more suitable for use in image denoising models to denoise images.

[0212] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0213] Please refer to Figure 16 This illustration shows a schematic diagram of a computer device provided in one embodiment of this application. The computer device can be any electronic device with data computing, processing, and storage functions, such as a mobile phone, tablet computer, PC (Personal Computer), or server. The computer device can be implemented as a model-using device to implement the image denoising method provided in the above embodiments; or, the computer device can be implemented as a model training device to implement the training method of the image denoising model provided in the above embodiments. Specifically:

[0214] The computer device 1600 includes a central processing unit (such as a CPU, GPU, or FPGA) 1601, a system memory 1604 including RAM (Random-Access Memory) 1602 and ROM (Read-Only Memory) 1603, and a system bus 1605 connecting the system memory 1604 and the central processing unit 1601. The computer device 1600 also includes a basic input / output system 1606 to facilitate information transfer between various devices within the server, and a mass storage device 1607 for storing the operating system 1613, application programs 1614, and other program modules 1615.

[0215] The basic input / output system 1606 includes a display 1608 for displaying information and an input device 1609 for user input, such as a mouse or keyboard. Both the display 1608 and the input device 1609 are connected to the central processing unit 1601 via an input / output controller 1610 connected to the system bus 1605. The basic input / output system 1606 may also include the input / output controller 1610 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1610 also provides output to a display screen, printer, or other types of output devices.

[0216] The mass storage device 1607 is connected to the central processing unit 1601 via a mass storage controller (not shown) connected to the system bus 1605. The mass storage device 1607 and its associated computer-readable media provide non-volatile storage for the computer device 1600. That is, the mass storage device 1607 may include computer-readable media (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0217] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage medium is not limited to the above-mentioned types. The system memory 1304 and mass storage device 1607 described above can be collectively referred to as memory.

[0218] According to an embodiment of this application, the computer device 1600 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 1600 can be connected to the network 1612 via the network interface unit 1611 connected to the system bus 1605, or the network interface unit 1611 can be used to connect to other types of networks or remote computer systems (not shown).

[0219] The memory also includes at least one instruction, at least one program, code set, or instruction set, which is stored in the memory and configured to be executed by one or more processors to implement the above-described image denoising method or image denoising model training method.

[0220] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one instruction, at least one program, code set, or instruction set. When executed by a processor of a computer device, the at least one instruction, at least one program, code set, or instruction set implements the image denoising method or image denoising model training method provided in the above embodiments.

[0221] Optionally, the computer-readable storage medium may include: ROM (Read-Only Memory), RAM (Random-Access Memory), SSD (Solid State Drives), or optical disc, etc. The random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0222] In an exemplary embodiment, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned image denoising method or image denoising model training method.

[0223] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0224] The above are merely exemplary embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application shall be included within the protection scope of this application.

Claims

1. A training method for an image denoising model, characterized in that, The image denoising model includes an encoder, a first decoder, and a second decoder; the method includes: The training samples of the image denoising model are obtained, including noise-free sample images, pure noise images, and noisy sample images generated based on the noise-free sample images and the pure noise images. The encoder extracts feature information from the noisy sample image; The first decoder processes the feature information of the noisy sample image to obtain a predicted noise image corresponding to the noisy sample image. The predicted noise image is used to indicate the noise information contained in the noisy sample image predicted by the first decoder. Based on the predicted noise image and the noisy sample image, a first predicted denoising image corresponding to the noisy sample image is generated. The second decoder processes the feature information of the noisy sample image to obtain the second predicted denoised image corresponding to the noisy sample image; Based on the first predicted denoised image, the second predicted denoised image, and the noise-free sample image, the training loss of the image denoising model is determined, and the parameters of the image denoising model are adjusted based on the training loss.

2. The method according to claim 1, characterized in that, The step of determining the training loss of the image denoising model based on the first predicted denoised image, the second predicted denoised image, and the noise-free sample image, and adjusting the parameters of the image denoising model based on the training loss, includes: Based on the first predicted denoised image and the noise-free sample image, a first training loss is determined, which is used to measure the difference between the first predicted denoised image and the noise-free sample image. Based on the first training loss, adjust the parameters of the encoder and the first decoder.

3. The method according to claim 2, characterized in that, The step of determining the first training loss based on the first predicted denoised image and the noise-free sample image includes: Based on the first predicted denoised image and the noise-free sample image, a first pixel-level loss, a first semantic feature loss, and a first image gradient loss are determined; wherein, the first pixel-level loss is used to measure the difference between the pixels of the first predicted denoised image and the pixels of the noise-free sample image, the first semantic feature loss is used to measure the difference between the semantic features of the first predicted denoised image and the semantic features of the noise-free sample image, and the first image gradient loss is used to measure the difference between the image gradient of the first predicted denoised image and the image gradient of the noise-free sample image; The first training loss is determined based on the first pixel-level loss, the first semantic feature loss, and the first image gradient loss.

4. The method according to claim 1, characterized in that, The step of determining the training loss of the image denoising model based on the first predicted denoised image, the second predicted denoised image, and the noise-free sample image, and adjusting the parameters of the image denoising model based on the training loss, includes: A second training loss is determined based on the second predicted denoised image and the noise-free sample image, the second training loss being used to measure the difference between the second predicted denoised image and the noise-free sample image; Based on the second training loss, the parameters of the encoder and the second decoder are adjusted.

5. The method according to claim 4, characterized in that, The step of determining the second training loss based on the second predicted denoised image and the noise-free sample image includes: Based on the second predicted denoised image and the noise-free sample image, a second pixel-level loss, a second semantic feature loss, and a second image gradient loss are determined; wherein, the second pixel-level loss is used to measure the difference between the pixels of the second predicted denoised image and the pixels of the noise-free sample image, the second semantic feature loss is used to measure the difference between the semantic features of the second predicted denoised image and the semantic features of the noise-free sample image, and the second image gradient loss is used to measure the difference between the image gradient of the second predicted denoised image and the image gradient of the noise-free sample image; The second training loss is determined based on the second pixel-level loss, the second semantic feature loss, and the second image gradient loss.

6. The method according to claim 1, characterized in that, The image denoising model further includes a discriminator; the method further includes: The discriminator performs pixel-level discrimination on the first predicted denoised image and the noise-free sample image respectively, and obtains the pixel-level discrimination results corresponding to the first predicted denoised image and the noise-free sample image respectively; Based on the pixel-level discrimination results, the discrimination loss corresponding to the discriminator is determined, and the discrimination loss is used to measure the accuracy of the pixel-level discrimination results output by the discriminator; The discrimination loss is used to adjust the parameters of the discriminator, and also to adjust the parameters of the encoder and the first decoder.

7. The method according to claim 6, characterized in that, The discriminator includes a feature mapping layer, n cascaded downsampling layers, n cascaded upsampling layers, and a feature extraction layer, where n is an integer greater than 1. The step of performing pixel-level discrimination on the first predicted denoised image and the noise-free sample image by the discriminator to obtain pixel-level discrimination results corresponding to the first predicted denoised image and the noise-free sample image respectively includes: The first predicted denoised image or the noise-free sample image is processed by the feature mapping layer to obtain the initial feature map of the first predicted denoised image or the noise-free sample image. The initial feature map is downsampled through n steps by the n cascaded downsampling layers to obtain a downsampled feature map; The downsampled feature map is processed through n upsampling steps by the n cascaded upsampling layers to obtain the upsampled feature map; wherein, the downsampling layer and the upsampling layer are connected by skip connections; The upsampled feature map is processed by the feature extraction layer to obtain the pixel-level discrimination result of the first predicted denoised image or the noise-free sample image.

8. The method according to claim 1, characterized in that, The image denoising model further includes a classifier; the method further includes: The classifier processes the feature information of the noisy sample image or the noise-free sample image to obtain the noise prediction result of the noisy sample image or the noise-free sample image. The noise prediction result is used to indicate whether the noisy sample image or the noise-free sample image contains noise. Based on the noise prediction results and the actual noise results, a third training loss is determined, which is used to measure the difference between the noise prediction results and the actual noise results; Based on the third training loss, the parameters of the encoder and the classifier are adjusted.

9. The method according to claim 1, characterized in that, The step of generating a first predicted denoised image corresponding to the noisy sample image based on the predicted noisy image and the noisy sample image includes: Divide the values ​​of corresponding pixels in the noisy sample image and the predicted noisy image to obtain the first predicted denoised image; or, The first denoised image is obtained by subtracting the pixel values ​​at corresponding positions in the noisy sample image and the predicted noisy image.

10. The method according to any one of claims 1 to 9, characterized in that, After obtaining the training samples of the image denoising model, the method further includes: The same data augmentation process is performed on the noisy sample image and the noise-free sample image, the data augmentation process including at least one of the following: random scaling, random cropping, random horizontal flipping, random vertical flipping, random rotation, affine transformation, translation, scaling and rotation, and perspective transformation; The noisy sample image or the noise-free sample image after the data augmentation process is input into the encoder for feature extraction.

11. An image denoising method, characterized in that, The method includes: A noisy image to be processed by an image denoising model is obtained, wherein the image denoising model includes an encoder and a first decoder; The encoder extracts feature information from the noisy image; The first decoder processes the feature information of the noisy image to obtain a noise image corresponding to the noisy image. The noise image is used to indicate the noise information contained in the noisy image predicted by the first decoder. Based on the noisy image and the noisy image, a denoised image corresponding to the noisy image is generated.

12. The method according to claim 11, characterized in that, The step of generating a denoised image corresponding to the noisy image based on the noisy image and the noisy image includes: Divide the values ​​of corresponding pixels in the noisy image and the noisy image to obtain the first predicted denoised image; or, The first predicted denoised image is obtained by subtracting the values ​​of corresponding pixels in the noisy image from the values ​​of the noisy image.

13. The method according to claim 11, characterized in that, The encoder includes k cascaded lower-adoption layers, and the first decoder includes k cascaded upper-adoption layers, where k is an integer greater than 1. The step of processing the feature information of the noisy image through the first decoder to obtain the noise image corresponding to the noisy image includes: The noisy image is obtained by performing k-step upsampling on the feature information of the noisy image through the k cascaded upsampling layers; wherein the downsampling layer and the upsampling layer are connected by skip connections.

14. The method according to claim 11, characterized in that, The image denoising model further includes a classifier; the method further includes: The classifier processes the feature information of the noisy image to obtain a noise prediction result for the noisy image. The noise prediction result is used to indicate whether the noisy image contains noise. If the noise prediction result indicates that the noisy image contains noise, the step of processing the feature information of the noisy image through the first decoder to obtain the noise image corresponding to the noisy image is performed.

15. The method according to any one of claims 11 to 14, characterized in that, The method further includes: The noisy image is subjected to a first size adjustment process to obtain an adjusted noisy image; wherein, the adjusted noisy image is input to the encoder for feature extraction; The noise image is subjected to a second size adjustment process to obtain an adjusted noise image; wherein, the second size adjustment process is the reverse adjustment process corresponding to the first size adjustment process; The step of generating a denoised image corresponding to the noisy image based on the noisy image and the noisy image includes: Based on the adjusted noisy image and the noisy image, a denoised image corresponding to the noisy image is generated.

16. A training device for an image denoising model, characterized in that, The image denoising model includes an encoder, a first decoder, and a second decoder; the device includes: The sample acquisition module is used to acquire training samples of the image denoising model. The training samples include noise-free sample images, pure noise images, and noisy sample images generated based on the noise-free sample images and the pure noise images. The information extraction module is used to extract feature information from the noisy sample image through the encoder; The first processing module is configured to process the feature information of the noisy sample image through the first decoder to obtain a predicted noise image corresponding to the noisy sample image, wherein the predicted noise image is used to indicate the noise information contained in the noisy sample image predicted by the first decoder; and generate a first predicted denoising image corresponding to the noisy sample image based on the predicted noise image and the noisy sample image. The second processing module is used to process the feature information of the noisy sample image through the second decoder to obtain the second predicted denoised image corresponding to the noisy sample image; The parameter adjustment module is used to determine the training loss of the image denoising model based on the first predicted denoised image, the second predicted denoised image, and the noise-free sample image, and to adjust the parameters of the image denoising model based on the training loss.

17. An image denoising device, characterized in that, The device includes: An image acquisition module is used to acquire a noisy image to be processed by an image denoising model, wherein the image denoising model includes an encoder and a first decoder; The information extraction module is used to extract feature information of the noisy image through the encoder; The information processing module is used to process the feature information of the noisy image through the first decoder to obtain a noise image corresponding to the noisy image. The noise image is used to indicate the noise information contained in the noisy image predicted by the first decoder. The image generation module is used to generate a denoised image corresponding to the noisy image based on the noisy image and the noisy image.

18. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being loaded and executed by the processor to implement the training method of the image denoising model as described in any one of claims 1 to 10, or to implement the image denoising method as described in any one of claims 11 to 15.

19. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the training method of the image denoising model as described in any one of claims 1 to 10, or to implement the image denoising method as described in any one of claims 11 to 15.

20. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium. A processor reads and executes the computer instructions from the computer-readable storage medium to implement the training method of the image denoising model as described in any one of claims 1 to 10, or to implement the image denoising method as described in any one of claims 11 to 15.

Images