Image noise reduction model training method, image noise reduction method, device and equipment

Abstract

The invention provides an image noise reduction model training method, an image noise reduction method, an image noise reduction device and image noise reduction equipment, and relates to the technical field of computer vision. The method comprises the following steps: acquiring a first sample image and a first true value image of the RAW; inputting the first sample image into a first noise reduction network to obtain a second sample image; inputting the second sample image into a conversion network to obtain a third sample image of RGB; inputting the first true value image into a conversion network to obtain a second true value image of RGB; inputting the third sample image and the second true value image into a feature extraction layer in a second noise reduction network to obtain a first feature vector and a second feature vector; calculating a first loss value of the second sample image and the first true value image, a first feature vector and a second loss value of the second feature vector; and adjusting parameters of the first noise reduction network in combination with the first loss value and the second loss value to obtain a trained image noise reduction model. According to the invention, the model for noise reduction of the RAW format image can be trained based on the perceptual loss training mode.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an image denoising model training method, an image denoising method, an apparatus, and a device. Background Technology

[0002] In existing technology, an electronic device can input a noisy first sample image in Red-Green-Blue (RGB) format into an image denoising network to be trained to obtain a predicted image; then, based on the difference between the predicted image and the corresponding noiseless first ground truth image of the first sample image, a first loss value is calculated; the predicted image and the first ground truth image are respectively input into a pre-trained neural network (wherein, the pre-trained neural network is pre-trained based on a noisy second sample image in RGB format and the corresponding noiseless second ground truth image) to obtain the feature vector of the predicted image and the feature vector of the first ground truth image output by the feature extraction layer in the pre-trained neural network; then, the electronic device can calculate the degree of difference between the feature vector of the predicted image and the feature vector of the first ground truth image to obtain a second loss value (this second loss value can be called a perceptual loss); then, based on the first loss value and the second loss value, the network parameters of the image denoising network are adjusted to obtain the trained image denoising network.

[0003] This perceptual loss training method, which trains the image denoising network based on perceptual loss values, can more accurately identify the difference between the predicted image and the first ground truth image by using the color distribution and spatial structure represented by the feature vector of the image. This enables the trained image denoising network to denoise the image based on the color distribution and spatial structure of the image, generating images that conform to human visual perception and improving image quality.

[0004] However, existing technologies do not provide a method for training perceptual loss for raw (RAW) format images. How to train a model for denoising RAW format images based on perceptual loss training is an urgent problem to be solved. Summary of the Invention

[0005] The purpose of this invention is to provide an image denoising model training method, image denoising method, apparatus, and device, for training a model for denoising RAW format images based on a perceptual loss training approach. The specific technical solution is as follows:

[0006] In a first aspect, embodiments of the present invention provide an image denoising model training method, the method comprising:

[0007] Acquire a noisy first sample image in the original RAW format, and a noiseless first ground truth image in RAW format with the same image content as the first sample image;

[0008] The first sample image is input into the first denoising network of the image denoising model with the initial structure to obtain the second sample image in RAW format; the image denoising model also includes a transformation network and a second denoising network.

[0009] The second sample image is input into the conversion network to obtain a third sample image in RGB format; and the first ground truth image is input into the conversion network to obtain a second ground truth image in RGB format.

[0010] The third sample image is input into the second denoising network to obtain the first feature vector output by the feature extraction layer in the second denoising network; and the second ground truth image is input into the second denoising network to obtain the second feature vector output by the feature extraction layer.

[0011] The difference between the second sample image and the first ground truth image is calculated to obtain a first loss value; and the difference between the first feature vector and the second feature vector is calculated to obtain a second loss value.

[0012] By combining the first loss value and the second loss value, the network parameters of the first denoising network are adjusted to obtain the trained image denoising model.

[0013] Optionally, the conversion network is pre-trained based on a fourth sample image in RAW format and a third ground truth image in RGB format that has the same image content as the fourth sample image; when the network parameters of the first noise reduction network are adjusted by combining the first loss value and the second loss value, the network parameters of the conversion network are frozen.

[0014] And / or,

[0015] The second denoising network is pre-trained based on a noisy fifth sample image in RGB format and a noiseless fourth ground truth image in RGB format that has the same content as the fifth sample image; when the network parameters of the first denoising network are adjusted by combining the first loss value and the second loss value, the network parameters of the second denoising network are frozen.

[0016] Optionally, the conversion network is trained through the following steps: acquiring a fourth sample image in RAW format and a third ground truth image in RGB format with the same image content as the fourth sample image; inputting the fourth sample image into the conversion network with the initial structure to obtain a sixth sample image in RGB format; calculating a third loss value based on the difference between the sixth sample image and the third ground truth image in the three RGB channels; and adjusting the network parameters of the conversion network based on the third loss value to obtain the trained conversion network.

[0017] Optionally, the conversion network includes: an input layer, a downsampling sub-network, an upsampling sub-network, and an output layer;

[0018] The step of inputting the fourth sample image into the initial structure of the transformation network to obtain the sixth sample image in RGB format includes: inputting the fourth sample image into the input layer to obtain a third feature vector; inputting the third feature vector into the downsampling sub-network to obtain a fourth feature vector; obtaining a fifth feature vector based on the fourth feature vector and the upsampling sub-network; and inputting the concatenation result of the third feature vector and the fifth feature vector into the output layer to obtain the sixth sample image in RGB format.

[0019] Optionally, the downsampling subnetwork includes a first convolutional layer and a second convolutional layer, and the upsampling subnetwork includes a third convolutional layer and a fourth convolutional layer; the kernel size and stride of the first convolutional layer, the second convolutional layer, the third convolutional layer and the fourth convolutional layer are all the same;

[0020] The step of inputting the third feature vector into the downsampling sub-network to obtain the fourth feature vector includes: inputting the third feature vector into the first convolutional layer to obtain the first sub-feature vector; and inputting the first sub-feature vector into the second convolutional layer to obtain the second sub-feature vector.

[0021] The step of obtaining the fifth feature vector based on the fourth feature vector and the upsampling sub-network includes: inputting the second sub-feature vector into the third convolutional layer to obtain the third sub-feature vector; and inputting the concatenation result of the first sub-feature vector and the third sub-feature vector into the fourth convolutional layer to obtain the fifth feature vector.

[0022] Optionally, the second denoising network is trained through the following steps: acquiring a noisy fifth sample image in RGB format and a noiseless fourth ground truth image in RGB format with the same image content as the fifth sample image; inputting the fifth sample image into the second denoising network with an initial structure to obtain a seventh sample image in RGB format; calculating a fourth loss value based on the difference between the seventh sample image and the fourth ground truth image in the three RGB channels; and adjusting the network parameters of the second denoising network based on the fourth loss value to obtain the trained second denoising network.

[0023] Secondly, embodiments of the present invention provide an image noise reduction method, the method comprising:

[0024] Obtain the first image to be denoised in its original RAW format;

[0025] The first image to be denoised is input into the first denoising network of a pre-trained image denoising model to obtain the second image to be denoised; wherein, the image denoising model is trained based on the above-described image denoising model training method;

[0026] The second image to be denoised is input into the conversion network in the image denoising model to obtain a third image to be denoised in RGB format;

[0027] The third image to be denoised is input into the second denoising network in the image denoising model to obtain the denoised RGB format image corresponding to the first image to be denoised.

[0028] Optionally, the conversion network includes: an input layer, a downsampling sub-network, an upsampling sub-network, and an output layer;

[0029] The step of inputting the second image to be denoised into the conversion network of the image denoising model to obtain the third image to be denoised in RGB format includes:

[0030] The second image to be denoised is input into the input layer to obtain the sixth feature vector;

[0031] The sixth feature vector is input into the downsampling sub-network to obtain the seventh feature vector;

[0032] Based on the seventh feature vector and the upsampling sub-network, the eighth feature vector is obtained;

[0033] The concatenation result of the sixth and eighth feature vectors is input into the output layer to obtain the third image to be denoised in RGB format.

[0034] Optionally, the downsampling subnetwork includes a first convolutional layer and a second convolutional layer, and the upsampling subnetwork includes a third convolutional layer and a fourth convolutional layer; the kernel size and stride of the first convolutional layer, the second convolutional layer, the third convolutional layer and the fourth convolutional layer are all the same;

[0035] The step of inputting the sixth feature vector into the downsampling sub-network to obtain the seventh feature vector includes:

[0036] The sixth feature vector is input into the first convolutional layer to obtain the fourth sub-feature vector;

[0037] The fourth sub-feature vector is input into the second convolutional layer to obtain the fifth sub-feature vector;

[0038] The process of obtaining the eighth feature vector based on the seventh feature vector and the upsampling sub-network includes:

[0039] The fifth sub-feature vector is input into the third convolutional layer to obtain the sixth sub-feature vector;

[0040] The concatenation result of the fourth sub-feature vector and the sixth sub-feature vector is input into the fourth convolutional layer to obtain the eighth feature vector.

[0041] Thirdly, embodiments of the present invention provide an image denoising model training apparatus, the apparatus comprising:

[0042] The first acquisition module is used to acquire a noisy first sample image in the original RAW format and a noiseless first ground truth image in RAW format that has the same image content as the first sample image.

[0043] The first input module is used to input the first sample image into the first denoising network of the image denoising model with the initial structure to obtain the second sample image in RAW format; the image denoising model also includes a transformation network and a second denoising network.

[0044] The second input module is used to input the second sample image into the conversion network to obtain a third sample image in RGB format; and to input the first ground truth image into the conversion network to obtain a second ground truth image in RGB format.

[0045] The third input module is used to input the third sample image into the second denoising network to obtain the first feature vector output by the feature extraction layer in the second denoising network; and to input the second ground truth image into the second denoising network to obtain the second feature vector output by the feature extraction layer.

[0046] The calculation module is used to calculate the difference between the second sample image and the first ground truth image to obtain a first loss value; and to calculate the difference between the first feature vector and the second feature vector to obtain a second loss value;

[0047] The adjustment module is used to adjust the network parameters of the first denoising network by combining the first loss value and the second loss value, so as to obtain the trained image denoising model.

[0048] Fourthly, embodiments of the present invention provide an image noise reduction apparatus, the apparatus comprising:

[0049] The second acquisition module is used to acquire the first image to be denoised in the original RAW format;

[0050] The fourth input module is used to input the first image to be denoised into the first denoising network in the pre-trained image denoising model to obtain the second image to be denoised; wherein the image denoising model is trained based on the above-mentioned image denoising model training method;

[0051] The fifth input module is used to input the second image to be denoised into the conversion network in the image denoising model to obtain the third image to be denoised in RGB format;

[0052] The sixth input module is used to input the third image to be denoised into the second denoising network in the image denoising model to obtain the denoised RGB format image corresponding to the first image to be denoised.

[0053] This invention also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0054] Memory, used to store computer programs;

[0055] The processor, when executing the program stored in memory, implements the above-mentioned image denoising model training method and image denoising method.

[0056] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described image denoising model training method and image denoising method.

[0057] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described image denoising model training method and image denoising method.

[0058] Beneficial effects of the embodiments of the present invention:

[0059] In this embodiment of the invention, the image denoising model includes a first denoising network, a transformation network, and a second denoising network. A noisy first sample image in RAW format is input into the first denoising network to obtain a second sample image in RAW format. The transformation network converts the second sample image in RAW format into a third sample image in RGB format, and also converts the first ground truth image in RAW format into a second ground truth image in RGB format. The third sample image in RGB format is then input into the second denoising network to obtain a first feature vector output by the feature extraction layer. The second ground truth image in RGB format is then input into the second denoising network to obtain a second feature vector output by the feature extraction layer. The difference between the second sample image in RAW format and the first ground truth image in RAW format is calculated to obtain a first loss value (i.e., the loss between the prediction result and the ground truth value of the first denoising network). The difference between the first feature vector and the second feature vector is also calculated to obtain a second loss value (i.e., the perceptual loss between the third sample image and the second ground truth image). Finally, the network parameters of the first denoising network are adjusted by combining the first and second loss values ​​to obtain the trained image denoising model.

[0060] This invention adjusts the network parameters of the first denoising network by combining the first loss value and the second loss value. The second loss value, which is used as a perceptual loss value, can assist the first loss value. This allows for parameter tuning of the first denoising network used for denoising RAW format images, thereby improving the denoising effect of the trained first denoising network. This enables the training of a model for denoising RAW format images based on a perceptual loss training method.

[0061] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0063] Figure 1 A flowchart illustrating an image denoising model training method provided in an embodiment of the present invention;

[0064] Figure 2 This is a flowchart illustrating the first training method for the conversion network in the image denoising model provided in this embodiment of the invention.

[0065] Figure 3This is a flowchart illustrating the second training method for the conversion network in the image denoising model provided in this embodiment of the invention.

[0066] Figure 4 This is a flowchart illustrating a training method for the second denoising network in the image denoising model provided in an embodiment of the present invention.

[0067] Figure 5 A schematic flowchart of the first image noise reduction method provided in an embodiment of the present invention;

[0068] Figure 6 A schematic flowchart of the second image denoising method provided in an embodiment of the present invention;

[0069] Figure 7 This is a schematic diagram of the structure of a conversion network provided in an embodiment of the present invention;

[0070] Figure 8 A schematic diagram illustrating the principle of an image denoising model training method provided in an embodiment of the present invention.

[0071] Figure 9 This is a schematic diagram of the structure of an image denoising model training device provided in an embodiment of the present invention;

[0072] Figure 10 This is a schematic diagram of the structure of an image noise reduction device provided in an embodiment of the present invention;

[0073] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0074] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on the present invention are within the scope of protection of the present invention.

[0075] In existing technologies, electronic devices can input noisy sample images in RGB format into an image denoising network to be trained to obtain a predicted image. Then, based on the difference between the predicted image and the corresponding noise-free ground truth image of the sample image, a loss value is calculated. This loss value reflects the difference in pixel values ​​at the same pixel position between the predicted image and the ground truth image. If the network parameters of the image denoising network are adjusted only based on this loss value, the trained image denoising network may also treat the original texture pixels and edge pixels in the image as noise and eliminate them, resulting in negative effects such as over-smoothing, distortion artifacts, and reduced saturation in the denoised image.

[0076] To mitigate the aforementioned negative effects, electronic devices can calculate a perceptual loss value and adjust the network parameters of the image denoising network accordingly. This perceptual loss value aids training, enabling the trained image denoising network to denoise images based on their color distribution and spatial structure, generating images that conform to human visual perception and improving image quality. Furthermore, when denoising raw (RAW) format images, the electronic device can accurately identify noise in the image based on the original information such as brightness and color contained within the RAW format image, and then eliminate the identified noise. In other words, combining perceptual loss training with RAW format image denoising is a meaningful and highly necessary approach.

[0077] However, existing technologies do not provide a method for training perceptual loss for RAW format images. How to train a model for denoising RAW format images based on perceptual loss training is an urgent problem to be solved.

[0078] In order to enable the training of a model for denoising RAW format images based on the perceptual loss training method, this invention provides an image denoising model training method, an image denoising method, an apparatus, and a device.

[0079] The following section first introduces an image denoising model training method provided by an embodiment of the present invention.

[0080] The image denoising model training method provided in this embodiment of the invention can be applied to electronic devices. For example, the electronic device can be a laptop or a desktop computer.

[0081] An image denoising model training method provided in this embodiment of the invention may include the following steps:

[0082] Obtain a noisy first sample image in the original RAW format, and a noiseless first ground truth image in RAW format with the same image content as the first sample image;

[0083] The first sample image is input into the first denoising network of the image denoising model with the initial structure to obtain the second sample image in RAW format; the image denoising model also includes a transformation network and a second denoising network.

[0084] The second sample image is input into the conversion network to obtain a third sample image in RGB format; and the first ground truth image is input into the conversion network to obtain a second ground truth image in RGB format.

[0085] The third sample image is input into the second denoising network to obtain the first feature vector output by the feature extraction layer in the second denoising network; and the second ground truth image is input into the second denoising network to obtain the second feature vector output by the feature extraction layer.

[0086] The difference between the second sample image and the first ground truth image is calculated to obtain the first loss value; and the difference between the first feature vector and the second feature vector is calculated to obtain the second loss value.

[0087] By combining the first loss value and the second loss value, the network parameters of the first denoising network are adjusted to obtain the trained image denoising model.

[0088] This invention adjusts the network parameters of the first denoising network by combining the first loss value and the second loss value. The second loss value, which is used as a perceptual loss value, can assist the first loss value. This allows for parameter tuning of the first denoising network used for denoising RAW format images, thereby improving the denoising effect of the trained first denoising network. This enables the training of a model for denoising RAW format images based on a perceptual loss training method.

[0089] The following is combined with Figure 1 This invention introduces an image denoising model training method provided by an embodiment of the present invention, such as... Figure 1 As shown, the method may include steps S101-S106.

[0090] S101, acquire the first sample image with noise in the original RAW format, and the first ground truth image without noise in RAW format with the same image content as the first sample image.

[0091] Understandably, to train a model for denoising RAW images, the electronic device first acquires a first sample image and a first ground truth image in RAW format. The first sample image and the first ground truth image have identical content; the first sample image is noisy, while the first ground truth image is noiseless. However, this doesn't mean the first ground truth image is completely noiseless; it simply means it has less noise than the first sample image. Therefore, the first ground truth image can be considered noiseless, while the first sample image is noisy. For example, the photographer can set the ISO sensitivity of the image acquisition device to the lowest value (e.g., set it to 100) and then use the device to acquire a noise-free image.

[0092] For example, the image acquisition device can acquire a first ground truth image without noise according to preset acquisition conditions, and then the electronic device can acquire the first ground truth image and add noise to the first ground truth image to obtain a first sample image.

[0093] Specifically, when the image acquisition device acquires images according to preset acquisition conditions, the analog gain of the image sensor contained in the image acquisition device is 1x. When the analog gain is 1x, the image acquisition device will not amplify the electrical signal acquired by the image sensor, and thus will not introduce noise caused by electrical signal amplification into the acquired image. The acquisition environment of the image acquisition device is under normal lighting conditions (e.g., normal lighting conditions are light intensity of 300-500 lux). The noise of the image acquired under normal lighting conditions is less than that of the image acquired under low light conditions (e.g., low light conditions are light intensity of 1-100 lux). The exposure time of the image acquisition device when acquiring images is within a preset time range (e.g., the preset time range is 1 / 500th of a second to 1 / 120th of a second). When the exposure time of the acquired image is within the preset time range, abnormal brightness of the acquired image (e.g., overexposure or underexposure) can be avoided, thereby avoiding the problem of missing content in the image caused by abnormal brightness.

[0094] During the process of adding noise to the first ground truth image, the electronic device can add noise to the first ground truth image according to preset noise parameters.

[0095] In the first implementation, the preset noise parameters may include the Poisson mean. The electronic device may randomly change the pixel value of each pixel in the first true image according to the Poisson distribution. The mean of the pixel values ​​of each pixel after the change is the Poisson mean, thus obtaining the noisy first sample image.

[0096] In the second implementation, the preset noise parameters can include Gaussian mean and Gaussian standard deviation. The electronic device can generate a random pixel value for each pixel in the first ground truth image according to the Gaussian mean and Gaussian standard deviation, and add the random pixel value to the pixel value of that pixel to obtain a noisy first sample image. In the third implementation, the preset noise parameters can include Poisson mean, Gaussian mean, and Gaussian standard deviation. The electronic device can add noise to the first ground truth image according to the first implementation, and then add noise again to the first ground truth image with added noise according to the second implementation to obtain a noisy first sample image.

[0097] For example, the electronic device can also acquire a public dataset containing a ground truth image and corresponding noisy sample images, and obtain the first sample image and the first ground truth image from the public dataset. The public dataset can be a Smartphone Image Denoising Dataset (SIDD) or a Low-Light Denoising Dataset (LOLD), etc.

[0098] S102, input the first sample image into the first denoising network in the image denoising model of the initial structure to obtain the second sample image in RAW format.

[0099] It is understood that the image denoising model includes a first denoising network, a transformation network, and a second denoising network. The first denoising network is used to process RAW format images, and both its input and output are RAW format images. For example, the initial structure of the first denoising network can be a convolutional neural network, a recurrent neural network, or a transformer network. The network parameters of the initial structure of the first denoising network are the objects that need to be adjusted during training.

[0100] S103, input the second sample image into the conversion network to obtain the third sample image in RGB format; and input the first ground truth image into the conversion network to obtain the second ground truth image in RGB format.

[0101] As can be understood, the conversion network is used to convert RAW format images to RGB format images. An electronic device can input a second sample image in RAW format into the conversion network to obtain a third sample image in RGB format, and input a first ground truth image in RAW format into the conversion network to obtain a second ground truth image in RGB format. The conversion network can also be called a demosaicing network.

[0102] In the prior art, electronic devices can perform interpolation calculations on RAW format images based on interpolation algorithms to obtain RGB format images, thereby converting RAW format images into RGB format images. Specifically, the electronic devices use different interpolation functions to calculate the pixel values ​​of each pixel in the RAW format image whose row coordinates belong to different parity, that is, they use piecewise functions to calculate the pixel values ​​of each pixel.

[0103] If the RAW format image output by the first denoising network is converted to RGB format using the interpolation algorithm in the existing technology and then input into the second denoising network, the gradient backpropagation link from the second denoising network to the first denoising network is interrupted because the piecewise function is discontinuous and the derivative of the loss function and the piecewise function does not exist. As a result, the perceptual loss value obtained based on the second denoising network cannot be applied to the first denoising network, and therefore the first denoising network cannot be trained based on the perceptual loss value.

[0104] In this embodiment of the invention, a conversion network is set between the first denoising network and the second denoising network. The network parameters in the conversion network convert the RAW format image to the RGB format image. In this way, during the gradient backpropagation process, the first gradient of the loss function and the network parameters of the second denoising network can be calculated. Then, the second gradient of the first gradient and the network parameters of the conversion network can be calculated. Then, the third gradient of the second gradient and the network parameters of the first denoising network can be calculated. Then, the network parameters of the first denoising network can be adjusted according to the third gradient to train the first denoising network and obtain the trained first denoising network.

[0105] Understandably, the network parameters of the transformation network can be initial values ​​or values ​​obtained through training. If the network parameters of the transformation network are initial values, then during the training of the image denoising model, the network parameters of both the transformation network and the first denoising network can be adjusted. If the network parameters of the transformation network are values ​​obtained through training, then during the training of the image denoising model, the network parameters of both the transformation network and the first denoising network can be adjusted; alternatively, the network parameters of the first denoising network can be adjusted without adjusting the network parameters of the transformation network.

[0106] In one implementation, in the subsequent step S106, the electronic device can combine the first loss value and the second loss value to adjust the network parameters of the conversion network and the first noise reduction network to obtain the trained conversion network and the first noise reduction network.

[0107] In another implementation, in the subsequent step S106, when the electronic device adjusts the network parameters of the first denoising network by combining the first loss value and the second loss value, it can freeze the network parameters of the transformation network, keeping them fixed. This avoids adjusting the network parameters of the trained transformation network, reduces the number of parameters that need to be adjusted, and improves training efficiency. For example, the electronic device can set the training attribute of the transformation network parameters to non-training (false).

[0108] For example, the network parameters of the transformation network are values ​​obtained through training. The transformation network is pre-trained on a fourth sample image in RAW format and a third ground truth image in RGB format that has the same image content as the fourth sample image.

[0109] It is understandable that electronic devices can use RAW format images as sample images and RGB format images with the same content as the sample images as ground truth images to train the conversion network, thus obtaining a pre-trained conversion network. The specific training process of the conversion network will be described in detail in subsequent embodiments.

[0110] S104, input the third sample image into the second denoising network to obtain the first feature vector output by the feature extraction layer in the second denoising network; and input the second ground truth image into the second denoising network to obtain the second feature vector output by the feature extraction layer.

[0111] Understandably, the second denoising network is used to denoise noisy RGB format images. The feature extraction layer in the second denoising network can extract feature vectors representing color distribution and spatial structure from the input image. The electronic device can input a third sample image into the second denoising network to obtain the first feature vector output by the feature extraction layer; and input a second ground truth image into the second denoising network to obtain the second feature vector output by the feature extraction layer. The initial structure of the second denoising network can be a trained Visual Geometry Group 16 (VGG16) network or a Residual Connection (ResNet) network.

[0112] Understandably, the network parameters of the second denoising network can be initial values ​​or values ​​obtained through training. If the network parameters of the second denoising network are initial values, then during the training of the image denoising model, the network parameters of both the second and first denoising networks can be adjusted. If the network parameters of the second denoising network are values ​​obtained through training, then during the training of the image denoising model, the network parameters of both the second and first denoising networks can be adjusted; alternatively, the network parameters of the first denoising network can be adjusted without adjusting the network parameters of the second denoising network.

[0113] In one implementation, in the subsequent step S106, the electronic device can combine the first loss value and the second loss value to adjust the network parameters of the second noise reduction network and the first noise reduction network to obtain the trained second noise reduction network and the first noise reduction network.

[0114] In another implementation, in subsequent step S106, when the electronic device adjusts the network parameters of the first denoising network by combining the first loss value and the second loss value, it can freeze the network parameters of the second denoising network, keeping them fixed. This avoids adjusting the network parameters of the trained second denoising network, reducing the number of parameters that need to be adjusted and improving training efficiency. For example, the electronic device can set the training attribute of the network parameters of the second denoising network to not train (false).

[0115] For example, the network parameters of the second denoising network are values ​​obtained through training. The second denoising network is pre-trained based on a noisy fifth sample image in RGB format and a noiseless fourth ground truth image in RGB format that has the same image content as the fifth sample image.

[0116] It is understood that electronic devices can use noisy RGB images as sample images and noiseless RGB images with the same content as the sample images as ground truth images to train the second denoising network, thus obtaining a pre-trained second denoising network. The specific training process of the second denoising network will be described in detail in subsequent embodiments.

[0117] S105, calculate the difference between the second sample image and the first ground truth image to obtain the first loss value; and calculate the difference between the first feature vector and the second feature vector to obtain the second loss value.

[0118] It is understandable that the second sample image is obtained by denoising the first sample image using the first denoising network, and the first ground truth image is the noise-free image corresponding to the first sample image. The difference between the second sample image and the first ground truth image is positively correlated with the denoising effect of the first denoising network. In the subsequent step S106, the electronic device adjusts the network parameters based on the first loss value, which can make the second sample image output by the adjusted first denoising network tend to the first ground truth image, thereby improving the denoising effect of the first denoising network.

[0119] For example, in calculating the first loss value, the electronic device can determine the corresponding second pixel position in the first ground truth image for each first pixel position in the second sample image, and then calculate the difference between the pixel value at the first pixel position and the pixel value at the corresponding second pixel position. Since both the second sample image and the first ground truth image are RAW format images, the pixel value at each pixel position is a single-channel pixel value. The electronic device can calculate the average of the differences between the pixel values ​​at each first pixel position and the pixel values ​​at the corresponding second pixel position to obtain the first loss value. The first loss value can characterize the pixel differences between the two images at each pixel point.

[0120] Understandably, the first feature vector is obtained by processing the third sample image using the second denoising network, and it reflects the color distribution and spatial structure of the third sample image. The second feature vector is obtained by processing the second ground truth image using the second denoising network, and it reflects the color distribution and spatial structure of the second ground truth image. The difference between the first and second feature vectors characterizes the difference in color distribution and spatial structure between the third sample image and the second ground truth image. The second loss value characterizes the color space difference between the two images in terms of color distribution and spatial structure.

[0121] Since the third sample image in RGB format is obtained by converting the second sample image in RAW format, and the second ground truth image in RGB format is obtained by converting the first ground truth image in RAW format, the difference between the first feature vector and the second feature vector is calculated. The resulting second loss value can characterize the difference in color distribution and spatial structure between the second sample image and the first ground truth image. The second loss value can also be called the perceptual loss value, meaning that the perceptual loss value can reflect the denoising effect of the first denoising network, and the perceptual loss value can be positively correlated with the denoising effect of the first denoising network. In the subsequent step S106, the electronic device adjusts the network parameters based on the second loss value, which can make the second sample image output by the first denoising network after parameter adjustment more closely resemble the first ground truth image, thereby improving the denoising effect of the first denoising network.

[0122] For example, in calculating the second loss value, the electronic device can first calculate the similarity between the first feature vector and the second feature vector, and then perform inverse correlation processing on the similarity to obtain the second loss value. Specifically, the electronic device can calculate the cosine similarity, Manhattan distance, or Chebyshev distance between the first and second feature vectors to obtain the similarity. The inverse correlation processing can be using the reciprocal of the similarity as the second loss value, the inverse of the similarity as the second loss value, or the negative logarithm of the similarity as the second loss value.

[0123] S106, By combining the first loss value and the second loss value, the network parameters of the first denoising network are adjusted to obtain the trained image denoising model.

[0124] It is understandable that electronic devices can combine the first loss value and the second loss value to adjust the network parameters of the first denoising network. That is, by combining the pixel difference and color space difference between the second sample image and the first ground truth image, the network parameters of the first denoising network are adjusted so that the output image of the adjusted first denoising network tends to reduce the pixel difference and color space difference, so as to approach a noise-free image, thereby obtaining a trained image denoising model.

[0125] For example, the electronic device can superimpose the first loss value and the second loss value according to a preset weight to obtain a loss value to be utilized, and adjust the network parameters of the first noise reduction network based on the loss value to be utilized. The preset weight of the first loss value can be within a preset weight range, such as 50% to 60%. The weight of the second loss value is one minus the preset weight of the first loss value.

[0126] When adjusting the network parameters of each network in an image denoising model, the following implementation methods exist.

[0127] In the first implementation, the electronic device can freeze the network parameters of the conversion network and / or the second denoising network, reducing the amount of parameter tuning and improving tuning efficiency. Specifically, the electronic device can freeze the network parameters of the conversion network and adjust the network parameters of the first and second denoising networks in the image denoising model; or, the electronic device can freeze the network parameters of the second denoising network and adjust the network parameters of the first and conversion networks in the image denoising model; or, the electronic device can freeze the network parameters of the conversion network and the second denoising network and adjust the network parameters of the first denoising network in the image denoising model.

[0128] In the second implementation, the electronic device does not freeze the network parameters of the conversion network and the second denoising network, but adjusts the network parameters of the first denoising network, the conversion network, and the second denoising network in the image denoising model.

[0129] Understandably, when an image denoising model needs to denoise RAW format images in a specified scene, the electronic device can acquire a first sample image based on a first ground truth image captured in the specified scene (the specific acquisition method is similar to the aforementioned acquisition method of the first sample image); then, it can be trained based on the first ground truth image and the first sample image. The specified scene could be a roadside, a parking lot, or a school, etc.

[0130] Since the first ground truth image is collected in the specified scene and the first sample image is obtained based on the first ground truth image, both the first ground truth image and the first sample image contain features of the specified scene. Electronic devices can use images of the specified scene to train the first denoising network, the conversion network, and the second denoising network, so that the first denoising network can effectively denoise images in RAW format of the specified scene, the conversion network can accurately convert images in RAW format of the specified scene into images in RGB format, and the second denoising network can extract accurate feature vectors for images in RGB format of the specified scene, thereby improving the denoising effect of the image denoising model on images in the specified scene.

[0131] Figure 2 This is a flowchart illustrating the first training method for the conversion network in the image denoising model provided in this embodiment of the invention. Figure 2 As shown, the conversion network is trained through the following steps.

[0132] S201, acquire the fourth sample image in RAW format, and the third ground truth image in RGB format that has the same image content as the fourth sample image.

[0133] Understandably, the electronic device can first acquire the fourth sample image in RAW format, and then use an interpolation algorithm to convert it into a third ground truth image in RGB format. The fourth sample image is one type of image in RAW format, and the corresponding third ground truth image is another type of image in RGB format, with both types of images having identical content.

[0134] For example, the fourth sample image is an image captured by an image acquisition device under normal lighting conditions. The image content of the fourth sample image does not have any missing parts due to overexposure or underexposure. Consequently, the image content of the third ground truth image in RGB format also does not have any missing parts. Therefore, in the subsequent training process, the image with no missing parts can be used to calculate the loss value, thereby improving the accuracy of the trained conversion network in converting RAW format images to RGB format images.

[0135] In another implementation, the electronic device can first acquire a third ground truth image in RGB format, then reduce the RGB three channels of each pixel in the third ground truth image to a single channel, and distribute the pixel values ​​of the RGB three channels of that pixel to the single channels of other pixels, thus obtaining a fourth sample image in RAW format. However, RGB format images have significantly less original information than RAW format images. This implementation can only simulate RAW format images from RGB format images, and its accuracy is lower than that of directly acquiring RAW format images from an image acquisition device.

[0136] S202, input the fourth sample image into the transformation network of the initial structure to obtain the sixth sample image in RGB format.

[0137] Understandably, an electronic device can input the fourth sample image into the initial structure's conversion network to obtain the sixth sample image in RGB format output by the initial structure's conversion network. The input layer of the initial structure's conversion network can process RAW format images, and the output layer can output RGB format images. However, the RGB format image output by the initial structure's conversion network may only be RGB in format; the pixel values ​​of the RGB three channels in the output image may be inaccurate, failing to achieve the accuracy of the RGB three-channel pixel values ​​obtained from interpolation algorithms. Adjusting the network parameters of the conversion network subsequently can improve the accuracy of the RGB format image output by the conversion network.

[0138] S203, calculate the third loss value based on the difference between the sixth sample image and the third ground truth image in the three RGB channels.

[0139] Understandably, the difference between the RGB channels of the sixth sample image and the third ground truth image at the same pixel position is calculated; for each same pixel position, the difference between each of the three RGB channels at that same pixel position is calculated, and then the average of the differences of the three channels is calculated to obtain the difference at that same pixel position; then the average of the differences at each same pixel position is calculated to obtain the third loss value.

[0140] In one implementation, there are multiple fourth sample images, and each fourth sample image has a corresponding third ground truth image. The electronic device can use a preset loss function to calculate the third loss value. The preset loss function is as follows:

[0141]

[0142] Where loss represents the third loss value, N represents the number of fourth sample images, and Raw i Dem represents the i-th fourth sample image among multiple fourth sample images. net (Raw i ) represents the sixth sample image obtained after the i-th fourth sample image is input into the transformation network; gt i Let represent the third ground truth image corresponding to the i-th fourth sample image.

[0143] S204. Based on the third loss value, the network parameters of the transformation network are adjusted to obtain the trained transformation network.

[0144] Understandably, the third loss value represents the difference between the sixth sample image and the third ground truth image. Adjusting the network parameters of the transformation network according to the third loss value can make the RGB format image output by the adjusted transformation network closer to the RGB format ground truth image, thus obtaining the trained transformation network.

[0145] In this embodiment of the invention, the conversion network is trained using a fourth sample image in RAW format and a third ground truth image in RGB format that has the same image content as the fourth sample image. The completed conversion network can output an RGB format image with accurate pixel values ​​in the RGB three channels.

[0146] exist Figure 2 Based on the training method of the transformation network shown, the transformation network includes: an input layer, a downsampling sub-network, an upsampling sub-network, and an output layer, as follows: Figure 3 As shown, step S202 includes steps S2021-S2024.

[0147] S2021, input the fourth sample image into the input layer to obtain the third feature vector.

[0148] Understandably, the transformation network comprises an input layer, a downsampling sub-network, an upsampling sub-network, and an output layer. The input layer may contain at least one convolutional layer, and the output layer may contain at least one convolutional layer. The electronic device uses the convolutional layers of the input layer to perform preliminary feature extraction on the single-channel pixel values ​​of the fourth sample image in RAW format, obtaining a third feature vector. In one implementation, the input layer also includes an activation layer corresponding to each convolutional layer. The activation function in the activation layer can introduce non-linear features, allowing the model to learn more complex patterns.

[0149] S2022, the third feature vector is input into the downsampling subnetwork to obtain the fourth feature vector.

[0150] Understandably, a downsampling subnetwork can contain convolutional layers, and the stride of the convolutional layers in the downsampling subnetwork is greater than the stride of the convolutional layers in the input layer. The number of channels in the convolutional layers of the downsampling subnetwork is less than the number of channels in the convolutional layers in the input layer. Electronic devices, through the downsampling subnetwork, can expand the receptive field based on the larger stride of the convolutional layers, extract richer features, and at the same time, reduce the dimensionality of the third feature vector based on the smaller number of channels, thus reducing the computational load while retaining key features.

[0151] S2023, based on the fourth feature vector and the upsampling sub-network, the fifth feature vector is obtained.

[0152] Understandably, the upsampling subnetwork contains convolutional layers, and the stride of these convolutional layers is the same as that of the downsampling subnetwork. The number of channels in the convolutional layers of the upsampling subnetwork is also greater than that of the downsampling subnetwork. Electronic devices can extract richer features through the convolutional layers of the upsampling subnetwork. Furthermore, based on the larger number of channels, the fourth feature vector is up-dimensioned, preserving more image details from the RAW format within a larger vector dimension.

[0153] S2024, the concatenation result of the third feature vector and the fifth feature vector is input into the output layer to obtain the sixth sample image in RGB format.

[0154] Understandably, the electronic device can input the output vector of the input layer (the third feature vector) and the output vector of the downsampling sub-network (the fifth feature vector) into the output layer together. Specifically, the output vector of the input layer is concatenated with the output vector of the downsampling sub-network via a skip connection, and then the concatenated result is input into the output layer. The output layer contains convolutional layers. Through the output layer, the electronic device can extract features from the concatenated result of the third and fifth feature vectors, while simultaneously converting the concatenated result into an RGB three-channel vector dimension, resulting in the sixth sample image in RGB format.

[0155] In one implementation, the output layer includes a first type of convolutional layer and a second type of convolutional layer, with the second type of convolutional layer receiving the output of the activation layer corresponding to the last first type of convolutional layer. The output layer also includes an activation layer corresponding to each first type of convolutional layer; the activation function in the activation layer can introduce non-linear features, enabling the model to learn more complex patterns. The second type of convolutional layer does not have a corresponding activation layer, preventing the activation layer from disrupting the continuity of the output values.

[0156] In this embodiment of the invention, since the third feature vector is a shallow feature extracted from the RAW format image, it contains rich details (such as the edges and textures of the image) and is closer to the pixel distribution of the original image; the fifth feature vector is obtained by extracting features through the downsampling sub-network and the upsampling sub-network, and contains deeper semantic features, but may lose some details in the image. The electronic device can input the third feature vector and the fifth feature vector together into the output layer. By combining the shallow features and semantic features, the sixth sample image in RGB format can be output more accurately.

[0157] In one implementation, the downsampling subnetwork includes a first convolutional layer and a second convolutional layer, and the upsampling subnetwork includes a third convolutional layer and a fourth convolutional layer; the kernel size and stride of the first, second, third, and fourth convolutional layers are all the same.

[0158] Step S2022 includes steps S2022a-S2022b, and step S2023 includes steps S2023a-S2023b.

[0159] S2022a, the third feature vector is input into the first convolutional layer to obtain the first sub-feature vector.

[0160] S2022b, the first sub-feature vector is input into the second convolutional layer to obtain the second sub-feature vector.

[0161] S2023a, the second sub-feature vector is input into the third convolutional layer to obtain the third sub-feature vector.

[0162] S2023b inputs the concatenation result of the first and third sub-feature vectors into the fourth convolutional layer to obtain the fifth feature vector.

[0163] Understandably, the first convolutional layer can contain a convolutional layer with a stride of 2 and its corresponding activation layer, as well as multiple convolutional layers with a stride of 1 and their corresponding activation layers. The convolutional layer with a stride of 2 is used to perform dimensionality reduction on the third feature vector, and the activation layer corresponding to this convolutional layer with a stride of 2 is used to set non-positive values ​​to 0 to eliminate noise. The result of the dimensionality reduction can be input into the multiple convolutional layers with a stride of 1 and their corresponding activation layers in the first convolutional layer for feature extraction.

[0164] The second convolutional layer can contain a convolutional layer with a stride of 2 and its corresponding activation layer, as well as multiple convolutional layers with a stride of 1 and their corresponding activation layers. The convolutional layer with a stride of 2 and its corresponding activation layer is used to perform dimensionality reduction on the first sub-feature vector, and the activation layer corresponding to this convolutional layer with a stride of 2 is used to eliminate noise. The result of the dimensionality reduction can be input into the multiple convolutional layers with a stride of 1 and their corresponding activation layers in the second convolutional layer for feature extraction.

[0165] The third convolutional layer can contain a convolutional layer with a stride of 2, and multiple convolutional layers with a stride of 1 and their corresponding activation layers. The convolutional layer with a stride of 2 is used to perform dimensionality reduction on the second sub-feature vector. The result of the dimensionality reduction can be input into the multiple convolutional layers with a stride of 1 and their corresponding activation layers in the third convolutional layer for feature extraction.

[0166] The fourth convolutional layer can contain a convolutional layer with a stride of 2, and multiple convolutional layers with a stride of 1 and their corresponding activation layers. The convolutional layer with a stride of 2 is used to perform dimensionality upscaling on the third sub-feature vector and the first sub-feature vector. The result of the dimensionality reduction can be input into the multiple convolutional layers with a stride of 1 and their corresponding activation layers in the fourth convolutional layer for feature extraction.

[0167] The electronic device can input the third feature vector into the first convolutional layer to obtain the first sub-feature vector, then input the first sub-feature vector into the second convolutional layer to obtain the second sub-feature vector, thus completing the downsampling process; then input the second sub-feature vector into the third convolutional layer to obtain the third sub-feature vector, and input the concatenation result of the first and third sub-feature vectors into the fourth convolutional layer to obtain the fifth feature vector, thus completing the upsampling process.

[0168] The downsampling subnetwork and the upsampling subnetwork have a symmetrical structure. The convolutional layers in both subnetworks have the same kernel size and stride. The number of channels in the first and fourth convolutional layers is the same, and the number of channels in the second and third convolutional layers is the same. The number of channels in the first convolutional layer is greater than that in the second convolutional layer. This symmetrical structure allows electronic devices to extract more features while accurately restoring details lost during downsampling.

[0169] Electronic devices can sequentially use the first, second, third, and fourth convolutional layers to downsample the third feature vector before upsampling it. This avoids expanding the vector dimension and introducing unnecessary noise by performing upsampling first, thus improving the accuracy of the extracted feature vector.

[0170] Regarding step S2023b, there is a skip connection between the output of the first convolutional layer and the output of the third convolutional layer. The electronic device can concatenate the first sub-feature vector and the third sub-feature vector using this skip connection to obtain the concatenated result. Since the first sub-feature vector originates from the early stage of downsampling processing (the first convolutional layer), it retains more image details compared to the output of the second convolutional layer, but lacks more deep semantic features compared to the third sub-feature vector. The electronic device can combine the first and third sub-feature vectors and then perform a feature extraction with expanded dimensions based on the fourth convolutional layer. This allows for the extraction of a more accurate fifth feature vector based on more image details and semantic features.

[0171] Figure 4 This is a flowchart illustrating a training method for the second denoising network in the image denoising model provided in an embodiment of the present invention. Figure 4 As shown, the second noise reduction network is trained through the following steps.

[0172] S401, acquire a noisy fifth sample image in RGB format, and a noiseless fourth ground truth image in RGB format that has the same image content as the fifth sample image.

[0173] S402, input the fifth sample image into the second noise reduction network of the initial structure to obtain the seventh sample image in RGB format.

[0174] S403, calculate the fourth loss value based on the difference between the seventh sample image and the fourth ground truth image in the three RGB channels.

[0175] S404. Based on the fourth loss value, the network parameters of the second noise reduction network are adjusted to obtain the trained second noise reduction network.

[0176] It is understandable that after acquiring the first sample image in RAW format and the first ground truth image in RGB format, the electronic device can use an interpolation algorithm to process the first sample image to obtain a noisy fifth sample image in RGB format, and use an interpolation algorithm to process the first ground truth image to obtain a noiseless fourth ground truth image in RGB format.

[0177] In another implementation, the electronic device can also use the trained conversion network to process the first sample image in RAW format to obtain a noisy fifth sample image in RGB format; and use the conversion network to process the first ground truth image to obtain a noiseless fourth ground truth image in RGB format.

[0178] The electronic device can input the fifth sample image into the second denoising network with the initial structure, process the fifth sample image through the initial network parameters in the second denoising network to obtain the seventh sample image in RGB format; then calculate the difference between the seven sample image and the fourth ground truth image at the same pixel position in the RGB three channels; for each same pixel position, calculate the difference between each of the three RGB channels at that same pixel position; then calculate the average of the differences of the three channels to obtain the difference at that same pixel position; then calculate the average of the differences at each same pixel position to obtain the fourth loss value.

[0179] The fourth loss value represents the difference between the seventh sample image and the fourth ground truth image. Adjusting the network parameters of the second denoising network according to the fourth loss value makes the image output by the adjusted second denoising network closer to the ground truth image, thus obtaining the trained second denoising network. The feature extraction layer in the trained second denoising network can also extract feature vectors that can more accurately represent the color distribution and spatial structure of the image, thereby improving the accuracy of the perceptual loss value and improving the denoising effect of the image denoising model trained based on the perceptual loss value.

[0180] Figure 5 This is a flowchart illustrating the first image denoising method provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the method includes steps S501-S504.

[0181] S501: Obtain the first image to be denoised in the original RAW format.

[0182] S502, the first image to be denoised is input into the first denoising network in the pre-trained image denoising model to obtain the second image to be denoised.

[0183] The image denoising model was trained using the image denoising model training method provided in the aforementioned embodiments.

[0184] S503, input the second image to be denoised into the conversion network in the image denoising model to obtain the third image to be denoised in RGB format.

[0185] S504, input the third image to be denoised into the second denoising network in the image denoising model to obtain the denoised RGB format image corresponding to the first image to be denoised.

[0186] It is understandable that the electronic device that applies the image denoising method and the electronic device that applies the image denoising model training method can be the same device or two separate devices.

[0187] Electronic devices can acquire the first image to be denoised in the original RAW format. For example, the electronic device can be an image acquisition device that can acquire the first image to be denoised in RAW format; the electronic device can also be a desktop computer or a laptop computer that can acquire the first image to be denoised in RAW format acquired by the image acquisition device.

[0188] Then, the electronic device can input the first image to be denoised into the first denoising network to obtain the second image to be denoised in RAW format; input the second image to be denoised in RAW format into the conversion network to obtain the third image to be denoised in RGB format; and then input the third image to be denoised into the second denoising network for further denoising processing to obtain the denoised RGB format image corresponding to the first image to be denoised.

[0189] It is understandable that each network in the trained image denoising model can be used independently. For example, an electronic device can use a first denoising network to denoise a RAW format image; an electronic device can use a conversion network to process a RAW format image to obtain an RGB format image; and an electronic device can use a second denoising network to denoise an RGB format image.

[0190] In this embodiment of the invention, since the image denoising model is trained based on the aforementioned image denoising model training method, the first denoising network is trained based on the first loss value and the perceptual loss value as the second loss value. The first denoising network can more accurately denoise RAW format images and improve the image quality of RAW format images.

[0191] exist Figure 5Based on the image denoising method shown, the transformation network includes: an input layer, a downsampling sub-network, an upsampling sub-network, and an output layer, as follows: Figure 6 As shown, step S503 includes steps S5031-S5034.

[0192] S5031, input the second image to be denoised into the input layer to obtain the sixth feature vector.

[0193] S5032, input the sixth feature vector into the downsampling subnetwork to obtain the seventh feature vector.

[0194] S5033, based on the seventh feature vector and the upsampling sub-network, obtains the eighth feature vector.

[0195] S5034 inputs the concatenation result of the sixth and eighth feature vectors into the output layer to obtain the third image to be denoised in RGB format.

[0196] It is understandable that the specific implementation of processing the second image to be denoised based on the structure of each layer in the conversion network is similar to the specific implementation of processing the fourth sample image based on the structure of each layer in the conversion network in the aforementioned embodiment, and will not be elaborated on here.

[0197] In one implementation, the downsampling subnetwork includes a first convolutional layer and a second convolutional layer, and the upsampling subnetwork includes a third convolutional layer and a fourth convolutional layer; the kernel size and stride of the first convolutional layer, the second convolutional layer, the third convolutional layer and the fourth convolutional layer are the same, S5032 includes steps S5032a-S5032b, and S5033 includes steps S5033a-S5033b.

[0198] S5032a, the sixth feature vector is input into the first convolutional layer to obtain the fourth sub-feature vector.

[0199] S5032b inputs the fourth sub-feature vector into the second convolutional layer to obtain the fifth sub-feature vector.

[0200] S5033a, the fifth sub-feature vector is input into the third convolutional layer to obtain the sixth sub-feature vector.

[0201] S5033b inputs the concatenation result of the fourth and sixth sub-feature vectors into the fourth convolutional layer to obtain the eighth feature vector.

[0202] It is understandable that the specific implementation of processing the sixth feature vector based on the downsampling sub-network and the upsampling sub-network is similar to the specific implementation of processing the third feature vector based on the downsampling sub-network and the upsampling sub-network in the previous embodiment, and will not be elaborated on here.

[0203] Figure 7 This is a schematic diagram of a conversion network provided in an embodiment of the present invention. Figure 7 As shown, the conversion network includes an input layer, a downsampling sub-network, an upsampling sub-network, and an output layer. The downsampling sub-network contains a first convolutional layer and a second convolutional layer, while the upsampling sub-network contains a third convolutional layer and a fourth convolutional layer. If the input data is a second image to be denoised, after processing by the conversion network, a third image to be denoised in RGB format can be obtained (i.e., the output data). The output of the input layer is concatenated with the output of the upsampling sub-network using a skip connection, and the concatenated result serves as the input to the output layer. Similarly, the output of the first convolutional layer of the downsampling sub-network is concatenated with the third convolutional layer of the upsampling sub-network using a skip connection, and the concatenated result serves as the input to the fourth convolutional layer of the upsampling sub-network.

[0204] 'a' represents a convolutional layer with a stride of 1 and its corresponding activation layer, 'b' represents a convolutional layer with a stride of 2 and its corresponding activation layer, 'c' represents a convolutional layer with a stride of 2, and 'd' represents a convolutional layer with a stride of 1. The input layer contains multiple 'a's, the first convolutional layer contains one 'b' and multiple 'a's, the second convolutional layer contains one 'b' and multiple 'a's, the third convolutional layer contains one 'c' and multiple 'a's, and the fourth convolutional layer contains one 'c' and multiple 'a's. The output layer contains multiple 'a's (referred to as the first type of convolutional layer in the aforementioned embodiments) and one 'd' (referred to as the second type of convolutional layer in the aforementioned embodiments).

[0205] Figure 8 This is a schematic diagram illustrating the principle of an image denoising model training method provided in an embodiment of the present invention. Figure 8 As shown, the image denoising model includes a first denoising network (Denoise), a transformation network (Demosaic), and a second denoising network. The electronic device can acquire a noisy first sample image in RAW format and a noiseless first ground truth image in RAW format with the same image content as the first sample image; then, input the first sample image into the first denoising network to obtain a second sample image in RAW format; then, input the second sample image into the transformation network to obtain a third sample image in RGB format; and input the first ground truth image into the transformation network to obtain a second ground truth image in RGB format; then, input the third sample image into the second denoising network to obtain a first feature vector output by the feature extraction layer in the second denoising network; and finally, input the second ground truth image into the second denoising network to obtain a second feature vector output by the feature extraction layer.

[0206] Then, during the loss calculation process, the electronic device can calculate the difference between the second sample image and the first ground truth image to obtain the first loss value; and calculate the difference between the first feature vector and the second feature vector to obtain the second loss value; and combine the first loss value and the second loss value to adjust the network parameters of the first denoising network to obtain the trained image denoising model.

[0207] Figure 9 This is a schematic diagram of the structure of an image denoising model training device provided in an embodiment of the present invention, as shown below. Figure 9 As shown, the device includes:

[0208] The first acquisition module 910 is used to acquire a noisy first sample image in the original RAW format and a noiseless first ground truth image in RAW format that has the same image content as the first sample image.

[0209] The first input module 920 is used to input the first sample image into the first denoising network of the image denoising model of the initial structure to obtain the second sample image in RAW format; the image denoising model further includes a transformation network and a second denoising network.

[0210] The second input module 930 is used to input the second sample image into the conversion network to obtain a third sample image in RGB format; and to input the first ground truth image into the conversion network to obtain a second ground truth image in RGB format.

[0211] The third input module 940 is used to input the third sample image into the second denoising network to obtain a first feature vector output by the feature extraction layer in the second denoising network; and to input the second ground truth image into the second denoising network to obtain a second feature vector output by the feature extraction layer.

[0212] The calculation module 950 is used to calculate the difference between the second sample image and the first ground truth image to obtain a first loss value; and to calculate the difference between the first feature vector and the second feature vector to obtain a second loss value;

[0213] The adjustment module 960 is used to adjust the network parameters of the first denoising network by combining the first loss value and the second loss value, so as to obtain the trained image denoising model.

[0214] Optionally, the conversion network is pre-trained based on a fourth sample image in RAW format and a third ground truth image in RGB format that has the same image content as the fourth sample image; when the network parameters of the first noise reduction network are adjusted by combining the first loss value and the second loss value, the network parameters of the conversion network are frozen.

[0215] And / or,

[0216] The second denoising network is pre-trained based on a noisy fifth sample image in RGB format and a noiseless fourth ground truth image in RGB format that has the same content as the fifth sample image; when the network parameters of the first denoising network are adjusted by combining the first loss value and the second loss value, the network parameters of the second denoising network are frozen.

[0217] Optionally, the conversion network is trained through the following steps: acquiring a fourth sample image in RAW format and a third ground truth image in RGB format with the same image content as the fourth sample image; inputting the fourth sample image into the conversion network with the initial structure to obtain a sixth sample image in RGB format; calculating a third loss value based on the difference between the sixth sample image and the third ground truth image in the three RGB channels; and adjusting the network parameters of the conversion network based on the third loss value to obtain the trained conversion network.

[0218] Optionally, the conversion network includes: an input layer, a downsampling sub-network, an upsampling sub-network, and an output layer;

[0219] The step of inputting the fourth sample image into the initial structure of the transformation network to obtain the sixth sample image in RGB format includes: inputting the fourth sample image into the input layer to obtain a third feature vector; inputting the third feature vector into the downsampling sub-network to obtain a fourth feature vector; obtaining a fifth feature vector based on the fourth feature vector and the upsampling sub-network; and inputting the concatenation result of the third feature vector and the fifth feature vector into the output layer to obtain the sixth sample image in RGB format.

[0220] Optionally, the downsampling sub-network includes a first convolutional layer and a second convolutional layer, and the upsampling sub-network includes a third convolutional layer and a fourth convolutional layer; the kernel size and stride of the first, second, third, and fourth convolutional layers are all the same; the step of inputting the third feature vector into the downsampling sub-network to obtain the fourth feature vector includes: inputting the third feature vector into the first convolutional layer to obtain a first sub-feature vector; inputting the first sub-feature vector into the second convolutional layer to obtain a second sub-feature vector; the step of obtaining the fifth feature vector based on the fourth feature vector and the upsampling sub-network includes: inputting the second sub-feature vector into the third convolutional layer to obtain a third sub-feature vector; inputting the concatenation result of the first and third sub-feature vectors into the fourth convolutional layer to obtain the fifth feature vector.

[0221] Optionally, the second denoising network is trained through the following steps: acquiring a noisy fifth sample image in RGB format and a noiseless fourth ground truth image in RGB format with the same image content as the fifth sample image; inputting the fifth sample image into the second denoising network with an initial structure to obtain a seventh sample image in RGB format; calculating a fourth loss value based on the difference between the seventh sample image and the fourth ground truth image in the three RGB channels; and adjusting the network parameters of the second denoising network based on the fourth loss value to obtain the trained second denoising network.

[0222] This invention adjusts the network parameters of the first denoising network by combining the first loss value and the second loss value. The second loss value, which is used as a perceptual loss value, can assist the first loss value. This allows for parameter tuning of the first denoising network used for denoising RAW format images, thereby improving the denoising effect of the trained first denoising network. This enables the training of a model for denoising RAW format images based on a perceptual loss training method.

[0223] Figure 10 This is a schematic diagram of the structure of an image noise reduction device provided in an embodiment of the present invention, as shown below. Figure 10 As shown, the device includes:

[0224] The second acquisition module 1010 is used to acquire the first image to be denoised in the original RAW format;

[0225] The fourth input module 1020 is used to input the first image to be denoised into the first denoising network in the pre-trained image denoising model to obtain the second image to be denoised; wherein, the image denoising model is trained based on the above-mentioned image denoising model training method;

[0226] The fifth input module 1030 is used to input the second image to be denoised into the conversion network in the image denoising model to obtain a third image to be denoised in RGB format;

[0227] The sixth input module 1040 is used to input the third image to be denoised into the second denoising network in the image denoising model to obtain the denoised RGB format image corresponding to the first image to be denoised.

[0228] Optionally, the conversion network includes: an input layer, a downsampling sub-network, an upsampling sub-network, and an output layer;

[0229] The fifth input module 1030 includes: a first input unit for inputting the second image to be denoised into the input layer to obtain a sixth feature vector; a second input unit for inputting the sixth feature vector into the downsampling sub-network to obtain a seventh feature vector; a third input unit for obtaining an eighth feature vector based on the seventh feature vector and the upsampling sub-network; and a fourth input unit for inputting the concatenation result of the sixth and eighth feature vectors into the output layer to obtain a third image to be denoised in RGB format.

[0230] Optionally, the downsampling sub-network includes a first convolutional layer and a second convolutional layer, and the upsampling sub-network includes a third convolutional layer and a fourth convolutional layer; the kernel size and stride of the first, second, third, and fourth convolutional layers are all the same; the second input unit is specifically used to input the sixth feature vector into the first convolutional layer to obtain a fourth sub-feature vector; input the fourth sub-feature vector into the second convolutional layer to obtain a fifth sub-feature vector; the third input unit is specifically used to input the fifth sub-feature vector into the third convolutional layer to obtain a sixth sub-feature vector; and input the concatenation result of the fourth and sixth sub-feature vectors into the fourth convolutional layer to obtain an eighth feature vector.

[0231] In this embodiment of the invention, since the image denoising model is trained based on the aforementioned image denoising model training method, the first denoising network is trained based on the first loss value and the perceptual loss value as the second loss value. The first denoising network can more accurately denoise RAW format images and improve the image quality of RAW format images.

[0232] This invention also provides an electronic device, such as... Figure 11 As shown, it includes a processor 1101, a communication interface 1102, a memory 1103, and a communication bus 1104, wherein the processor 1101, the communication interface 1102, and the memory 1103 communicate with each other through the communication bus 1104.

[0233] The memory 1103 is used to store computer programs; the processor 1101 is used to execute the program stored in the memory 1103 to implement the image denoising model training method and image denoising method described in the foregoing embodiments.

[0234] The communication bus mentioned in the aforementioned electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned electronic device and other devices. The memory can include Random Access Memory (RAM), or Non-Volatile Memory (NVM), such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0235] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0236] In another embodiment of the present invention, a computer-readable storage medium is provided, which stores a computer program. When executed by a processor, the computer program implements the steps of any of the above-described image denoising model training methods and image denoising methods. In yet another embodiment of the present invention, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to execute any of the above-described image denoising model training methods and image denoising methods.

[0237] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0238] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0239] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0240] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for training an image denoising model, characterized in that, The method includes: Acquire a noisy first sample image in the original RAW format, and a noiseless first ground truth image in RAW format with the same image content as the first sample image; The first sample image is input into the first denoising network of the image denoising model with the initial structure to obtain the second sample image in RAW format; the image denoising model also includes a transformation network and a second denoising network. The second sample image is input into the conversion network to obtain a third sample image in RGB format; and the first ground truth image is input into the conversion network to obtain a second ground truth image in RGB format. The third sample image is input into the second denoising network to obtain the first feature vector output by the feature extraction layer in the second denoising network; and the second ground truth image is input into the second denoising network to obtain the second feature vector output by the feature extraction layer. The difference between the second sample image and the first ground truth image is calculated to obtain a first loss value; and the difference between the first feature vector and the second feature vector is calculated to obtain a second loss value. By combining the first loss value and the second loss value, the network parameters of the first denoising network are adjusted to obtain the trained image denoising model.

2. The method according to claim 1, characterized in that, The conversion network is pre-trained based on a fourth sample image in RAW format and a third ground truth image in RGB format that has the same image content as the fourth sample image. When adjusting the network parameters of the first noise reduction network by combining the first loss value and the second loss value, the network parameters of the conversion network are frozen. And / or, The second noise reduction network is pre-trained based on a noisy fifth sample image in RGB format and a noiseless fourth ground truth image in RGB format that has the same content as the fifth sample image. When adjusting the network parameters of the first noise reduction network by combining the first loss value and the second loss value, the network parameters of the second noise reduction network are frozen.

3. The method according to claim 2, characterized in that, The conversion network is trained through the following steps: Obtain a fourth sample image in RAW format, and a third ground truth image in RGB format that has the same image content as the fourth sample image; The fourth sample image is input into the transformation network of the initial structure to obtain the sixth sample image in RGB format; The third loss value is calculated based on the differences between the sixth sample image and the third ground truth image in the three RGB channels; Based on the third loss value, the network parameters of the transformation network are adjusted to obtain the trained transformation network.

4. The method according to claim 3, characterized in that, The conversion network includes: an input layer, a downsampling sub-network, an upsampling sub-network, and an output layer; The step of inputting the fourth sample image into the transformation network of the initial structure to obtain the sixth sample image in RGB format includes: The fourth sample image is input into the input layer to obtain the third feature vector; The third feature vector is input into the downsampling sub-network to obtain the fourth feature vector; Based on the fourth feature vector and the upsampled sub-network, the fifth feature vector is obtained; The concatenation result of the third and fifth feature vectors is input into the output layer to obtain the sixth sample image in RGB format.

5. The method according to claim 4, characterized in that, The downsampling subnetwork includes a first convolutional layer and a second convolutional layer, and the upsampling subnetwork includes a third convolutional layer and a fourth convolutional layer; the kernel size and stride of the first convolutional layer, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer are all the same; The step of inputting the third feature vector into the downsampling sub-network to obtain the fourth feature vector includes: The third feature vector is input into the first convolutional layer to obtain the first sub-feature vector; The first sub-feature vector is input into the second convolutional layer to obtain the second sub-feature vector; The process of obtaining the fifth feature vector based on the fourth feature vector and the upsampling sub-network includes: The second sub-feature vector is input into the third convolutional layer to obtain the third sub-feature vector; The concatenation result of the first and third sub-feature vectors is input into the fourth convolutional layer to obtain the fifth feature vector.

6. The method according to claim 2, characterized in that, The second noise reduction network is trained through the following steps: Obtain a noisy fifth sample image in RGB format, and a noiseless fourth ground truth image in RGB format that has the same image content as the fifth sample image. The fifth sample image is input into the second noise reduction network of the initial structure to obtain the seventh sample image in RGB format; The fourth loss value is calculated based on the differences between the seventh sample image and the fourth ground truth image in the three RGB channels; Based on the fourth loss value, the network parameters of the second noise reduction network are adjusted to obtain the trained second noise reduction network.

7. An image denoising method, characterized in that, The method includes: Obtain the first image to be denoised in its original RAW format; The first image to be denoised is input into the first denoising network of a pre-trained image denoising model to obtain a second image to be denoised; wherein the image denoising model is trained based on the method described in any one of claims 1-6. The second image to be denoised is input into the conversion network in the image denoising model to obtain a third image to be denoised in RGB format; The third image to be denoised is input into the second denoising network in the image denoising model to obtain the denoised RGB format image corresponding to the first image to be denoised.

8. The method according to claim 7, characterized in that, The conversion network includes: an input layer, a downsampling sub-network, an upsampling sub-network, and an output layer; The step of inputting the second image to be denoised into the conversion network of the image denoising model to obtain the third image to be denoised in RGB format includes: The second image to be denoised is input into the input layer to obtain the sixth feature vector; The sixth feature vector is input into the downsampling sub-network to obtain the seventh feature vector; Based on the seventh feature vector and the upsampling sub-network, the eighth feature vector is obtained; The concatenation result of the sixth and eighth feature vectors is input into the output layer to obtain the third image to be denoised in RGB format.

9. The method according to claim 8, characterized in that, The downsampling subnetwork includes a first convolutional layer and a second convolutional layer, and the upsampling subnetwork includes a third convolutional layer and a fourth convolutional layer; the kernel size and stride of the first convolutional layer, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer are all the same; The step of inputting the sixth feature vector into the downsampling sub-network to obtain the seventh feature vector includes: The sixth feature vector is input into the first convolutional layer to obtain the fourth sub-feature vector; The fourth sub-feature vector is input into the second convolutional layer to obtain the fifth sub-feature vector; The process of obtaining the eighth feature vector based on the seventh feature vector and the upsampling sub-network includes: The fifth sub-feature vector is input into the third convolutional layer to obtain the sixth sub-feature vector; The concatenation result of the fourth sub-feature vector and the sixth sub-feature vector is input into the fourth convolutional layer to obtain the eighth feature vector.

10. An image denoising model training device, characterized in that, The device includes: The first acquisition module is used to acquire a noisy first sample image in the original RAW format and a noiseless first ground truth image in RAW format that has the same image content as the first sample image. The first input module is used to input the first sample image into the first denoising network of the image denoising model with the initial structure to obtain the second sample image in RAW format; the image denoising model also includes a transformation network and a second denoising network. The second input module is used to input the second sample image into the conversion network to obtain a third sample image in RGB format; and to input the first ground truth image into the conversion network to obtain a second ground truth image in RGB format. The third input module is used to input the third sample image into the second denoising network to obtain the first feature vector output by the feature extraction layer in the second denoising network; and to input the second ground truth image into the second denoising network to obtain the second feature vector output by the feature extraction layer. The calculation module is used to calculate the difference between the second sample image and the first ground truth image to obtain a first loss value; and to calculate the difference between the first feature vector and the second feature vector to obtain a second loss value; The adjustment module is used to adjust the network parameters of the first denoising network by combining the first loss value and the second loss value, so as to obtain the trained image denoising model.

11. An image noise reduction device, characterized in that, The device includes: The second acquisition module is used to acquire the first image to be denoised in the original RAW format; The fourth input module is used to input the first image to be denoised into the first denoising network in the pre-trained image denoising model to obtain the second image to be denoised; wherein the image denoising model is trained based on the method described in any one of claims 1-6. The fifth input module is used to input the second image to be denoised into the conversion network in the image denoising model to obtain the third image to be denoised in RGB format; The sixth input module is used to input the third image to be denoised into the second denoising network in the image denoising model to obtain the denoised RGB format image corresponding to the first image to be denoised.

12. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-9.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-9.

14. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1-9.

Images