Sample image generation method and device, model training method and device, equipment and medium

Abstract

The embodiment of the invention provides a sample image generation method and device, a model training method and device, equipment and a medium. The method comprises the steps of obtaining a noiseless image and a target image shot by a camera; predicting noise parameters of the camera based on the features of the target image; adding shot noise, readout noise and row noise in the noiseless image according to the noise parameters to obtain a noise image; and determining the noiseless image and the noise image as sample images corresponding to the camera. The electronic equipment can realize automatic generation of the sample image by predicting the noise parameter and adding the noise based on the predicted noise parameter, and the efficiency of obtaining the sample image is improved. Moreover, a large number of noise images can be generated on the basis of a public noiseless image only through one image shot by a camera, and the cost of obtaining a sample image is reduced. In addition, a noise image generated by adding the most important noise in three types of real images including shot noise, readout noise and row noise can be well fit with a real noise image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to sample image generation methods, model training methods, devices, equipment and media. Background Technology

[0002] The rapid development of mobile phones, cameras, and other devices has brought great convenience to people's lives. However, these devices inevitably introduce noise into images during the image acquisition process, thus affecting the final image quality. Currently, the main noise reduction method is to process images based on pre-trained image noise reduction models.

[0003] To improve the accuracy of image denoising models, it is necessary to manually collect a large number of images from various devices, and then generate a training sample set for the image denoising model based on the collected images. However, manually collecting images from devices is not only inefficient, but also consumes a lot of manpower and material resources. Summary of the Invention

[0004] The purpose of this application is to provide a sample image generation method, model training method, apparatus, device, and medium to improve the efficiency of acquiring sample images and reduce the cost of acquiring sample images. The specific technical solution is as follows:

[0005] In a first aspect, embodiments of this application provide a method for generating a sample image, the method comprising:

[0006] Acquire noise-free images and target images captured by the camera;

[0007] Extract features from the target image, and predict the first noise parameter of the camera based on the extracted features;

[0008] Based on the first noise parameter, shot noise, readout noise, and line noise are added to the noise-free image to obtain a noisy image;

[0009] The noise-free image and the noisy image are identified as sample images corresponding to the camera.

[0010] Optionally, the first noise parameter includes: camera gain, readout noise parameter, and line noise parameter;

[0011] The step of adding shot noise, readout noise, and line noise to the noise-free image according to the first noise parameter to obtain a noisy image includes:

[0012] From the noise distribution corresponding to each pixel of the noise-free image, the noise value corresponding to each pixel is randomly sampled to obtain the first noise image;

[0013] The pixel values ​​of each pixel in the first noisy image are adjusted according to the camera gain to obtain a second noisy image after adding shot noise to the noise-free image.

[0014] Based on the expected value and standard deviation included in the readout noise parameters, a first Gaussian distribution is determined, and a readout noise image whose distribution conforms to the first Gaussian distribution is generated, wherein the readout noise image has the same size as the noise-free image;

[0015] Based on the standard deviation included in the row noise parameters, a second Gaussian distribution is determined, and a row noise image is generated in which the distribution of row noise in each row conforms to the second Gaussian distribution, wherein the expectation of the second Gaussian distribution is zero, and the row noise image has the same size as the noise-free image.

[0016] The second noise image, the readout noise image, and the row noise image are added together to obtain a noise image corresponding to the noise-free image.

[0017] Optionally, the noise distribution for each pixel can be generated as follows:

[0018] Calculate the ratio between the pixel value of a pixel and the camera gain, and use the ratio as the parameter of a Poisson distribution to construct a Poisson distribution as the noise distribution corresponding to the pixel.

[0019] The step of adjusting the pixel values ​​of each pixel in the first noisy image according to the camera gain to obtain a second noisy image after adding shot noise to the noise-free image includes:

[0020] The pixel value of each pixel in the first noisy image is multiplied by the camera gain to obtain a second noisy image after adding shot noise to the noise-free image.

[0021] Optionally, predicting the first noise parameter of the camera based on the extracted features includes:

[0022] Based on the extracted features, the noise distribution corresponding to the noise in the target image is determined;

[0023] The readout noise parameters and line noise parameters of the camera are determined based on the noise distribution. The noise distribution is the result of the superposition of the first noise distribution corresponding to the readout noise parameters and the second noise distribution corresponding to the line noise parameters. All noise in the first noise distribution conforms to a Gaussian distribution constructed with the readout noise parameters, and the distribution of each line noise in the second noise distribution conforms to a Gaussian distribution constructed with the line noise parameters.

[0024] The camera gain of the camera is determined based on the pixel values ​​of the pixels reflected by the extracted features.

[0025] Optionally, determining the camera gain based on the pixel values ​​of the pixels reflected by the extracted features includes:

[0026] Based on the pixel values ​​of the extracted features, the grayscale histogram corresponding to the target image is determined.

[0027] The camera gain is determined based on the distribution of gray values ​​in the gray-scale histogram. The relationship between the distribution of gray values ​​and the camera gain is that the more the gray values ​​are distributed towards lower gray values, the lower the camera gain; and the more the gray values ​​are distributed towards higher gray values, the higher the camera gain.

[0028] Optionally, the step of extracting features from the target image, predicting a first noise parameter of the camera based on the extracted features, and adding shot noise, readout noise, and line noise to the noise-free image according to the first noise parameter to obtain a noisy image includes:

[0029] The noise-free image and the target image are input into the noise image generation model to obtain the noise image output by the noise image generation model. The noise image is an image obtained by the noise image generation model by adding shot noise, readout noise and line noise to the noise-free image according to a first noise parameter. The first noise parameter is the noise parameter of the camera predicted by the noise image generation model based on the features of the extracted target image.

[0030] Optionally, the training method for the noisy image generation model includes:

[0031] Obtain noise-free images and noise parameters of the samples;

[0032] Based on the sample noise parameters, shot noise, readout noise, and row noise are added to the sample noise-free image to obtain a sample noise image;

[0033] The sample noise image is input into the feature extraction layer of the noise image generation model to obtain the sample features of the sample noise image extracted by the feature extraction layer. The noise image generation model further includes a noise parameter prediction layer and an image generation layer.

[0034] The sample features are input into the noise parameter prediction layer to obtain the second noise parameter predicted by the noise parameter prediction layer based on the sample features.

[0035] The noise-free image of the sample and the second noise parameter are input into the image generation layer to obtain a comparison noise image obtained by the image generation layer adding shot noise, readout noise and row noise to the noise-free image of the sample according to the second noise parameter.

[0036] The parameters of the noise parameter prediction layer are adjusted based on the difference between the sample noise image and the comparison noise image.

[0037] Secondly, embodiments of this application provide a method for training a noise reduction model, the method comprising:

[0038] Obtain the sample image corresponding to the camera generated by any of the methods described in the first aspect above;

[0039] The image denoising model is trained based on the sample images.

[0040] Optionally, the image denoising model is the RRDB model.

[0041] Thirdly, embodiments of this application provide a sample image generation apparatus, the apparatus comprising:

[0042] The image acquisition module is used to acquire noise-free images and target images captured by the camera;

[0043] A noise parameter prediction module is used to extract features from the target image and predict the first noise parameter of the camera based on the extracted features.

[0044] A noise image generation module is used to add shot noise, readout noise, and line noise to the noise-free image according to the first noise parameter to obtain a noise image;

[0045] The sample image determination module is used to determine the noise-free image and the noisy image as sample images corresponding to the camera.

[0046] Optionally, the first noise parameter includes: camera gain, readout noise parameter, and line noise parameter; the noise image generation module includes: a first image generation unit, used to randomly sample the noise value corresponding to each pixel from the noise distribution corresponding to each pixel of the noise-free image to obtain a first noise image; a second image generation unit, used to adjust the pixel value of each pixel of the first noise image according to the camera gain to obtain a second noise image after adding shot noise to the noise-free image; and a third image generation unit, used to determine a first high-resolution image based on the expectation and standard deviation included in the readout noise parameter. The first Gaussian distribution is used to generate a readout noise image whose readout noise distribution conforms to the first Gaussian distribution, wherein the readout noise image has the same size as the noise-free image; the fourth image generation unit is used to determine a second Gaussian distribution based on the standard deviation included in the row noise parameters, and generate a row noise image whose row noise distribution conforms to the second Gaussian distribution, wherein the expectation of the second Gaussian distribution is zero, and the row noise image has the same size as the noise-free image; the image overlay module is used to add the second noise image, the readout noise image and the row noise image to obtain a noise image corresponding to the noise-free image.

[0047] Optionally, the noise distribution corresponding to each pixel is generated in the following manner: the ratio between the pixel value of the pixel and the camera gain is calculated, and the ratio is used as the parameter of the Poisson distribution to construct the Poisson distribution as the noise distribution corresponding to the pixel; the second image generation unit is specifically used to multiply the pixel value of each pixel of the first noise image by the camera gain to obtain the second noise image after adding shot noise to the noise-free image.

[0048] Optionally, the noise parameter prediction module includes: a noise distribution determination unit, configured to determine the noise distribution corresponding to the noise in the target image based on the extracted features; a noise parameter determination unit, configured to determine the readout noise parameter and row noise parameter of the camera according to the noise distribution, wherein the superposition result of the first noise distribution corresponding to the readout noise parameter and the second noise distribution corresponding to the row noise parameter is the noise distribution, the distribution of all noise in the first noise distribution conforms to the Gaussian distribution constructed with the readout noise parameter, and the distribution of each row noise in the second noise distribution conforms to the Gaussian distribution constructed with the row noise parameter; and a camera gain determination unit, configured to determine the camera gain of the camera based on the pixel values ​​of the pixels reflected by the extracted features.

[0049] Optionally, the camera gain determination unit is specifically used to determine the grayscale histogram corresponding to the target image based on the pixel values ​​of the pixels reflected by the extracted features; and to determine the camera gain of the camera based on the distribution of grayscale values ​​in the grayscale histogram, wherein the correspondence between the distribution of grayscale values ​​and the camera gain is that the more the distribution of grayscale values ​​is biased towards lower grayscale values, the lower the camera gain, and the more the distribution of grayscale values ​​is biased towards higher grayscale values, the higher the camera gain.

[0050] Optionally, the noise parameter prediction module is specifically used to input the target image into the noise image generation model, extract features of the target image based on the noise image generation model, and predict the first noise parameter of the camera based on the extracted features; the noise image generation module is specifically used to input the noise-free image into the noise image generation model, and add shot noise, readout noise, and line noise to the noise-free image based on the first noise parameter to obtain a noise image.

[0051] Optionally, the training method of the noise image generation model includes: acquiring a sample noise-free image and sample noise parameters; adding shot noise, readout noise, and row noise to the sample noise-free image according to the sample noise parameters to obtain a sample noise image; inputting the sample noise image into the feature extraction layer of the noise image generation model to obtain sample features of the sample noise image extracted by the feature extraction layer, wherein the noise image generation model further includes: a noise parameter prediction layer and an image generation layer; inputting the sample features into the noise parameter prediction layer to obtain a second noise parameter predicted by the noise parameter prediction layer based on the sample features; inputting the sample noise-free image and the second noise parameter into the image generation layer to obtain a comparison noise image obtained by the image generation layer adding shot noise, readout noise, and row noise to the sample noise-free image according to the second noise parameter; and adjusting the parameters of the noise parameter prediction layer based on the difference between the sample noise image and the comparison noise image.

[0052] Fourthly, embodiments of this application provide a noise reduction model training device, the device comprising:

[0053] The sample image acquisition module is used to acquire a sample image corresponding to the camera generated based on any of the methods described in the first aspect above.

[0054] The model training module is used to train an image denoising model based on the sample images.

[0055] Optionally, the image denoising model is the RRDB model.

[0056] Fifthly, embodiments of this application provide an electronic device, including:

[0057] Memory, used to store computer programs;

[0058] When a processor executes a program stored in memory, it implements the method described in the first aspect or the method described in the second aspect.

[0059] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method described in the first aspect or the method described in the second aspect.

[0060] Beneficial effects of the embodiments in this application:

[0061] In the solution provided in this application embodiment, an electronic device can acquire a noise-free image and a target image captured by a camera; extract features from the target image; predict a first noise parameter of the camera based on the extracted features; add shot noise, readout noise, and line noise to the noise-free image according to the first noise parameter to obtain a noisy image; and determine the noise-free image and the noisy image as sample images corresponding to the camera. For each type of camera, the electronic device can use images captured by that type of camera to predict the noise parameter of that type of camera, and add shot noise, readout noise, and line noise to the noise-free image accordingly to obtain a noisy image corresponding to that type of camera. Finally, the noise-free image and the noisy image are used as sample images corresponding to that type of camera. This method of predicting noise parameters and adding noise based on the predicted noise parameters can realize the automatic generation of sample images, greatly improving the efficiency of acquiring sample images. Furthermore, for each type of camera, only one image captured by that type of camera is needed to generate a large number of noisy images corresponding to that type of camera based on a large number of noise-free images included in the public dataset, greatly reducing the cost of acquiring sample images. In addition, because the noisy image is generated based on the predicted noise parameter of the camera, the noisy image can well match the actual noise level of the camera. Furthermore, the shot noise, readout noise, and line noise added to the noise-free image are the three most important types of noise in real-world images. This allows the noise image generated by adding these three types of noise to closely match the real noise image.

[0062] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0064] Figure 1 A schematic flowchart illustrating a sample image generation method provided in an embodiment of this application;

[0065] Figure 2 Based on Figure 1 The illustrated embodiment is a flowchart illustrating a method for adding shot noise, readout noise, and line noise to an image.

[0066] Figure 3 Based on Figure 1 A schematic diagram of a method for generating a noisy image using a model, as shown in the embodiment;

[0067] Figure 4 Based on Figure 3 A flowchart illustrating a model training method in the embodiment shown;

[0068] Figure 5 This is a schematic diagram of the structure of a sample image generation device provided in an embodiment of this application;

[0069] Figure 6 This is a schematic diagram of the structure of a noise reduction model training device provided in an embodiment of this application;

[0070] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0071] Figure 8 This is a schematic diagram of the structure of another electronic device provided in an embodiment of this application. Detailed Implementation

[0072] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0073] To improve the efficiency and reduce the cost of acquiring sample images, embodiments of this application provide a sample image generation method, a denoising model training method, a sample image generation apparatus, a denoising model training apparatus, an electronic device, a computer-readable storage medium, and a computer program product. The following first describes a sample image generation method provided by an embodiment of this application.

[0074] The sample image generation method provided in this application can be applied to any electronic device that needs to generate sample images. For example, it can be applied to terminal devices such as mobile phones and computers, as well as servers. No specific limitations are made here; for clarity, it will be referred to as an electronic device.

[0075] like Figure 1 As shown, a sample image generation method includes:

[0076] S101, acquires a noise-free image and a target image captured by the camera.

[0077] S102, extract features from the target image, and predict the first noise parameter of the camera based on the extracted features.

[0078] S103, according to the first noise parameter, add shot noise, readout noise and line noise to the noise-free image to obtain a noisy image.

[0079] S104, the noise-free image and the noisy image are identified as the sample images corresponding to the camera.

[0080] In the solution provided in this application embodiment, an electronic device can acquire a noise-free image and a target image captured by a camera; extract features from the target image; predict a first noise parameter of the camera based on the extracted features; add shot noise, readout noise, and line noise to the noise-free image according to the first noise parameter to obtain a noisy image; and determine the noise-free image and the noisy image as sample images corresponding to the camera. For each type of camera, the electronic device can use images captured by that type of camera to predict the noise parameter of that type of camera, and add shot noise, readout noise, and line noise to the noise-free image accordingly to obtain a noisy image corresponding to that type of camera. Finally, the noise-free image and the noisy image are used as sample images corresponding to that type of camera. This method of predicting noise parameters and adding noise based on the predicted noise parameters can realize the automatic generation of sample images, greatly improving the efficiency of acquiring sample images. Furthermore, for each type of camera, only one image captured by that type of camera is needed to generate a large number of noisy images corresponding to that type of camera based on a large number of noise-free images included in the public dataset, greatly reducing the cost of acquiring sample images. In addition, because the noisy image is generated based on the predicted noise parameter of the camera, the noisy image can well match the actual noise level of the camera. Furthermore, the shot noise, readout noise, and line noise added to the noise-free image are the three most important types of noise in real-world images. This allows the noise image generated by adding these three types of noise to closely match the real noise image.

[0081] When generating sample images, users can either use a camera to capture real-world scenes to obtain target images, or they can acquire noise-free images from public datasets such as ImageNet, depending on their specific needs. Then, users can input both the target image and the noise-free image into an electronic device, allowing the device to obtain both the target image and the noise-free image.

[0082] Of course, electronic devices can also establish a communication connection with cameras. When sample images need to be generated, the electronic device can send image acquisition commands to the camera, controlling the camera to capture images of the real scene and thus obtain the target image returned by the camera. Furthermore, the electronic device can automatically download images as noise-free images from public datasets via pre-set links. In this way, the electronic device can also acquire both noise-free images and target images.

[0083] The two methods listed above for electronic devices to acquire noise-free images and target images captured by cameras are merely examples. Electronic devices can also acquire noise-free images and target images captured by cameras through other methods, which are not specifically limited here. Furthermore, after acquiring the noise-free image and target image, the electronic device can also crop them, for example, cropping both the noise-free image and the target image to a size of 512×512, and normalizing the pixel values ​​in the noise-free image and target image.

[0084] During camera shooting, light is sampled through the lens and image sensor, and the sampled light signals are converted into digital images. Both of these stages can introduce noise. This noise is superimposed on the ideal noise-free image, so that the final image captured by the camera is the target image with this noise superimposed on the ideal noise-free image.

[0085] After acquiring a target image, the electronic device can extract its features and then predict the camera's first noise parameter based on those features. Specifically, the electronic device can acquire a single target image and predict the camera's first noise parameter based on that single image, thus saving on image acquisition costs. Alternatively, the electronic device can acquire multiple target images, predict the camera's first noise parameter for each image separately, and then use the average of the predicted first noise parameters as the final first noise parameter, thereby improving the accuracy of the predicted noise parameter.

[0086] In one implementation, the electronic device can extract features from a target image, wherein the features may include at least one of: grayscale value distribution, histogram statistics, frequency domain characteristics, and texture information. Based on the extracted features, statistical analysis or machine learning methods are used to identify and determine the noise distribution in the image.

[0087] Furthermore, the noise parameters of a camera generally include readout noise parameters, line noise parameters, and camera gain, and the noise distribution in the image is obtained by superimposing the noise distribution corresponding to readout noise and the noise distribution corresponding to line noise.

[0088] Therefore, for readout noise parameters and line noise parameters, the electronic device can predict the camera's readout noise parameters and line noise parameters based on the noise distribution after obtaining the noise distribution.

[0089] To ensure the accuracy of the predicted readout noise parameters and row noise parameters, the electronic device, after obtaining these parameters, can construct a Gaussian distribution using the readout noise parameters and then generate a noise image where the distribution of all noise in the image conforms to this Gaussian distribution. This is the first noise distribution, meaning the first noise distribution conforms to this Gaussian distribution across the entire image. Simultaneously, the electronic device can also construct a Gaussian distribution using the row noise parameters and generate a noise image where the distribution of noise in each row of the image conforms to this Gaussian distribution. This is the second noise distribution, meaning the second noise distribution conforms to this Gaussian distribution across each row. Finally, the electronic device can superimpose the first and second noise distributions and determine if the superposition result is the determined noise distribution. If not, the predicted readout noise parameters and row noise parameters are adjusted until the superposition result of the first noise distribution corresponding to the readout noise parameters and the second noise distribution corresponding to the row noise parameters is the correct noise distribution, thus predicting accurate readout noise parameters and row noise parameters.

[0090] For camera gain, electronic devices can determine the grayscale histogram of an image based on the pixel values ​​reflected by the extracted features. Then, based on the distribution of grayscale values ​​in the grayscale histogram, the camera gain is determined. The correspondence between the distribution of grayscale values ​​and camera gain is as follows: the more the grayscale values ​​are distributed towards lower grayscale values, the lower the camera gain; the more the grayscale values ​​are distributed towards higher grayscale values, the higher the camera gain. In other words, a leftward shift in the grayscale histogram (i.e., more dark pixels) indicates a lower camera gain; conversely, a rightward shift (i.e., more bright pixels) indicates a higher camera gain.

[0091] After obtaining the first noise parameter, the electronic device can add shot noise, readout noise, and line noise to the noise-free image according to the various sub-parameters included in the first noise parameter. The specific methods for adding shot noise, readout noise, and line noise to the noise-free image according to the noise parameter will be described in detail in subsequent embodiments and will not be repeated here.

[0092] The electronic device can acquire only one noise-free image. Based on this, the electronic device can obtain a noisy image through the above steps S102-S103. At this time, the electronic device can determine the noise-free image and the noisy image as an image pair, and then use the image pair as the sample image corresponding to the camera.

[0093] The electronic device can also acquire multiple noise-free images. Based on this, the electronic device can obtain multiple noisy images through the above steps S102-S103. At this time, for each noise-free image, the electronic device can determine the noise-free image and the noisy image obtained by adding shot noise, readout noise and line noise to the noise-free image as an image pair. Then, the determined multiple image pairs are used as sample images corresponding to the camera to generate a large number of sample images corresponding to the camera.

[0094] In the solution provided in this application, for each type of camera, the electronic device can use images captured by that type of camera to predict the noise parameters of that camera, and accordingly add shot noise, readout noise, and line noise to the noise-free image to obtain a noisy image corresponding to that type of camera. Finally, the noise-free image and the noisy image are used as sample images corresponding to that type of camera. This method of predicting noise parameters and adding noise based on the predicted noise parameters can achieve automatic generation of sample images, greatly improving the efficiency of acquiring sample images. Furthermore, for each type of camera, only one image captured by that type of camera is needed to generate a large number of noisy images corresponding to that type of camera based on a large number of noise-free images included in the public dataset, greatly reducing the cost of acquiring sample images. In addition, because the noisy image is generated based on the predicted noise parameters of the camera, the noisy image can well match the actual noise level of the camera. Moreover, the shot noise, readout noise, and line noise added to the noise-free image are the three most important types of noise in real-world images, which allows the noisy image generated by adding these three types of noise to closely match the real noise image.

[0095] As one embodiment of this application, the first noise parameter may include: camera gain, readout noise parameter, and line noise parameter.

[0096] Correspondingly, such as Figure 2 As shown, step S103 above may include:

[0097] S201, randomly sample the noise value corresponding to each pixel from the noise distribution corresponding to each pixel of the noise-free image to obtain the first noise image.

[0098] For each pixel in a noise-free image, an electronic device can determine the noise distribution corresponding to each pixel in, but is not limited to, the following two methods:

[0099] In the first implementation, due to the quantum nature of light, the number of photons collected by the sensor is indeed unstable. This uncertainty leads to shot noise, an unavoidable noise source in photoelectric detection. Shot noise is related to the random arrival of photons; the arrival of each photon is an independent event, following a Poisson distribution.

[0100] Therefore, for each pixel in a noise-free image, the electronic device can calculate the ratio between the pixel value and the camera gain, and then use the ratio as a parameter of the Poisson distribution to construct the noise distribution corresponding to that pixel.

[0101] For example, assuming the predicted camera gain is 1.5, for the first pixel in a noise-free image, its pixel values ​​are R=39, G=156, and B=93. The electronic device can then calculate the ratios of the R, G, and B values ​​of the first pixel to the camera gain, obtaining 26, 104, and 62 respectively. Furthermore, the electronic device can construct a first Poisson distribution using 26 as the parameter, a second Poisson distribution using 104 as the parameter, and a third Poisson distribution using 62 as the parameter. These first, second, and third Poisson distributions represent the noise distribution corresponding to that pixel.

[0102] In the second implementation, in some cases, a Gaussian distribution with the same variance can approximate a Poisson distribution well. Therefore, for each pixel in a noise-free image, the electronic device can calculate the ratio between the pixel value and the camera gain, and then use the ratio as the mean and variance of the Gaussian distribution to construct a Gaussian distribution as the noise distribution corresponding to that pixel.

[0103] For example, assuming the predicted camera gain is 1.5, for the first pixel in a noise-free image, its pixel values ​​are R=39, G=156, and B=93. The electronic device can then calculate the ratios of the R, G, and B values ​​of the first pixel to the camera gain, obtaining 26, 104, and 62 respectively. Furthermore, the electronic device can construct a first Gaussian distribution using 26 as the mean and variance, a second Gaussian distribution using 104 as the mean and variance, and a third Gaussian distribution using 62 as the mean and variance. These first, second, and third Gaussian distributions represent the noise distribution corresponding to that pixel.

[0104] After obtaining the noise distribution corresponding to each pixel of a noise-free image, the electronic device can randomly sample the noise value corresponding to each pixel from the noise distribution corresponding to each pixel of the noise-free image to obtain a first noise image. The pixel value of each pixel in the first noise image is the sampled noise value corresponding to each pixel.

[0105] Taking the noise distribution corresponding to the aforementioned pixels as a Poisson distribution as an example, for the first pixel in a noise-free image, the electronic device can sample the R value from the first Poisson distribution, the G value from the second Poisson distribution, and the B value from the third Poisson distribution, and then determine the sampled RGB values ​​as the noise value corresponding to the first pixel. For other pixels in the noise-free image, the electronic device can perform a similar operation to obtain the first noisy image.

[0106] S202, the pixel values ​​of each pixel in the first noisy image are adjusted according to the camera gain to obtain a second noisy image after adding shot noise to the noise-free image.

[0107] Because when constructing the noise distribution corresponding to the pixels of a noise-free image, the ratio between the pixel value of the pixel and the camera gain is used as the parameter of the noise distribution, in order to ensure that the pixel values ​​of the finally generated noise image match those of the noise-free image, after obtaining the first noise image, the electronic device can multiply the pixel value of each pixel of the first noise image with the camera gain to obtain the second noise image after adding shot noise to the noise-free image.

[0108] When the noise distribution used in step S201 is a Poisson distribution, the resulting second noise image can be represented as:

[0109] C = P(I / K) × K;

[0110] Where C represents the second noisy image, I represents the noise-free image, K represents the camera gain, and P(I / K) represents the Poisson distribution constructed using "I / K" as the parameters of the Poisson distribution.

[0111] S203, based on the expectation and standard deviation included in the readout noise parameters, determine the first Gaussian distribution, and generate a readout noise image whose distribution conforms to the first Gaussian distribution.

[0112] After obtaining the readout noise parameters, the electronic device can construct a first Gaussian distribution by determining the expected and standard deviation of the readout noise included in the parameters as the expected and standard deviation of the first Gaussian distribution, respectively. The electronic device can then generate a readout noise image with the same size as the noise-free image, where the noise in the readout noise image conforms to the first Gaussian distribution over the entire image.

[0113] Readout noise in a readout noise image can be represented as:

[0114] Nread~N(μ,σ read 2 );

[0115] Where Nread represents the readout noise in the readout noise image, μ represents the expected readout noise, and σ read This represents the standard deviation of the readout noise.

[0116] S204. Based on the standard deviation included in the line noise parameters, determine the second Gaussian distribution and generate a line noise image whose line noise distribution conforms to the second Gaussian distribution.

[0117] After obtaining the row noise parameters, the electronic device can define 0 as the expected value of the second Gaussian distribution and determine the standard deviation of the second Gaussian distribution by defining the standard deviation included in the row noise parameters, thereby constructing the second Gaussian distribution. The electronic device can then generate a row noise image with the same size as the noise-free image, where the row noise distribution conforms to the second Gaussian distribution. Specifically, the noise in the row noise image conforms to the second Gaussian distribution in each row; that is, the noise in the first row conforms to the second Gaussian distribution, the noise in the second row also conforms to the second Gaussian distribution, and so on, until the noise in the Nth row also conforms to the second Gaussian distribution.

[0118] Line noise in a line noise image can be represented as:

[0119] Nrow~N(0,σ row 2 );

[0120] Where Nrow represents the row noise in the row noise image, 0 represents the expected value of the row noise, and σ row This represents the standard deviation of row noise.

[0121] S205, add the second noise image, the readout noise image and the row noise image to obtain the noise image corresponding to the noise-free image.

[0122] After obtaining the second noise image, reading out the noise image and the row noise image, the electronic device can add the pixel values ​​of the pixels at the same position in the three noise images to generate a new noise image, which is then used as the noise image corresponding to the noise-free image.

[0123] The three most important types of noise in real-world images are shot noise, readout noise, and line noise. The solution provided in this application implements shot noise by adding noise-free images and camera gain, readout noise by adding readout noise parameters, and line noise by adding line noise parameters. This allows the final generated noisy image to closely match the noisy images in the real world.

[0124] As one embodiment of this application, when the electronic device performs step S103, in addition to being able to achieve it through steps S201-S204, it can also be achieved in the following ways:

[0125] In the first embodiment, the electronic device may only execute steps S201-S202, using the second noise image as the noise image. This achieves the goal of adding shot noise only to a noise-free image to obtain a noise image.

[0126] In the second embodiment, the electronic device can first execute step S203, and then add the readout noise image obtained from executing step S203 to the noise-free image to obtain a noise image. This achieves the goal of adding readout noise only to the noise-free image to obtain a noise image.

[0127] In the third embodiment, the electronic device can first execute step S204, and then add the row noise image obtained from executing step S204 to the noise-free image to obtain a noise image. This achieves the goal of adding row noise only to the noise-free image to obtain a noise image.

[0128] In the fourth embodiment, the electronic device can execute steps S201-S203, and then add the second noise image obtained from executing steps S201-S202 and the readout noise image obtained from executing step S203 to obtain a noise image. This achieves the addition of shot noise and readout noise to a noise-free image to obtain a noise image.

[0129] In the fifth embodiment, the electronic device can execute steps S201-S202 and S204, and then add the second noise image obtained from executing steps S201-S202 and the row noise image obtained from executing step S204 to obtain a noise image. This achieves the addition of shot noise and row noise to a noise-free image to obtain a noise image.

[0130] In the sixth embodiment, the electronic device can execute steps S203-S204, and then add the readout noise image obtained from step S203, the row noise image obtained from step S204, and the noise-free image to obtain a noise image. This achieves the addition of readout noise and row noise to a noise-free image to obtain a noise image.

[0131] In the solution provided in this application embodiment, the electronic device can flexibly select and combine different types of noise to add to a noise-free image according to the needs of different application scenarios, thereby meeting diverse noise image generation needs.

[0132] As one embodiment of this application, when the electronic device performs the above steps S102-S103, it can be implemented in the following way:

[0133] Input the noise-free image and the target image into the noise image generation model, and obtain the noise image output by the noise image generation model.

[0134] Electronic devices can have a pre-trained noise image generation model built in. In this way, when the electronic device predicts the first noise parameter of the camera and adds shot noise, readout noise and line noise to the noise-free image according to the first noise parameter, it can specifically input the noise-free image and the target image into the noise image generation model. The noise image generation model realizes the first noise parameter of the camera and adds shot noise, readout noise and line noise to the noise-free image according to the first noise parameter.

[0135] In one implementation, such as Figure 3 As shown, a noisy image generation model can include a feature extraction layer, a noise parameter prediction layer, and an image generation layer.

[0136] After an electronic device inputs a noise-free image and a target image into a noise image generation model, the noise image generation model can first process the target image, that is, extract the image features of the target image through a feature extraction layer, and then input the extracted image features into a noise parameter prediction layer. The noise parameter prediction layer can predict the first noise parameter of the camera based on the image features.

[0137] After obtaining the first noise parameter, the noise image generation model can input it into the image generation layer. Simultaneously, the noise-free image input into the noise image generation model is also input into the image generation layer. The image generation layer can then add shot noise, readout noise, and line noise to the noise-free image based on the first noise parameter to obtain the noise image. Finally, the noise image generation model can output the noise image.

[0138] In the solution provided in this application embodiment, a noisy image generation model can be pre-trained to learn how to extract image features, predict camera noise parameters based on the features, and add shot noise, readout noise, and line noise to the image according to the noise parameters. This noisy image generation model is then deployed in an electronic device. The electronic device can input a noise-free image and a target image into the noisy image generation model, which then performs the process of "predicting the first noise parameter of the camera and adding shot noise, readout noise, and line noise to the noise-free image according to the first noise parameter," thereby predicting more accurate noise parameters and generating a more accurate noisy image based on these more accurate noise parameters.

[0139] As one implementation method of this application, such as Figure 4 As shown, the training methods for the above-mentioned noisy image generation model can include:

[0140] S401, obtain the noise-free image of the sample and the noise parameters of the sample.

[0141] For noise-free sample images, images can be obtained from public datasets. As for sample noise parameters, they can be set manually by the user or randomly selected from a pre-defined parameter range by the model's training equipment; no specific limitations are imposed here.

[0142] S402, add shot noise, readout noise and row noise to the sample noise-free image according to the sample noise parameters to obtain the sample noise image.

[0143] After obtaining the sample noise parameters, shot noise, readout noise, and row noise can be added to the noise-free sample image based on these parameters to obtain a noisy sample image. The specific methods for adding shot noise, readout noise, and row noise based on the noise parameters have been described above. Figure 2 The embodiments shown are described in detail, and will not be repeated here.

[0144] S403, input the sample noise image into the feature extraction layer of the noise image generation model to obtain the sample features of the sample noise image extracted by the feature extraction layer.

[0145] S404, Input the sample features into the noise parameter prediction layer to obtain the second noise parameter predicted by the noise parameter prediction layer based on the sample features.

[0146] S405, input the sample noise-free image and the second noise parameter into the image generation layer, and obtain the comparison noise image obtained by the image generation layer adding shot noise, readout noise and row noise to the sample noise-free image according to the second noise parameter.

[0147] A noisy image generation model can include a feature extraction layer, a noise parameter prediction layer, and an image generation layer. Specifically, the noise image generation model can generate and compare noise images in the following ways:

[0148] After obtaining the sample noisy image, it can be input into the feature extraction layer of the noise image generation model to extract sample features. These features are then input into the noise parameter prediction layer, which predicts a second noise parameter based on the features. Finally, the second noise parameter is input into the image generation layer, along with the noise-free sample image input into the noise image generation model. The image generation layer adds shot noise, readout noise, and line noise to the noise-free sample image based on the second noise parameter to obtain a comparison noise image.

[0149] It should be noted that, in one implementation, the connections between the feature extraction layer, the noise parameter prediction layer, and the image generation layer, as well as the connections between the feature extraction layer and the image generation layer and the external environment, can be pre-established. In this way, the outputs of the feature extraction layer, the noise parameter prediction layer, and the image generation layer can be automatically output to the next layer or to the external environment according to the above connections. Furthermore, the noise-free sample image and the noisy sample image input to the noise image generation model can also be automatically input to the image generation layer and the feature extraction layer according to the above connections. Thus, the model training device only needs to interact with the overall model, without needing to concern itself with the complex relationships between the layers within the model, reducing the difficulty of model training.

[0150] In another implementation, the inputs and outputs of the feature extraction layer, noise parameter prediction layer, and image generation layer can all be connected to the outside world. This allows the model's training device to input data into each layer through its input interface and obtain its output data through its output interface. This improves the model's modularity. Each layer can be designed, debugged, and optimized independently without needing to consider the influence of other layers, facilitating expansion or modification according to specific needs.

[0151] S406, adjusts the parameters of the noise parameter prediction layer based on the difference between the sample noisy image and the comparison noisy image.

[0152] After obtaining the noisy image for comparison, the difference between the sample noisy image and the comparison noisy image can be calculated using metrics such as MSE (Mean Squared Error), SSIM (Structural Similarity Index), or PSNR (Peak Signal-to-Noise Ratio). Then, it is determined whether the difference is greater than a preset difference threshold.

[0153] If the difference is not greater than the threshold, the training of the noise image generation model is complete. If the difference is greater than the threshold, the training of the noise image generation model is incomplete. In this case, the parameters of the noise parameter prediction layer can be adjusted using gradient descent or backpropagation algorithms, and the training process can be repeated until the difference between the sample noise image and the comparison noise image is no greater than the preset difference threshold.

[0154] It should be noted that the training device for the noisy image generation model can be the aforementioned electronic device, or other devices different from the aforementioned electronic device, without specific limitations.

[0155] In the solution provided in this application embodiment, a noise parameter that can be accurately predicted can be obtained through the above training method, and a noise image generation model that generates a more accurate noise image can be generated based on the accurate noise parameter. Thus, when the electronic device generates a noise image based on the noise image generation model, the noise image can be made to be more consistent with the image actually captured by the camera.

[0156] As one embodiment of this application, the camera described above can be an ISP (Image Signal Processing) camera.

[0157] In the solution provided in this application, for each type of ISP camera, the electronic device can use images captured by that type of camera to predict the noise parameters of that type of camera, and accordingly add shot noise, readout noise, and line noise to the noise-free image to obtain a noisy image corresponding to that type of ISP camera. Finally, the noise-free image and the noisy image are used as sample images corresponding to that type of ISP camera. This method of predicting noise parameters and adding noise based on the predicted noise parameters can automatically generate sample images by fitting the imaging principle of the ISP camera, greatly improving the efficiency of acquiring sample images. Furthermore, for each type of ISP camera, only one image captured by that type of ISP camera is needed to generate a large number of noisy images corresponding to that type of ISP camera based on a large number of noise-free images included in the public dataset, greatly reducing the cost of acquiring sample images. In addition, because the noisy image is generated based on the predicted noise parameters of the ISP camera, the noisy image can well match the actual noise level of the ISP camera.

[0158] Corresponding to the above sample image generation method, this application embodiment also provides a denoising model training method, which will be described below.

[0159] A noise reduction model training method includes: acquiring sample images corresponding to a camera generated based on the sample image generation method described in any of the above embodiments, and training an image noise reduction model based on the sample images.

[0160] In one implementation, the image denoising model can be universally trained. Specifically, target images captured by multiple types of cameras can be acquired. Then, for each type of camera, a sample image corresponding to that type of camera is generated based on the target images captured by that type of camera using the sample image generation method described in any embodiment. This allows the acquisition of sample images corresponding to multiple types of cameras. The image denoising model can then be trained solely based on these sample images, enabling it to be used for denoising on various types of cameras. Thus, when the trained image denoising model is integrated into the aforementioned multiple types of cameras, each type of camera can achieve excellent image denoising results using the trained model.

[0161] In another implementation, the image denoising model can be trained specifically. Specifically, target images captured by a particular type of camera can be acquired, and then sample images corresponding to that type of camera can be generated using the sample image generation method described in any embodiment. The image denoising model can then be trained only on the sample images corresponding to that type of camera, allowing it to specialize in denoising for that specific camera type. Thus, when the trained image denoising model is integrated into that type of camera, the camera can achieve excellent image denoising results based on the trained model.

[0162] It should be noted that the training device for the image denoising model can be the aforementioned electronic device, or other devices, without specific limitations. The aforementioned image denoising model can be a CNN (Convolutional Neural Network) or GAN (Generative Adversarial Network) model, etc., without specific limitations.

[0163] The solution provided in this application can both adapt the image denoising model to the denoising needs of multiple types of cameras through universal training, thereby improving its versatility, and optimize the denoising effect of the model for specific cameras through targeted training, thereby achieving specialization and meeting the denoising needs in different scenarios.

[0164] As one implementation of this application, the image denoising model described above can be an RRDB (Residual-in-Residual Dense Block) model. The RRDB model is a super-resolution reconstruction network architecture further optimized based on SRGAN (Super-Resolution Generative Adversarial Networks). It improves the model's representational ability and training stability by introducing deeper residual networks and denser connections.

[0165] In the solution provided in this application embodiment, when using the RRDB model as the image denoising model, the RRDB model, through the structure of the residuals and dense connections, can extract deeper features from the image. In image denoising, these deeper features help distinguish noise from the real content of the image, thereby removing noise more effectively. Furthermore, the dense connections in the RRDB model allow it to fully utilize the feature information of each layer, which helps to preserve image details and edge information during the denoising process. Therefore, the RRDB model can better balance denoising effect and image detail preservation.

[0166] Corresponding to the above-described sample image generation method, this application also provides a sample image generation apparatus, which will be described below.

[0167] like Figure 5 As shown, a sample image generation device includes:

[0168] Image acquisition module 510 is used to acquire noise-free images and target images captured by the camera;

[0169] The noise parameter prediction module 520 is used to extract features from the target image and predict the first noise parameter of the camera based on the extracted features.

[0170] The noise image generation module 530 is used to add shot noise, readout noise and line noise to a noise-free image according to the first noise parameter to obtain a noise image;

[0171] The sample image determination module 540 is used to determine the noise-free image and the noisy image as the sample image corresponding to the camera.

[0172] In the solution provided in this application embodiment, an electronic device can acquire a noise-free image and a target image captured by a camera; extract features from the target image; predict a first noise parameter of the camera based on the extracted features; add shot noise, readout noise, and line noise to the noise-free image according to the first noise parameter to obtain a noisy image; and determine the noise-free image and the noisy image as sample images corresponding to the camera. For each type of camera, the electronic device can use images captured by that type of camera to predict the noise parameter of that type of camera, and add shot noise, readout noise, and line noise to the noise-free image accordingly to obtain a noisy image corresponding to that type of camera. Finally, the noise-free image and the noisy image are used as sample images corresponding to that type of camera. This method of predicting noise parameters and adding noise based on the predicted noise parameters can realize the automatic generation of sample images, greatly improving the efficiency of acquiring sample images. Furthermore, for each type of camera, only one image captured by that type of camera is needed to generate a large number of noisy images corresponding to that type of camera based on a large number of noise-free images included in the public dataset, greatly reducing the cost of acquiring sample images. In addition, because the noisy image is generated based on the predicted noise parameter of the camera, the noisy image can well match the actual noise level of the camera. Furthermore, the shot noise, readout noise, and line noise added to the noise-free image are the three most important types of noise in real-world images. This allows the noise image generated by adding these three types of noise to closely match the real noise image.

[0173] As one embodiment of this application, the first noise parameter may include: camera gain, readout noise parameter, and line noise parameter;

[0174] The aforementioned noisy image generation module 530 may include:

[0175] The first image generation unit is used to randomly sample the noise value corresponding to each pixel from the noise distribution corresponding to each pixel of the noise-free image to obtain the first noise image;

[0176] The second image generation unit is used to adjust the pixel values ​​of each pixel in the first noisy image according to the camera gain, so as to obtain a second noisy image after adding shot noise to the noiseless image.

[0177] The third image generation unit is used to determine the first Gaussian distribution based on the expectation and standard deviation included in the readout noise parameters, and generate a readout noise image whose readout noise distribution conforms to the first Gaussian distribution, wherein the readout noise image has the same size as the noise-free image.

[0178] The fourth image generation unit is used to determine the second Gaussian distribution based on the standard deviation included in the row noise parameters, and generate a row noise image in which the row noise distribution of each row conforms to the second Gaussian distribution, wherein the expectation of the second Gaussian distribution is zero, and the row noise image is the same size as the noise-free image.

[0179] The image overlay module is used to add the second noise image, the readout noise image, and the row noise image to obtain a noise image corresponding to the noise-free image.

[0180] As one implementation method of this application, the noise distribution corresponding to each pixel can be generated in the following manner:

[0181] Calculate the ratio between the pixel value of a pixel and the camera gain, and use the ratio as the parameter of the Poisson distribution to construct the Poisson distribution as the noise distribution corresponding to the pixel.

[0182] The aforementioned second image generation unit can be specifically used to multiply the pixel value of each pixel in the first noisy image by the camera gain to obtain a second noisy image after adding shot noise to the noise-free image.

[0183] As one embodiment of this application, the noise parameter prediction module 520 described above may include:

[0184] The noise distribution determination unit is used to determine the noise distribution corresponding to the noise in the target image based on the extracted features;

[0185] The noise parameter determination unit is used to determine the readout noise parameter and line noise parameter of the camera based on the noise distribution. The noise distribution is the result of the superposition of the first noise distribution corresponding to the readout noise parameter and the second noise distribution corresponding to the line noise parameter. The distribution of all noise in the first noise distribution conforms to the Gaussian distribution constructed with the readout noise parameter, and the distribution of each line noise in the second noise distribution conforms to the Gaussian distribution constructed with the line noise parameter.

[0186] The camera gain determination unit is used to determine the camera gain based on the pixel values ​​of the pixels reflected by the extracted features.

[0187] As one embodiment of this application, the camera gain determination unit is specifically used to determine the grayscale histogram corresponding to the target image based on the pixel values ​​of the pixels reflected by the extracted features; and to determine the camera gain of the camera based on the distribution of grayscale values ​​in the grayscale histogram, wherein the correspondence between the distribution of grayscale values ​​and the camera gain is that the more the distribution of grayscale values ​​is biased towards lower grayscale values, the lower the camera gain, and the more the distribution of grayscale values ​​is biased towards higher grayscale values, the higher the camera gain.

[0188] As one embodiment of this application, the noise parameter prediction module 520 described above can be specifically used to input the target image into the noise image generation model, extract the features of the target image based on the noise image generation model, and predict the first noise parameter of the camera based on the extracted features.

[0189] The aforementioned noise image generation module 530 can be specifically used to input a noise-free image into a noise image generation model, and rely on the noise image generation model to add shot noise, readout noise and line noise to the noise-free image according to the first noise parameter to obtain a noise image.

[0190] As one embodiment of this application, the training method of the above-mentioned noisy image generation model may include:

[0191] Obtain noise-free images and noise parameters of the samples;

[0192] Based on the sample noise parameters, shot noise, readout noise, and row noise are added to the sample noise-free image to obtain a sample noise image;

[0193] The sample noise image is input into the feature extraction layer of the noise image generation model to obtain the sample features of the sample noise image extracted by the feature extraction layer. The noise image generation model also includes a noise parameter prediction layer and an image generation layer.

[0194] Input the sample features into the noise parameter prediction layer to obtain the second noise parameter predicted by the noise parameter prediction layer based on the sample features;

[0195] The noise-free image of the sample and the second noise parameter are input into the image generation layer to obtain the comparison noise image obtained by the image generation layer adding shot noise, readout noise and row noise to the noise-free image of the sample according to the second noise parameter.

[0196] The parameters of the noise parameter prediction layer are adjusted based on the difference between the sample noisy image and the comparison noisy image.

[0197] Corresponding to the above-mentioned noise reduction model training method, this application embodiment also provides a noise reduction model training device, which will be described below.

[0198] like Figure 6 As shown, a noise reduction model training device includes:

[0199] The sample image acquisition module 610 is used to acquire a sample image corresponding to the camera generated based on the sample image generation method described in any of the above embodiments;

[0200] The model training module 620 is used to train an image denoising model based on sample images.

[0201] The solution provided in this application can both adapt the image denoising model to the denoising needs of multiple types of cameras through universal training, thereby improving its versatility, and optimize the denoising effect of the model for specific cameras through targeted training, thereby achieving specialization and meeting the denoising needs in different scenarios.

[0202] As one embodiment of this application, the above image denoising model is the RRDB model.

[0203] Corresponding to the above-described sample image generation method, this application also provides an electronic device, such as... Figure 7 As shown, it includes:

[0204] Memory 701 is used to store computer programs;

[0205] The processor 702, when executing the program stored in the memory 701, implements the sample image generation method described in any of the above embodiments.

[0206] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 702, the communication interface, and the memory 701 communicating with each other via the communication bus.

[0207] In the solution provided in this application embodiment, an electronic device can acquire a noise-free image and a target image captured by a camera; extract features from the target image; predict a first noise parameter of the camera based on the extracted features; add shot noise, readout noise, and line noise to the noise-free image according to the first noise parameter to obtain a noisy image; and determine the noise-free image and the noisy image as sample images corresponding to the camera. For each type of camera, the electronic device can use images captured by that type of camera to predict the noise parameter of that type of camera, and add shot noise, readout noise, and line noise to the noise-free image accordingly to obtain a noisy image corresponding to that type of camera. Finally, the noise-free image and the noisy image are used as sample images corresponding to that type of camera. This method of predicting noise parameters and adding noise based on the predicted noise parameters can realize the automatic generation of sample images, greatly improving the efficiency of acquiring sample images. Furthermore, for each type of camera, only one image captured by that type of camera is needed to generate a large number of noisy images corresponding to that type of camera based on a large number of noise-free images included in the public dataset, greatly reducing the cost of acquiring sample images. In addition, because the noisy image is generated based on the predicted noise parameter of the camera, the noisy image can well match the actual noise level of the camera. Furthermore, the shot noise, readout noise, and line noise added to the noise-free image are the three most important types of noise in real-world images. This allows the noise image generated by adding these three types of noise to closely match the real noise image.

[0208] Corresponding to the above-described noise reduction model training method, this application also provides an electronic device, such as... Figure 8 As shown, it includes:

[0209] Memory 801 is used to store computer programs;

[0210] When the processor 802 executes the program stored in the memory 801, it implements the noise reduction model training method described in any of the above embodiments.

[0211] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 802, the communication interface, and the memory 801 communicating with each other via the communication bus.

[0212] The solution provided in this application can both adapt the image denoising model to the denoising needs of multiple types of cameras through universal training, thereby improving its versatility, and optimize the denoising effect of the model for specific cameras through targeted training, thereby achieving specialization and meeting the denoising needs in different scenarios.

[0213] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0214] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0215] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0216] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0217] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program. When the computer program is executed by a processor, it implements the sample image generation method or the noise reduction model training method described in any of the above embodiments.

[0218] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the sample image generation method described in any of the above embodiments or to implement the noise reduction model training method described in any of the above embodiments.

[0219] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0220] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0221] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0222] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for generating sample images, characterized in that, The method includes: Acquire noise-free images and target images captured by the camera; Extract features from the target image, and predict the first noise parameter of the camera based on the extracted features; Based on the first noise parameter, shot noise, readout noise, and line noise are added to the noise-free image to obtain a noisy image; The noise-free image and the noisy image are identified as sample images corresponding to the camera.

2. The method according to claim 1, characterized in that, The first noise parameter includes: camera gain, readout noise parameter, and line noise parameter; The step of adding shot noise, readout noise, and line noise to the noise-free image according to the first noise parameter to obtain a noisy image includes: From the noise distribution corresponding to each pixel of the noise-free image, the noise value corresponding to each pixel is randomly sampled to obtain the first noise image; The pixel values ​​of each pixel in the first noisy image are adjusted according to the camera gain to obtain a second noisy image after adding shot noise to the noise-free image. Based on the expected value and standard deviation included in the readout noise parameters, a first Gaussian distribution is determined, and a readout noise image whose distribution conforms to the first Gaussian distribution is generated, wherein the readout noise image has the same size as the noise-free image; Based on the standard deviation included in the row noise parameters, a second Gaussian distribution is determined, and a row noise image is generated in which the distribution of row noise in each row conforms to the second Gaussian distribution, wherein the expectation of the second Gaussian distribution is zero, and the row noise image has the same size as the noise-free image. The second noise image, the readout noise image, and the row noise image are added together to obtain a noise image corresponding to the noise-free image.

3. The method according to claim 2, characterized in that, The noise distribution for each pixel is generated as follows: Calculate the ratio between the pixel value of a pixel and the camera gain, and use the ratio as the parameter of a Poisson distribution to construct a Poisson distribution as the noise distribution corresponding to the pixel. The step of adjusting the pixel values ​​of each pixel in the first noisy image according to the camera gain to obtain a second noisy image after adding shot noise to the noise-free image includes: The pixel value of each pixel in the first noisy image is multiplied by the camera gain to obtain a second noisy image after adding shot noise to the noise-free image.

4. The method according to claim 1, characterized in that, The prediction of the first noise parameter of the camera based on the extracted features includes: Based on the extracted features, the noise distribution corresponding to the noise in the target image is determined; The readout noise parameters and line noise parameters of the camera are determined based on the noise distribution. The noise distribution is the result of the superposition of the first noise distribution corresponding to the readout noise parameters and the second noise distribution corresponding to the line noise parameters. All noise in the first noise distribution conforms to a Gaussian distribution constructed with the readout noise parameters, and the distribution of each line noise in the second noise distribution conforms to a Gaussian distribution constructed with the line noise parameters. The camera gain of the camera is determined based on the pixel values ​​of the pixels reflected by the extracted features.

5. The method according to claim 4, characterized in that, Determining the camera gain based on the pixel values ​​of the extracted features includes: Based on the pixel values ​​of the extracted features, the grayscale histogram corresponding to the target image is determined. The camera gain is determined based on the distribution of gray values ​​in the gray-scale histogram. The relationship between the distribution of gray values ​​and the camera gain is that the more the gray values ​​are distributed towards lower gray values, the lower the camera gain; and the more the gray values ​​are distributed towards higher gray values, the higher the camera gain.

6. The method according to any one of claims 1-5, characterized in that, The feature of the target image is extracted, and the first noise parameter of the camera is predicted based on the extracted feature; Based on the first noise parameter, shot noise, readout noise, and line noise are added to the noise-free image to obtain a noisy image, including: The noise-free image and the target image are input into the noise image generation model to obtain the noise image output by the noise image generation model. The noise image is an image obtained by the noise image generation model by adding shot noise, readout noise and line noise to the noise-free image according to a first noise parameter. The first noise parameter is the noise parameter of the camera predicted by the noise image generation model based on the features of the extracted target image.

7. The method according to claim 6, characterized in that, The training methods for the noisy image generation model include: Obtain noise-free images of samples and sample noise parameters; Based on the sample noise parameters, shot noise, readout noise, and row noise are added to the sample noise-free image to obtain a sample noise image; The sample noise image is input into the feature extraction layer of the noise image generation model to obtain the sample features of the sample noise image extracted by the feature extraction layer. The noise image generation model further includes a noise parameter prediction layer and an image generation layer. The sample features are input into the noise parameter prediction layer to obtain the second noise parameter predicted by the noise parameter prediction layer based on the sample features. The noise-free image of the sample and the second noise parameter are input into the image generation layer to obtain a comparison noise image obtained by the image generation layer adding shot noise, readout noise and row noise to the noise-free image of the sample according to the second noise parameter. The parameters of the noise parameter prediction layer are adjusted based on the difference between the sample noise image and the comparison noise image.

8. A method for training a noise reduction model, characterized in that, The method includes: Obtain sample images of the camera generated based on the method described in any one of claims 1-7; The image denoising model is trained based on the sample images.

9. The method according to claim 8, characterized in that, The image denoising model is the RRDB model.

10. A sample image generation apparatus, characterized in that, The device includes: The image acquisition module is used to acquire noise-free images and target images captured by the camera; A noise parameter prediction module is used to extract features from the target image and predict the first noise parameter of the camera based on the extracted features. A noise image generation module is used to add shot noise, readout noise, and line noise to the noise-free image according to the first noise parameter to obtain a noise image; The sample image determination module is used to determine the noise-free image and the noisy image as sample images corresponding to the camera.

11. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-7 or the method of any one of claims 8-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1-7 or the method of any one of claims 8-9.

Images