Image processing method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention discloses an image processing method and device, a storage medium and electronic equipment, the image processing method is executed by the electronic equipment, and the image processing method comprises the following steps: acquiring a first image and an acquisition parameter of the first image; the collection parameters are input into a noise adding model, noise output by the noise adding model is obtained, and the noise adding capacity of the noise adding model is obtained through training based on a noiseless training image and a noisy training image in the training process; and adding the noise into the first image to obtain a second image.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to an image processing method, device, storage medium and electronic device. Background Art

[0002] It is a common image processing method to denoise a noisy image using a denoising model. During the training of the denoising model, random noise is usually added to the collected low-noise image to obtain a noisy training image for training the denoising model.

[0003] Then, since the noise in the training image is random noise, not the real noise of the image, after the denoising model is trained based on such training images, the denoising effect of the obtained denoising model is not ideal. Summary of the invention

[0004] The present application provides an image processing method, device, storage medium and electronic device.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, which is performed by an electronic device, and the method includes:

[0006] Acquire a first image and acquisition parameters of the first image;

[0007] Inputting the acquisition parameters into the noise adding model to obtain the noise output by the noise adding model, wherein the noise adding capability of the noise adding model is obtained by training based on noise-free training images and noise-containing training images during a training process;

[0008] The noise is added to the first image to obtain the second image.

[0009] Optionally, the noise addition model includes multiple noise modules; different noise modules are used to generate different types of noise.

[0010] Optionally, the noise adding model includes a first noise module; and inputting the acquisition parameter into the noise adding model to obtain the noise output by the noise adding model includes:

[0011] The acquisition parameters are input into the first noise module to obtain a first type of noise output by the first noise module based on random processing.

[0012] Optionally, the first type of noise includes at least one of the following:

[0013] M*N first noises, wherein the m*nth first noise is used to be superimposed with the m*nth pixel in the first image to obtain a pixel value of the m*nth pixel in the second image;

[0014] M second noises, wherein the mth second noise is used to be superimposed on the mth row of pixels in the first image to obtain pixel values ​​of the mth row of pixels in the second image;

[0015] Among them, M is the total number of pixel rows of the first image; N is the total number of pixel columns of the first image; m is a positive integer less than or equal to M; and n is a positive integer less than or equal to N.

[0016] Optionally, the first noise module includes: a random unit and a first noise unit;

[0017] The step of inputting the acquisition parameters into the first noise module to obtain a first type of noise output by the first noise module based on random processing includes:

[0018] Obtaining M*N first reference noises by utilizing random processing of the random unit;

[0019] The M*N first reference noises and the acquisition parameters are input into a first noise unit to obtain the M*N first noises output by the first noise unit.

[0020] Optionally, the step of inputting the M*N first reference noises and the acquisition parameters into a first noise unit, so that the first noise unit outputs the M*N first noises, comprises:

[0021] The first noise unit determines a first standard deviation based on the acquisition parameters and the pixel values ​​of the first image;

[0022] The M*N first reference noises are adjusted based on the first standard deviation to obtain the M*N first noises.

[0023] Optionally, the first noise module further includes: a second noise unit; the step of inputting the acquisition parameters into the first noise module to obtain a first type of noise output by the first noise module based on random processing includes:

[0024] Obtaining M second reference noises by random processing of the random unit;

[0025] The M second reference noises and the acquisition parameters are input into the second noise unit to obtain the M second noises output by the second noise unit.

[0026] Optionally, the noise adding model further includes: a dark current noise module, and the step of inputting the acquisition parameters into the noise adding model to obtain the noise output by the noise adding model includes:

[0027] The acquisition parameters are input into the dark current noise module to obtain dark current noise output by the dark current noise module, wherein the dark current noise is obtained after the dark current noise module adjusts M*N preset noises based on the acquisition parameters.

[0028] Optionally, the noise adding model further includes: a black level noise module, and the inputting of the acquisition parameters into the noise adding model to obtain the noise output by the noise adding model further includes:

[0029] The acquisition parameters are input into the black level noise module to obtain black level error noise corresponding to the acquisition parameters, which is determined by the black level noise module from a plurality of preset black level error noises based on the acquisition parameters.

[0030] Optionally, the noise adding model includes a first noise module, a dark current noise module and a black level noise module, and adding the noise to the first image to obtain the second image includes:

[0031] Add at least one of the first type of noise output by the first noise module, the dark current noise output by the dark current noise module, and the black level error noise output by the black level noise module to the first image to obtain the second image

[0032] Optionally, the noisy training image is an image captured by an image capture device in a dark field environment.

[0033] Optionally, a target model is trained based on the second image to obtain a denoising model; the denoising model has image denoising capability;

[0034] Based on the denoising model, denoising is performed on the third image with noise.

[0035] In a second aspect, an embodiment of the present application provides an image processing device, the device comprising:

[0036] An acquisition module, used to acquire a first image and acquisition parameters of the first image;

[0037] An input module, used for inputting the acquisition parameters into the denoising model to obtain the noise output by the denoising model, wherein the denoising capability of the denoising model is obtained during training based on noise-free training images and noise-containing training images;

[0038] An adding module is used to add the noise to the first image to obtain the second image.

[0039] In a third aspect, an embodiment of the present application provides a non-temporary computer-readable storage medium, which, when instructions in the storage medium are executed by a processor of a computer, enables the computer to execute the image processing method as described above.

[0040] In a fourth aspect, an embodiment of the present application provides an electronic device, including:

[0041] a memory for storing processor-executable instructions;

[0042] A processor connected to the memory;

[0043] Wherein, the processor is used to execute any of the image processing methods described above.

[0044] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0045] In the embodiment of the present application, the noise added to the first image is generated by the noise adding model based on the acquisition parameters of the first image, wherein the acquisition parameters of the first image can reflect the parameter information of each device in the image acquisition device when the first image is acquired. Since the parameter information of the image acquisition device is one of the main factors affecting the noise size and type in the acquired image, therefore, adding the noise generated based on the acquisition parameters of the first image to the first image can make the noise added to the image closer to the real noise of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0047] Figure 1 is a flowchart of an image processing method according to an exemplary embodiment;

[0048] Figure 2 is a schematic diagram of a process of performing noise addition processing on a first image by using a noise addition model according to an exemplary embodiment;

[0049] Figure 3 is a schematic diagram of a first image according to an exemplary embodiment;

[0050] Figure 4 is a schematic diagram of a second image according to an exemplary embodiment;

[0051] Figure 5 is a schematic structural diagram of an image processing device according to an exemplary embodiment;

[0052] Figure 6It is a schematic structural diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0053] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices consistent with some aspects of the present application as detailed in the appended claims.

[0054] Before explaining the embodiments of the present application in detail, the application scenarios of the embodiments of the present application are first described, as follows:

[0055] During the image acquisition process, the image may contain noise due to the performance of the image acquisition device, the acquisition parameters of the image acquisition device and the external environment. When the noise in the acquired image is large, it will affect the display effect of the image.

[0056] Based on this, in the related art, after obtaining an image with noise, the image with noise can be input into a trained denoising model, and the denoising model is used to remove the noise in the image to improve the display effect of the image.

[0057] In the process of training the denoising model, random noise is usually added to the collected low-noise image to obtain a noisy training image for training the denoising model.

[0058] Since the noise in the training image is random noise, not the real noise of the image, after the denoising model is trained based on such training images, the denoising effect of the obtained denoising model is not ideal.

[0059] In addition, deep learning-based image modeling methods mainly include convolutional neural networks, recurrent neural networks, generative adversarial networks, etc. These networks mainly learn image feature representation and pattern recognition capabilities through training on large-scale data, thereby achieving image classification, segmentation, reconstruction, denoising and other tasks. However, deep learning-based modeling methods require a large number of training images to train the model, and have high requirements on the quality and annotation of training images, which are prone to overfitting and insufficient generalization ability. At the same time, they also lack a theoretical basis for interpretability.

[0060] To solve the above problems, in an embodiment of the present application, an image processing method is provided, which is executed by an electronic device, such as Figure 1 The image processing method comprises:

[0061] S101: Acquire a first image and acquisition parameters of the first image;

[0062] S102: Inputting the acquisition parameters into a noise adding model to obtain the noise output by the noise adding model, wherein the noise adding capability of the noise adding model is obtained by training based on noise-free training images and noise-containing training images during a training process;

[0063] S103: Add the noise to the first image to obtain a second image.

[0064] The electronic device mentioned above may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart camera, a server or other electronic device with computing capabilities.

[0065] The first image mentioned above may be a low-noise image or a clean image without noise acquired by an image acquisition device.

[0066] Exemplarily, the first image mentioned above may be an image captured by an image capture device under low sensitivity conditions by increasing the exposure time.

[0067] Among them, the image acquisition device in the embodiment of the present application can be an image acquisition component on an electronic device, or it can be an image acquisition device that can establish a communication connection with the electronic device.

[0068] Exemplarily, the above-mentioned acquisition parameters are photosensitivity parameters, resolution, color space, exposure time, signal strength, etc. of the image acquisition device when acquiring the first image.

[0069] The photosensitivity parameter of the image acquisition device may be an ISO value of the image acquisition device when acquiring an image.

[0070] In one embodiment, the above-mentioned noise addition model is a model constructed based on common noise in the image.

[0071] Exemplarily, the above-mentioned noise addition model is a neural network model constructed based on one or more of shot noise, random thermal noise, black level error noise, dark current noise, row noise, read noise and quantization noise generated by the image acquisition device when acquiring images.

[0072] Among them, shot noise is the noise generated by the randomness of the absorption of photons by the image sensor in the electronic device. Shot noise serves Poisson distribution and can also be called Poisson noise.

[0073] In the embodiment of the present application, in order to facilitate the construction of the noise adding model, the shot noise can be approximated as Gaussian noise.

[0074] For example, in the embodiment of the present application, the shot noise with Gaussian distribution can be expressed by the following formula (1).

[0075] N shot ~(0,β shot ·I)……………(1)

[0076] Among them, N shot is the shot noise, β shot It is positively correlated with the ISO value of the image acquisition device, and I is the pixel value of the first image.

[0077] The random thermal noise is the noise generated by the heat of the image sensor when the image acquisition device acquires images, which can be expressed by the following formula (2):

[0078]

[0079] Among them, N DCSN is random thermal noise, is the variance of random thermal noise.

[0080] The black level error noise mentioned above is noise caused by the performance difference between each pixel unit of the image sensor and the difference in the amount of light incident on each pixel unit of the image sensor and the incident angle.

[0081] The black level error noise generated by each pixel unit of all images acquired by the image acquisition device under the same acquisition parameters is a fixed value, and the black level error noise generated by different pixel units is different. Under different acquisition parameters, the black level error noise generated by the same pixel unit is different.

[0082] That is, when the acquisition parameter of the image acquisition device is the first acquisition parameter, all images acquired by the image acquisition device have the same black level error noise, and when the acquisition parameter of the image acquisition device changes, the black level error noise on each pixel point of the image acquired by the image acquisition device will also change.

[0083] The silicon lattice inside the image sensor also generates electrons due to heat generation. This is dark current. Since these electrons are also collected by pixels, the noise that appears on the image is called dark current noise.

[0084] Based on the above analysis, the dark current noise of the image acquisition device when acquiring images can be expressed by the following formula (3).

[0085] N DC = k·N FP ………………(3)

[0086] Among them, N DCis the dark current noise, k is the first coefficient, where k is positively correlated with the ISO value of the image acquisition device, and N FP is the dark current fixed pattern noise.

[0087] The above-mentioned read noise refers to the noise generated when the image sensor converts an analog signal into a digital signal, and can be expressed by the following formula (4).

[0088]

[0089] Among them, N read is the read noise, N read ∈R M*N , is the variance of the read noise.

[0090] Wherein, M is the number of pixel rows of the first image, and N is the number of pixel columns of the first image.

[0091] The above-mentioned row noise is because when reading signals from the pixel array included in the image sensor, they are read row by row. When the readout circuit reads the same row of pixels, it will carry noise of the same magnitude. However, the readout time between rows is different, and the noise magnitude generated is also different. Therefore, the noise generated by the readout circuit when reading each row of pixels is called row noise.

[0092] In the embodiment of the present application, the row noise when the image acquisition device acquires an image can be represented by the following formula (5).

[0093]

[0094] Among them, N row is the line noise, N row ∈R M , is the variance of the row noise.

[0095] After obtaining shot noise, dark current noise, black level error noise, random thermal noise, read noise and row noise, considering that shot noise, dark current noise, black level error noise, read noise and random thermal noise are all noises in pixels. Therefore, in the embodiment of the present application, heteroscedastic Gaussian noise is constructed based on shot noise, random thermal noise and read noise. The constructed heteroscedastic Gaussian noise can be expressed by the following formula (6).

[0096] N HG ~N(0,β1·I+β2)………………(6)

[0097] Among them, N HG is heteroscedastic Gaussian noise, β1 is the first parameter, β1 is a parameter related to ISO value, and β1 can be the above βshot , I is the pixel value of the first image, β2 is the second parameter, and β2 is a parameter related to random thermal noise and read noise.

[0098] After constructing a noise model based on shot noise, dark current noise, black level error noise, random thermal noise, row noise and read noise, the noise model can be trained using training images until the noise model converges or reaches a preset number of training times.

[0099] Exemplarily, after constructing the noise adding model, a training image including shot noise, dark current noise, black level error noise, random thermal noise, row noise and / or read noise can be input into the noise adding model so that the noise adding model learns the distribution characteristics of various noises in the training image in the process of removing the various noises included in the training image.

[0100] The implementation process of removing dark current noise in the training image by the denoising model can be expressed by the following formula (7).

[0101] f FP (x3) = x3 - g FP (ISO)⊙N f (7)

[0102] Among them, f FP (x3) is the training image after removing the dark current noise, x3 is the training image with dark current noise, g FP (ISO) The third parameter, N f is the preset noise, and ⊙ is the dot multiplication operation.

[0103] The implementation process of removing black level error noise in the training image by using the denoising model can be expressed by formula (8).

[0104] f BLE (x4) = x4 - N BLE (ISO) (8)

[0105] f BLE (x4) is the training image after removing the black level error noise, x4 is the training image with black level error noise, D BLE (ISO) is the black level error noise.

[0106] During the training process of the noise adding model, the loss of the model can be calculated by formulas (9) and (10).

[0107]

[0108]

[0109] Among them, p zis the noise learned by the denoising model during training based on the denoising of the training images, i is the training image, Θ1 and Θ2 are the model parameters of the noise model, Jf(x) represents the Jacobian matrix of function f at x, and det represents the determinant of the input matrix.

[0110] In the embodiment of the present application, the noise model is a neural network model constructed based on real noise in the image, such as shot noise, dark current noise, random thermal noise, black level error noise, line noise and read noise, etc. Compared with the model constructed by the probability method, the difficulty of model construction is reduced and the automatic learning ability and generalization ability of the model are improved. Compared with the traditional neural network model, since each module of the noise model in the embodiment of the present application is constructed based on the real noise of the image, the training difficulty of the model can be reduced and the training effect of the model can be improved.

[0111] After obtaining the trained denoising model, the acquired first image and the acquisition parameters of the image acquisition device when acquiring the first image are input into the denoising model so that the denoising model can generate noise, and the noise generated by the denoising model is added to the first image to obtain a second image with noise.

[0112] Exemplarily, when the acquisition parameter of the first image is a first ISO value, after the first image is acquired by an image acquisition device, the first image and the first ISO value are input into a noise addition model. Based on this, the noise addition model generates a noise for each pixel based on the input first ISO value and the pixel value of each pixel in the first image, and adds the generated noise to the corresponding pixel point of the first image to obtain a second image with noise.

[0113] For example, after the first image and the first ISO value are input into the noise adding model, the noise adding model generates a first noise image of the same size as the first image based on the first ISO value, and then adds the pixel value of the first pixel in the first noise image to the pixel value of the second pixel corresponding to the first pixel in the first image to obtain a second image with noise, wherein the first pixel is any pixel in the first noise image.

[0114] In the embodiment of the present application, the noise added to the first image is generated by the noise adding model based on the acquisition parameters of the first image, wherein the acquisition parameters of the first image can reflect the parameter information of each device in the image acquisition device when the first image is acquired. Since the parameter information of the image acquisition device is one of the main factors affecting the noise size and type in the acquired image, therefore, adding the noise generated based on the acquisition parameters of the first image to the first image can make the noise added to the image closer to the real noise of the image.

[0115] Based on this, when the denoising model is trained using the second image with close to real noise as a training set, compared with training the denoising model by collecting a training set with random noise, the denoising model's ability to recognize and remove real noise can be improved, thereby improving the denoising effect of the trained denoising model on noisy images.

[0116] In addition, obtaining a second image with noise based on the denoising model and using the second image for training the denoising model can also reduce the difficulty of obtaining training images for training the denoising model and improve the training speed of the denoising model.

[0117] In one embodiment, the noise addition model includes a plurality of noise modules; different noise modules are used to generate different types of noise.

[0118] Exemplarily, the noise addition model of the embodiment of the present application may include one or more of a shot noise module, a black level noise module, a dark current noise module, a row noise module, a read noise module, and a random thermal noise module.

[0119] The shot noise module is used to generate shot noise based on the pixel values ​​of the first image and the first ISO value in the input noise model.

[0120] From the above introduction, it can be seen that the size of the black level error noise generated by the same pixel unit is related to the ISO value when the image acquisition device acquires the image. Therefore, in the electronic device of the embodiment of the present application, multiple preset black level error noises are stored for each pixel unit, wherein each preset black level error noise corresponds to an ISO value.

[0121] Based on this, the black level noise module is used to determine the black level error noise corresponding to the first ISO value for each pixel from a plurality of preset black level error noises based on the first ISO value of the input noise model.

[0122] The dark current noise module is used to generate dark current noise based on the first ISO value in the input denoising model.

[0123] The random thermal noise module is used to generate random thermal noise having the same number of pixels as that of the first image based on the size of the first image.

[0124] The row noise module is used to generate noise having the same number as the number of rows of the first image based on the size of the first image.

[0125] The read noise is used to generate a read noise having the same number as the number of pixels of the first image based on the size of the first image.

[0126] In one embodiment, the noise adding model includes a first noise module; the step of inputting the acquisition parameters into the noise adding model to obtain the noise output by the noise adding model includes:

[0127] The acquisition parameters are input into the first noise module to obtain a first type of noise output by the first noise module based on random processing.

[0128] During the training process of the noise adding model, the noise adding model can learn the distribution characteristics of heteroscedastic Gaussian noise and row noise based on the denoising processing of heteroscedastic Gaussian noise and row noise in the training image. Based on this, after the first noise module of the noise adding model obtains the acquisition parameters, the noise adding model will randomly generate the first type of noise based on the distribution characteristics of heteroscedastic Gaussian noise and row noise learned during the training process.

[0129] Exemplarily, the first type of noise includes at least one of the following:

[0130] M*N first noises, wherein the m*nth first noise is used to be superimposed with the m*nth pixel in the first image to obtain a pixel value of the m*nth pixel in the second image;

[0131] M second noises, wherein the mth second noise is used to be superimposed on the mth row of pixels in the first image to obtain pixel values ​​of the mth row of pixels in the second image;

[0132] Among them, M is the total number of pixel rows of the first image; N is the total number of pixel columns of the first image; m is a positive integer less than or equal to M; and n is a positive integer less than or equal to N.

[0133] For example, when the size of the first image captured by the image acquisition device is 100*200, the first noise module will generate a corresponding first noise for each pixel in the first image based on random processing, that is, generate 100*200 first noises.

[0134] The first noise generated by the first noise module of the noise adding model may be Gaussian heteroscedastic noise.

[0135] In addition, the first noise module will also generate a second noise for each row of the first image based on random processing. When the size of the first image is 100*200, the first noise will generate 200 second noises, where each second noise is the noise corresponding to each pixel row in the first image.

[0136] That is, the same second noise is loaded to each pixel row in the first image.

[0137] The second noise may be the above-mentioned row noise.

[0138] In one embodiment, the first noise module includes: a random unit and a first noise unit;

[0139] The step of inputting the acquisition parameters into the first noise module to obtain a first type of noise output by the first noise module based on random processing includes:

[0140] Obtaining M*N first reference noises by utilizing random processing of the random unit;

[0141] The M*N first reference noises and the acquisition parameters are input into a first noise unit to obtain the M*N first noises output by the first noise unit.

[0142] In an embodiment of the present application, during the training process of the denoising model, a training image with various noises is input into the denoising model, and the denoising model is used to denoise the training image, so that the denoising model can learn the distribution characteristics of various noises in the training image during the training process.

[0143] Based on this, when the first image and acquisition parameters are input into the denoising model, the random unit in the denoising model can generate M*N first reference noises based on random processing based on the distribution characteristics of heteroscedastic Gaussian noise learned from the training image.

[0144] In addition, considering that there will be differences in the noise in the images captured by the image acquisition device under different acquisition parameters, after obtaining the M*N first baseline noises output by the random unit, the M*N first baseline noises and the acquisition parameters when capturing the first image can also be input into the first noise unit, so that the first noise unit can adjust the M*N first baseline noises based on the acquisition parameters to obtain M*N first noises.

[0145] In one embodiment, the step of inputting the M*N first reference noises and the acquisition parameters into a first noise unit to obtain the first noise unit outputting the M*N first noises includes:

[0146] The first noise unit determines a first standard deviation based on the acquisition parameters and the pixel values ​​of the first image;

[0147] The M*N first reference noises are adjusted based on the first standard deviation to obtain the M*N first noises.

[0148] Exemplarily, after M*N first reference noises and acquisition parameters when acquiring the first image are input into the first noise unit, the first noise unit can generate a first standard deviation based on the acquisition parameters and the pixel values ​​of the first image, and adjust the M*N first reference noises based on the first standard deviation to obtain M*N first noises output by the first noise unit.

[0149] Exemplarily, when the acquisition parameter is the first ISO value, the first noise unit generates a first standard deviation based on the acquisition parameter and the pixel value of each pixel in the first image through the following formulas (11) to (13).

[0150]

[0151] Wherein, β1 is a first parameter, β2 is a parameter related to the shot noise of the image, β2 is a second parameter, β3 is a parameter related to the thermal random noise and read noise of the image, and I is a pixel value of the first image.

[0152] The above-mentioned β1 and β2 may be two multi-layer perceptrons included in the first noise unit, and are obtained by fitting the first ISO to based on the following formulas (12) and (13).

[0153] β1=gβ1(ISO) (12)

[0154] β2=gβ2(ISO) (13)

[0155] Exemplarily, the first noise unit may also adjust the M*N first reference noises by a first standard deviation based on the following formula (14) to obtain the M*N first noises output by the first noise unit.

[0156]

[0157] in, is the first noise, z is the first reference noise, σ HG is the first standard deviation, and ⊙ is the dot multiplication operation.

[0158] In one embodiment, the first noise module further includes: a second noise unit; the step of inputting the acquisition parameters into the first noise module to obtain the first type of noise output by the first noise module based on random processing includes:

[0159] Obtaining M second reference noises by random processing of the random unit;

[0160] The M second reference noises and the acquisition parameters are input into the second noise unit to obtain the M second noises output by the second noise unit.

[0161] From the above introduction, it can be seen that in the process of training the denoising model through training images, the denoising model can learn the true distribution of row noise from the training images. Therefore, after the first image is input into the denoising model, the random unit in the denoising model can also generate a second noise corresponding to each pixel row in the first image based on the number of pixel rows in the first image, that is, generate M second reference noises.

[0162] In order to improve the authenticity of the row noise generated by the noise addition model, after obtaining the M second reference noises obtained by the random unit based on random processing, the M second reference noises and acquisition parameters can also be input into the second noise unit, so that the second noise unit can adjust the second reference noise based on the acquisition parameters to obtain the second noise output by the second noise unit.

[0163] Exemplarily, when the acquisition parameters of the first image are input into the second noise unit, the second noise unit may generate a second standard deviation according to the acquisition parameters of the first image, and adjust the M second reference noises based on the second standard deviation to obtain M second noises.

[0164] For example, when the acquisition parameter of the first image is the first ISO value, the second noise unit calculates the second standard deviation based on the first ISO value by using the following formula (15).

[0165]

[0166] Among them, σ r is the second standard deviation, is the generating function of the second standard deviation.

[0167] After obtaining the first ISO value and M second reference noises, the second noise unit can calculate the second noise using the following formula (16).

[0168]

[0169] in, is the second noise, σ r is the second standard deviation, z r is the second reference noise, and ⊙ is the dot multiplication operation.

[0170] In one embodiment, the noise adding model further includes: a dark current noise module, and the step of inputting the acquisition parameters into the noise adding model to obtain the noise output by the noise adding model includes:

[0171] The acquisition parameters are input into the dark current noise module to obtain dark current noise output by the dark current noise module, wherein the dark current noise is obtained after the dark current noise module adjusts M*N preset noises based on the acquisition parameters.

[0172] In the electronic device of the embodiment of the present application, M*N preset noises are pre-stored. Based on this, after the acquisition parameters when acquiring the first image are input into the dark current noise module, the dark current noise module can generate a third parameter based on the acquisition parameter. Thereafter, the M*N preset noises stored in the electronic device are adjusted based on the third parameter, and the adjusted M*N preset noises are used as dark current noises.

[0173] The M*N preset noises mentioned above are dark current noises learned by the noise adding model from the input training image during the training process of the noise adding model.

[0174] Exemplarily, when the acquisition parameter is the first ISO value, the dark current noise module generates a third parameter based on the acquisition parameter, and the implementation process of generating dark current noise based on the third parameter can be expressed by the following formula (17).

[0175] f FP (ISO) = g FP (ISO)⊙N f (17)

[0176] Among them, f FP (ISO) is the dark current noise, N f is the preset noise, g FP () is the generating function of the third parameter, and ⊙ is the dot multiplication operation.

[0177] In one embodiment, the noise adding model further includes: a black level noise module, and the inputting of the acquisition parameters into the noise adding model to obtain the noise output by the noise adding model further includes:

[0178] The acquisition parameters are input into the black level noise module to obtain black level error noise corresponding to the acquisition parameters, which is determined by the black level noise module from a plurality of preset black level error noises based on the acquisition parameters.

[0179] From the above introduction, it can be known that a plurality of preset black level error noises are stored for each pixel unit of the image sensor in the electronic device, wherein each preset black level error noise corresponds to an ISO value.

[0180] Based on this, when the acquisition parameter in the input noise addition model is the first ISO value, the black level noise module of the noise addition model can determine the black level error noise corresponding to the first ISO value for each pixel from multiple preset black level error noises to obtain M*N black level error noises.

[0181] In one embodiment, the noise adding model includes a first noise module, a dark current noise module and a black level noise module, and adding the noise to the first image to obtain the second image includes:

[0182] At least one of the first type of noise output by the first noise module, the dark current noise output by the dark current noise module, and the black level error noise output by the black level noise module is added to the first image to obtain the second image.

[0183] Exemplarily, a second noise image may be generated based on the first noise, a third noise image may be generated based on the second noise, and the second image may be obtained by fusing the second noise image and / or the third noise image with the first image.

[0184] Exemplarily, the implementation process of adding dark current noise to the first image through the dark current noise module of the noise addition model can be expressed by formula (18).

[0185]

[0186] in, is an image with dark current noise, x1 is an image without dark current noise, N f is the preset noise, g FP () is the generating function of the third parameter, and ⊙ is the dot multiplication operation.

[0187] The implementation process of adding black level error noise to the first image through the black level noise module of the noise addition model can be expressed by formula (19).

[0188]

[0189] in, is the image after adding black level error noise, x2 is the image before adding black level error noise, D BLE (ISO) is the black level error noise.

[0190] Exemplarily, after obtaining the above-mentioned first noise, second noise, dark current noise and black level error noise, at least two of the first noise, second noise, dark current noise and black level error noise can be fused to obtain a fourth noise image, and then the fourth noise image can be fused with the first image to obtain the second image.

[0191] Among them, the process of fusing the fourth noise image with the first image to obtain the second image can be parameterized by the above-mentioned implementation process of fusing the first noise image with the first image to obtain the second image, and the embodiments of the present application will not be repeated here.

[0192] Exemplarily, a second noise image generated based on the first noise and a third noise image generated based on the second noise may be fused to obtain a fifth noise image, and then at least one of black level error noise and dark current noise may be added to the fifth noise image to obtain a sixth noise image, and then the sixth noise image may be fused with the first image to obtain a second image.

[0193] Among them, the implementation process of fusing the sixth noise image with the first image to obtain the second image can refer to the implementation process of fusing the first noise image with the first image to generate the second image, and the embodiments of the present application will not be repeated here.

[0194] In one embodiment, the noise adding model further includes a quantization noise module. Since the number of bits that can represent the pixel value of each pixel in the image is limited, after converting the analog signal collected from the image sensor into a digital signal, the obtained digital signal needs to be quantized into a pixel value, and quantization noise is generated in the process of quantizing the digital signal into a pixel value. The size of the quantization noise is related to the number of bits used to represent the pixel value of the first image.

[0195] Based on this, after the first image is input into the noise adding model, the quantization noise module included in the noise adding model can also add quantization noise into the first image to obtain a second image with quantization noise.

[0196] Exemplarily, the implementation process of adding the first noise, the second noise, the dark current noise, the black level error noise and the quantization noise to the first image by the noise addition model can be expressed by the following formula (20).

[0197]

[0198] Where D is the second image, I is the first image, is the second noise, is the first noise,

[0199] is the black level error noise, is the dark current noise, N q is the quantization noise.

[0200] Exemplarily, the noise adding model can also be implemented by adding the first noise, the second noise, the dark current noise, the black level error noise and the quantization noise to the first image as follows: Figure 2 shown.

[0201] In one embodiment, the above-mentioned noisy training image is an image captured by an image acquisition device in a dark field environment.

[0202] Since the image acquisition device cannot receive incident light from the external environment or the received incident light is relatively weak in a dark field environment, most of the pixel values ​​included in the image captured by the image acquisition device are noise. That is, the image captured by the image acquisition device in a dark field environment is mainly a noise image.

[0203] Based on this, by training the denoising model with training images that are mainly noise, the denoising model's ability to recognize various real noises such as shot noise, dark current noise, black level error noise, read noise and line noise can be improved, so that the trained denoising model can add more realistic noise to the first image.

[0204] It should be noted that when the image acquisition device acquires training images in a dark field environment, the ISO value of the image acquisition device can be changed to obtain training images at different ISO values, so that the noise addition model can learn the distribution characteristics of shot noise, dark current noise, black level error noise, read noise and line noise at different ISO values.

[0205] When the noisy training image used to train the noise adding model is an image captured in a dark field environment, the noise-free training image may be an image whose maximum pixel value is less than a preset value, or an image whose pixel value is close to zero.

[0206] In one embodiment, multiple image acquisition devices with different performances may be used to acquire images under low light conditions or in dark environments, and the acquired images may be used as training images for training the noise addition model.

[0207] For example, image acquisition can be performed under low light conditions or dark field environments using image acquisition devices with different types of image sensors or different lenses, and the acquired images can be used as training images for training the noise addition model.

[0208] In one embodiment, the method further comprises:

[0209] Training the target model based on the second image to obtain a denoising model, wherein the denoising model has image denoising capability;

[0210] Based on the denoising model, denoising is performed on the third image with noise.

[0211] After obtaining the second image with noise, the second image with noise can be input into the target model as a training set, and the second image can be denoised by the target model to realize the training of the target model until a preset number of cycles is reached or the target model converges. The trained target model is the denoising model in the embodiment of the present application.

[0212] The target model may be any network model that can remove noise from an image, such as a convolutional neural network model, an NLM (Non-Local means) model, a recurrent neural network model, or an adversarial network model.

[0213] In an embodiment of the present application, by obtaining a training image for training a denoising model through a denoising model, various required noises can be added to the training image. By training the denoising model with such training images, the denoising model's ability to recognize and remove various noises can be improved.

[0214] In one embodiment, after the denoising model is trained, the third image captured by the electronic device may be denoised, and the fourth image and the third image obtained after the denoising model is denoised may be displayed.

[0215] When the electronic device displays the third image and the fourth image, the third image or the fourth image may be stored based on a selection operation of the user.

[0216] In the embodiment of the present application, considering that the denoising effect of denoising the image captured by the electronic device through the denoising model may not necessarily satisfy the user, therefore, after the image captured by the electronic device is denoised through the denoising model, the denoised image and the original image captured by the electronic device can be displayed to allow the user to determine which image to store, which can further improve the user experience.

[0217] refer to Figure 3 and Figure 4 , Figure 3 is a first image captured by an image acquisition device, Figure 4 It is a second image obtained after adding noise to the first image through the noise addition model of the embodiment of the present application.

[0218] refer to Figures 2 to 4 , the embodiment of the present application also provides another image processing method, including:

[0219] Noise Modeling

[0220] In the embodiment of the present application, the image noise is divided into five parts according to the imaging process of the digital image, as shown in formula (21):

[0221]

[0222] in, is a noisy image, I is a clean image, N shot is the shot noise, N DC is the dark current noise,

[0223] N read is the read noise, N row is the line noise, N q is the quantization noise.

[0224] The following is the noise model of the above noises:

[0225] Shot noise is a kind of inevitable physical phenomenon that is caused by the uncertainty generated when photons are incident. Usually, shot noise is modeled by Poisson distribution controlled by signal and system gain. In the embodiment of the present application, shot noise with Poisson distribution is approximated as Gaussian distribution, as shown in the above formula (1):

[0226] N shot ~(0,β shot ·I) (1)

[0227] Among them, N shot is the shot noise, β shot It is positively correlated with the ISO value of the image acquisition device, and I is the pixel value of the first image.

[0228] Dark current noise: When the image acquisition device converts light into a digital signal, it uses the photoelectric effect to generate a current, and the value of the current represents the light intensity. However, in addition to the current generated by photons, the thermal effect in the sensor also generates a current called dark current, which generates dark current noise. Ideally, the dark current generated by the thermal effect can be subtracted, but the randomness, ISO dependence, and spatial non-uniformity of the thermal effect also need to be considered. Therefore, a three-component third noise module is established based on the physical characteristics of dark current noise, as shown in formula (22).

[0229] N DC = k·N FP +N BLE +N DCSN (twenty two)

[0230] Among them, N DC is the third noise, k is the first coefficient, k is positively correlated with the ISO value of the image acquisition device, N BLE is the black level error noise. N FP is the dark current fixed pattern noise, N DCSN is random thermal noise.

[0231] Dark current pulse noise representing the randomness of thermal effects, that is, thermal random noise in the embodiments of the present application.

[0232] Read noise and row noise: In the process of reading out the signal from the image sensor, the uncertainty of the electronic readout will inevitably produce read noise, which is a kind of noise related to the performance of the image sensor. In addition, in the image sensor, each column shares an analog-to-digital converter (ADC), that is, the reading process is performed row by row. The randomness of the ADC between different rows will cause row noise, which has a great influence on the horizontal striping artifact. These two noises can be modeled by the following zero-mean Gaussian distribution, as shown in equations (4) and (5).

[0233]

[0234]

[0235] Among them, N read is the read noise, N read ∈R M*N , is the variance of the read noise; N row is the line noise, N row ∈R M , is the variance of the row noise.

[0236] Quantization noise: Considering the limited accuracy of ADC, the signal needs to be quantized before it is stored as an image. It is generally believed that the probability distribution of quantization noise is uniform, such as the following equation (23):

[0237] N q ~(-1 / 2q,1 / 2q) (23)

[0238] Among them, N q is the quantization noise, q is the quantization step. Since the number of bits of the digital image is known, there is no need to estimate q. In actual use, due to the different number of bits used for storage, the q value is usually between 8 and 16.

[0239] Heteroscedastic Gaussian noise: Considering that shot noise, second dark current noise, and readout noise are all Gaussian distributions with a single pixel as the unit, a new concept of heteroscedastic Gaussian distribution is introduced here to model them, as shown in formula (6).

[0240] N HG ~N(0,β1·I+β2) (6)

[0241] Among them, N HG is heteroscedastic Gaussian noise, β1 is the first coefficient, β1 is the coefficient related to ISO, and β1 can be the above β shot , β2 is the second coefficient, and β2 is a coefficient related to random thermal noise and read noise.

[0242] According to the signal modeling method, the final noise image representation can be expressed as (24):

[0243]

[0244] in, is the second image, I is the first image, k is the first coefficient, N FP is the dark current fixed pattern noise, N BLE is the black level error noise, N row is the line noise, N q is the quantization noise

[0245] Noise model construction

[0246] like Figure 2 As shown, the noise model proposed in the embodiment of the present application is implemented in a normalized flow-based framework, and four components are deployed to model different types of noise.

[0247] First, the normalized flow-based framework is briefly introduced, followed by a detailed description of each component.

[0248] Stream-based normalization framework

[0249] The normalized flow based model consists of several differentiable functions and a bilateral mapping function that learns a transformation z = f(xΘ) with parameters Θ.

[0250] In the embodiment of the present application, the normalized flow-based noise model can transform the data sample x i From the real extremely low light noise P x Transform to z-space with a Gaussian distribution.

[0251] According to the variable transformation formula, the probability density function of the real extremely low optical noise space is shown in formula (25):

[0252] P x (x) = P z (f(x|Θ))|detJf(x|Θ)| (25)

[0253] Where Jf(x) represents the Jacobian matrix of function f at x, det represents the determinant of the input matrix, |·| represents the absolute value, and Θ is the model parameter of the noise model. Under the condition of , the mapping function f can be optimized by minimizing the negative log-likelihood function, as shown in formula (26).

[0254]

[0255] After training, we can get the Gaussian distribution P zA first reference noise z is collected in the -1 It can be obtained by x=f -1 (z) Synthetic real noise.

[0256] First dark current module:

[0257] In order to model the dark current noise that remains constant in all images, a learnable tensor N is defined in the embodiment of the present application. f ∈R M*N Then a multi-layer perceptron (MLP) is deployed to learn the first coefficient k in the aforementioned formula (3). Therefore, a dark current noise module is defined as shown in formula (7).

[0258] f FP (x3) = x3 - g FP (ISO)⊙N f (7)

[0259] Among them, f FP (x3) is the training image after removing the dark current noise, x3 is the training image with dark current noise, g FP (ISO) The third parameter, N f is the preset noise, and ⊙ is the dot multiplication operation.

[0260] The inverse mapping of formula (7) is shown in formula (18).

[0261]

[0262] in, is the image after adding dark current noise, x1 is the image before adding dark current noise, g FP (ISO) is the third parameter, N f is the preset noise, and ⊙ is the dot multiplication operation.

[0263] Since the black level error noise is nonlinearly related to ISO, and given that the number of ISO configurations is limited and not infinitely adjustable, in the embodiment of the present application, the black level error noise corresponding to each ISO configuration is modeled. Here, the bilateral mapping of the black level noise module is represented by formulas (19) and (8).

[0264]

[0265] Among them, f BLE (x4) is the image with black level error noise removed, x4 is the image with black level error noise, N BLE (ISO) is the black level error noise, x2 is the image without black level error noise, This is the image after adding black level error noise.

[0266] Noise Unit 1

[0267] This module maps the heteroscedastic Gaussian noise to the standard Gaussian distribution based on the heteroscedastic Gaussian noise model in formula (6). The specific mapping formulas are shown in formulas (27) and (11).

[0268]

[0269]

[0270] Wherein, β1 is the first parameter, β1 is a parameter related to the shot noise of the image, β2 is the second parameter, β2 is a parameter related to the thermal random noise and read noise of the image, I is the pixel value of the first image, x5 is the image with the first reference noise, z is the first reference noise, σ HG is the first standard deviation.

[0271] The above β1 and β2 can be obtained by fitting through formulas (12) and (13) based on the two multi-layer perceptrons included in the first noise unit.

[0272] β1=gβ1(ISO) (12)

[0273] β2=gβ2(ISO) (13)

[0274] In the test phase of the noise model, the operation of this noise unit can be expressed as formula (14).

[0275]

[0276] in, is the first noise, z is the first reference noise, σ HG is the first standard deviation.

[0277] Second noise unit

[0278] The second noise unit is based on the row noise model in equation (5), which is further applied to the network as a flow-based layer, as shown in equations (28) and (15).

[0279]

[0280]

[0281] Where h(x6) is the image with the second noise, z r is the second reference noise, σ r is the second standard deviation, is a function used to indicate the second noise unit structure.

[0282] In the model testing stage, the second noise unit of the noise-added model can be expressed by the following formula (16).

[0283]

[0284] in, is the second noise, σ r is the second standard deviation, z r is the second reference noise, and ⊙ is the dot multiplication operation.

[0285] Model training strategy:

[0286] The noise source can be roughly divided into fixed pattern (time-invariant) or random (time-varying) noise. In the training phase, according to the law of noise generation, it is also divided into fixed pattern noise module and random noise module, and noise generation is performed based on the fixed pattern noise module and random noise, as shown in formula (29).

[0287]

[0288] The network architecture of the random noise modeling stage can be expressed by formula (30):

[0289]

[0290] Its loss function can be expressed by formula (9).

[0291]

[0292] Then, the parameters of the noise modeling stage can be fixed, and the network structure of the fixed pattern noise modeling stage is shown in formula (31).

[0293] f3(x i |Θ3)=f HG (f BLE (f FP (x i ))) (31)

[0294] Its loss function can be expressed by formula (10).

[0295]

[0296] Noise Synthesis Scheme

[0297] After the denoising model is trained, it can be fed with a clean noise-free image and ISO parameters to generate a realistic noisy image. For a given image I, the synthesized noisy image It can be expressed by formula (20).

[0298]

[0299] refer to Figure 5 , the embodiment of the present application further provides an image processing device, including:

[0300] An acquisition module 501 is used to acquire a first image and acquisition parameters of the first image;

[0301] An input module 502 is used to input the acquisition parameters into the noise adding model to obtain the noise output by the noise adding model, wherein the noise adding capability of the noise adding model is obtained by training based on noise-free training images and noise-containing training images during a training process;

[0302] The adding module 503 is used to add the noise to the first image to obtain the second image.

[0303] Optionally, the noise addition model includes multiple noise modules; different noise modules are used to generate different types of noise.

[0304] Optionally, the noise adding model includes a first noise module; and the input module is used for:

[0305] The acquisition parameters are input into the first noise module to obtain a first type of noise output by the first noise module based on random processing.

[0306] Optionally, the first type of noise includes at least one of the following:

[0307] M*N first noises, wherein the m*nth first noise is used to be superimposed with the m*nth pixel in the first image to obtain a pixel value of the m*nth pixel in the second image;

[0308] M second noises, wherein the mth second noise is used to be superimposed on the mth row of pixels in the first image to obtain pixel values ​​of the mth row of pixels in the second image;

[0309] Among them, M is the total number of pixel rows of the first image; N is the total number of pixel columns of the first image; m is a positive integer less than or equal to M; and n is a positive integer less than or equal to N.

[0310] Optionally, the first noise module includes: a random unit and a first noise unit;

[0311] The input module is specifically used for:

[0312] Obtaining M*N first reference noises by utilizing random processing of the random unit;

[0313] The M*N first reference noises and the acquisition parameters are input into a first noise unit to obtain the M*N first noises output by the first noise unit.

[0314] Optionally, the input module is specifically used to include:

[0315] The first noise unit determines a first standard deviation based on the acquisition parameters and the pixel values ​​of the first image;

[0316] The M*N first reference noises are adjusted based on the first standard deviation to obtain the M*N first noises.

[0317] Optionally, the first noise module further includes: a second noise unit; and the input module is further configured to:

[0318] Obtaining M second reference noises by random processing of the random unit;

[0319] The M second reference noises and the acquisition parameters are input into the second noise unit to obtain the M second noises output by the second noise unit.

[0320] Optionally, the noise adding model further includes: a dark current noise module, and the input module is further used for:

[0321] The acquisition parameters are input into the dark current noise module to obtain dark current noise output by the dark current noise module, wherein the dark current noise is obtained after the dark current noise module adjusts M*N preset noises based on the acquisition parameters.

[0322] Optionally, the noise adding model further includes: a black level noise module, and the input module is further used for:

[0323] The acquisition parameters are input into the black level noise module to obtain black level error noise corresponding to the acquisition parameters, which is determined by the black level noise module from a plurality of preset black level error noises based on the acquisition parameters.

[0324] Optionally, the noise adding model includes a first noise module, a dark current noise module and a black level noise module, and the adding module is used to:

[0325] At least one of the first type of noise output by the first noise module, the dark current noise output by the dark current noise module, and the black level error noise output by the black level noise module is added to the first image to obtain the second image.

[0326] Optionally, the noisy training image is an image captured by an image capture device in a dark field environment.

[0327] Optionally, the device further comprises:

[0328] A training module, used for training a target model based on the second image to obtain a denoising model; the denoising model has image denoising capability;

[0329] The denoising module is used to perform denoising processing on the third image with noise based on the denoising model.

[0330] refer to Figure 6 In an embodiment of the present application, an electronic device 600 is provided, including:

[0331] Memory 604 for storing processor executable instructions;

[0332] A processor 620 connected to the memory 604;

[0333] The processor 620 is configured to execute the image processing method provided by any of the aforementioned technical solutions.

[0334] A block diagram of an electronic device 600 is shown according to an exemplary embodiment. For example, the electronic device 600 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc. having a display screen.

[0335] refer to Figure 6 , the electronic device 600 may include one or more of the following components: a processing component 602 , a memory 604 , a power component 606 , a multimedia component 608 , an audio component 610 , an input / output (I / O) interface 612 , a sensor component 614 , and a communication component 616 .

[0336] The processing component 602 generally controls the overall operation of the electronic device 600, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 602 may include one or more modules to facilitate the interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate the interaction between the multimedia component 608 and the processing component 602.

[0337] The memory 604 is configured to store various types of data to support operations on the electronic device 600. Examples of such data include instructions for any application or method operating on the electronic device 600, contact data, phone book data, messages, pictures, videos, etc. The memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0338] The power supply component 606 provides power to the various components of the electronic device 600. The power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 600.

[0339] The multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 608 includes a front camera and / or a rear camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0340] The audio component 610 is configured to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC), and when the electronic device 600 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 604 or sent via the communication component 616. In some embodiments, the audio component 610 also includes a speaker for outputting audio signals.

[0341] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0342] The sensor assembly 614 includes one or more sensors for providing various aspects of status assessment for the electronic device 600. For example, the sensor assembly 614 can detect the open / closed state of the electronic device 600, the relative positioning of the components, such as the display and keypad of the electronic device 600, and the sensor assembly 614 can also detect the position change of the electronic device 600 or a component of the electronic device 600, the presence or absence of contact between the user and the electronic device 600, the orientation or acceleration / deceleration of the electronic device 600, and the temperature change of the electronic device 600. The sensor assembly 614 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 614 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0343] The communication component 616 is configured to facilitate wired or wireless communication between the electronic device 600 and other devices. The electronic device 600 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 616 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0344] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0345] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, and the instructions can be executed by a processor 620 of an electronic device 600 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0346] An embodiment of the present application provides a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by a processor of a computer, the computer can execute the image processing method described in one or more of the aforementioned technical solutions.

[0347] When the processor executes the instruction, it can at least perform the following steps:

[0348] Acquire a first image and acquisition parameters of the first image;

[0349] Inputting the acquisition parameters into the noise adding model to obtain the noise output by the noise adding model, wherein the noise adding capability of the noise adding model is obtained by training based on noise-free training images and noise-containing training images during a training process;

[0350] The noise is added to the first image to obtain the second image.

[0351] Optionally, the performing noise processing on the first image based on the acquisition parameters and the noise model to obtain the second image includes:

[0352] Inputting the acquisition parameters into the noise adding model to obtain the noise output by the noise adding model;

[0353] The noise is added to the first image to obtain the second image.

[0354] Optionally, the noise addition model includes multiple noise modules; different noise modules are used to generate different types of noise.

[0355] Optionally, the noise module includes a first noise module; and the step of inputting the acquisition parameters into the noise adding model to obtain the noise output by the noise adding model includes:

[0356] The acquisition parameters are input into the first noise module to obtain a first type of noise output by the first noise module based on random processing.

[0357] Optionally, the first type of noise includes at least one of the following:

[0358] M*N first noises, wherein the m*nth first noise is used to be superimposed with the m*nth pixel in the first image to obtain a pixel value of the m*nth pixel in the second image;

[0359] M second noises, wherein the mth second noise is used to be superimposed on the mth row of pixels in the first image to obtain pixel values ​​of the mth row of pixels in the second image;

[0360] Among them, M is the total number of pixel rows of the first image; N is the total number of pixel columns of the first image; m is a positive integer less than or equal to M; and n is a positive integer less than or equal to N.

[0361] Optionally, the first noise module includes: a random unit and a first noise unit;

[0362] The step of inputting the acquisition parameters into the first noise module to obtain a first type of noise output by the first noise module based on random processing includes:

[0363] Obtaining M*N first reference noises by utilizing random processing of the random unit;

[0364] The M*N first reference noises and the acquisition parameters are input into a first noise unit to obtain the M*N first noises output by the first noise unit.

[0365] Optionally, the step of inputting the M*N first reference noises and the acquisition parameters into a first noise unit, so that the first noise unit outputs the M*N first noises, comprises:

[0366] The first noise unit determines a first standard deviation based on the acquisition parameters and the pixel values ​​of the first image;

[0367] The M*N first reference noises are adjusted based on the first standard deviation to obtain the M*N first noises.

[0368] Optionally, the first noise module further includes: a second noise unit; the step of inputting the acquisition parameters into the first noise module to obtain a first type of noise output by the first noise module based on random processing includes:

[0369] Obtaining M second reference noises by random processing of the random unit;

[0370] The M second reference noises and the acquisition parameters are input into the second noise unit to obtain the M second noises output by the second noise unit.

[0371] Optionally, the noise adding model further includes: a dark current noise module, and the step of inputting the acquisition parameters into the noise adding model to obtain the noise output by the noise adding model includes:

[0372] The acquisition parameters are input into the dark current noise module to obtain dark current noise output by the dark current noise module, wherein the dark current noise is obtained after the dark current noise module adjusts M*N preset noises based on the acquisition parameters.

[0373] Optionally, the noise module further includes: a black level noise module, wherein the acquisition parameters are input into the noise adding model to obtain the noise output by the noise adding model, and further includes:

[0374] The acquisition parameters are input into the black level noise module to obtain black level error noise corresponding to the acquisition parameters, which is determined by the black level noise module from a plurality of preset black level error noises based on the acquisition parameters.

[0375] Optionally, the noise adding model includes a first noise module, a dark current noise module and a black level noise module, and adding the noise to the first image to obtain the second image includes:

[0376] At least one of the first type of noise output by the first noise module, the dark current noise output by the dark current noise module, and the black level error noise output by the black level noise module is added to the first image to obtain the second image.

[0377] Optionally, the noisy training image is an image captured by an image capture device in a dark field environment.

[0378] Optionally, the method further comprises:

[0379] Training a second target model based on the second image to obtain a denoising model; the denoising model has image denoising capability;

[0380] Based on the denoising model, denoising is performed on the third image with noise.

[0381] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0382] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a first image and acquisition parameters of the first image; Inputting the acquisition parameters into a noise adding model to obtain the noise output by the noise adding model, wherein the noise adding capability of the noise adding model is obtained by training based on noise-free training images and noise-containing training images during a training process; The noise is added to the first image to obtain a second image.

2. The method according to claim 1, characterized in that The noise adding model includes a plurality of noise modules; different noise modules are used to generate different types of noise.

3. The method according to claim 1 or 2, characterized in that: The noise adding model includes a first noise module; the step of inputting the acquisition parameters into the noise adding model to obtain the noise output by the noise adding model includes: The acquisition parameters are input into the first noise module to obtain a first type of noise output by the first noise module based on random processing.

4. The method according to claim 3, characterized in that The first type of noise includes at least one of the following: M*N first noises, wherein the m*nth first noise is used to be superimposed with the m*nth pixel in the first image to obtain a pixel value of the m*nth pixel in the second image; M second noises, wherein the mth second noise is used to be superimposed on the mth row of pixels in the first image to obtain pixel values ​​of the mth row of pixels in the second image; Among them, M is the total number of pixel rows of the first image; N is the total number of pixel columns of the first image; m is a positive integer less than or equal to M; and n is a positive integer less than or equal to N.

5. The method according to claim 4, characterized in that The first noise module includes: a random unit and a first noise unit; The step of inputting the acquisition parameters into the first noise module to obtain a first type of noise output by the first noise module based on random processing includes: Obtaining M*N first reference noises by utilizing random processing of the random unit; The M*N first reference noises and the acquisition parameters are input into a first noise unit to obtain the M*N first noises output by the first noise unit.

6. The method according to claim 5, characterized in that The step of inputting the M*N first reference noises and the acquisition parameters into a first noise unit, so that the first noise unit outputs the M*N first noises, includes: The first noise unit determines a first standard deviation based on the acquisition parameters and the pixel values ​​of the first image; The M*N first reference noises are adjusted based on the first standard deviation to obtain the M*N first noises.

7. The method according to claim 5, characterized in that The first noise module further includes: a second noise unit; the step of inputting the acquisition parameters into the first noise module to obtain the first type of noise output by the first noise module based on random processing includes: Obtaining M second reference noises by random processing of the random unit; The M second reference noises and the acquisition parameters are input into the second noise unit to obtain the M second noises output by the second noise unit.

8. The method according to claim 1 or 2, characterized in that: The noise adding model further includes: a dark current noise module, wherein the acquisition parameters are input into the noise adding model to obtain the noise output by the noise adding model, including: The acquisition parameters are input into the dark current noise module to obtain dark current noise output by the dark current noise module, wherein the dark current noise is obtained after the dark current noise module adjusts M*N preset noises based on the acquisition parameters.

9. The method according to claim 1 or 2, characterized in that: The noise adding model further includes: a black level noise module, wherein the acquisition parameters are input into the noise adding model to obtain the noise output by the noise adding model, and further includes: The acquisition parameters are input into the black level noise module to obtain black level error noise corresponding to the acquisition parameters, which is determined by the black level noise module from a plurality of preset black level error noises based on the acquisition parameters.

10. The method according to claim 1 or 2, characterized in that: The noise adding model includes a first noise module, a dark current noise module and a black level noise module, and the adding of the noise to the first image to obtain the second image includes: At least one of the first type of noise output by the first noise module, the dark current noise output by the dark current noise module, and the black level error noise output by the black level noise module is added to the first image to obtain the second image.

11. The method according to claim 10, characterized in that The noisy training image is an image captured by an image acquisition device in a dark field environment.

12. The method according to claim 1, characterized in that The method further comprises: Training the target model based on the second image to obtain a denoising model; wherein the denoising model has image denoising capability; Based on the denoising model, denoising is performed on the third image with noise.

13. An image processing device, characterized in that: The device comprises: An acquisition module, used to acquire a first image and acquisition parameters of the first image; An input module, used for inputting the acquisition parameters into a noise adding model to obtain the noise output by the noise adding model, wherein the noise adding capability of the noise adding model is obtained by training based on noise-free training images and noise-containing training images during a training process; An adding module is used to add the noise to the first image to obtain a second image.

14. A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a computer, enables the computer to perform the image processing method according to any one of claims 1 to 12.

15. An electronic device, characterized in that: include: a memory for storing processor-executable instructions; A processor connected to the memory; Wherein, the processor is used to execute the image processing method as described in any one of claims 1 to 12.

Images