Image processing method and device, electronic equipment, and storage medium

Abstract

The present disclosure relates to an image processing method and device, electronic equipment and storage medium. The image processing method comprises: acquiring a collection image with an ambient illumination less than a preset illumination; acquiring analog gains corresponding to a plurality of sensitivities and dark field frame images corresponding to the plurality of sensitivities; the dark field frame images are acquired under the condition of no incident ambient light; and based on the analog gains, the dark field frame images and the collection image, noise images corresponding to the collection image under the plurality of sensitivities are obtained. According to the embodiments of the present disclosure, noise images in a dark light scene can be obtained.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of image processing, and particularly relates to an image processing method and device, an electronic device, and a storage medium. BACKGROUND

[0002] Digital images are inevitably disturbed by complex noise in the process of generation, storage and transmission. With the development of deep learning image denoising methods, deep learning-based image denoising methods have become the mainstream solution to image denoising problems. However, deep learning-based image denoising methods need to collect images and noise images corresponding to the collected images under different scenes as a data set. However, in a dark scene where the environmental illumination is less than a preset illumination, it is difficult to obtain noise images corresponding to the collected images under different exposure parameters. SUMMARY

[0003] To overcome the problems in the related art, the present disclosure provides an image processing method and device, an electronic device, and a storage medium, which can obtain noise images in a dark scene.

[0004] According to a first aspect of an embodiment of the present disclosure, an image processing method is provided, at least comprising:

[0005] obtaining a collected image with an environmental illumination less than a preset illumination;

[0006] obtaining a plurality of analog gains corresponding to a plurality of sensitivities and a plurality of dark field frame images corresponding to the plurality of sensitivities; the dark field frame image is obtained under the condition of no incident environmental light;

[0007] obtaining a plurality of noise images corresponding to the collected image under the plurality of sensitivities based on the analog gains, the dark field frame images and the collected image.

[0008] In some embodiments, the obtaining a plurality of noise images corresponding to the collected image under the plurality of sensitivities based on the analog gains, the dark field frame images and the collected image comprises:

[0009] obtaining a plurality of light noise images related to the incident environmental light under the plurality of sensitivities based on the analog gains and the collected image;

[0010] obtaining the noise images corresponding to the collected image under the plurality of sensitivities based on the light noise images and the dark field frame images corresponding to the same sensitivity under the plurality of sensitivities.

[0011] In some embodiments, the obtaining a plurality of light noise images related to the incident environmental light under the plurality of sensitivities based on the analog gains and the collected image comprises:

[0012] Poisson distribution processing is performed on a ratio between the collected image and the analog gain to obtain a Poisson processed image;

[0013] Based on a product of the Poisson processed image and the analog gain, a plurality of the light noise images related to the incident ambient light under a plurality of the ISO sensitivities are obtained.

[0014] In some embodiments, based on the light noise image and the dark field frame image corresponding to the same ISO sensitivity under a plurality of the ISO sensitivities, the noise image corresponding to the collected image under a plurality of the ISO sensitivities is obtained, including:

[0015] Based on a sum of the light noise image and the dark field frame image corresponding to the same ISO sensitivity under a plurality of the ISO sensitivities, the noise image corresponding to the collected image under a plurality of the ISO sensitivities is obtained.

[0016] In some embodiments, the method further includes:

[0017] Mean value and variance calculation is performed on all pixel points surrounded by a plurality of gray scale regions under different ISO sensitivities to obtain a plurality of mean values corresponding to a plurality of the gray scale regions under each ISO sensitivity and a plurality of variances corresponding to a plurality of the gray scale regions under each ISO sensitivity; wherein the plurality of the gray scale regions are obtained by dividing a preset gray scale image into different gray scale regions;

[0018] Fitting is performed on the plurality of mean values corresponding to a plurality of the gray scale regions and the plurality of variances corresponding to a plurality of the gray scale regions to obtain a fitting line under each ISO sensitivity.

[0019] Based on a slope of the fitting line under each ISO sensitivity, the analog gain corresponding to a plurality of the ISO sensitivities is obtained.

[0020] In some embodiments, the fitting is performed on the plurality of mean values corresponding to a plurality of the gray scale regions and the plurality of variances corresponding to a plurality of the gray scale regions to obtain a fitting line under each ISO sensitivity, including:

[0021] The mean value and the variance corresponding to the same gray scale region are associated;

[0022] The mean value corresponding to each gray scale region is taken as the abscissa, the variance corresponding to each gray scale region is taken as the ordinate, and a fitting line under each ISO sensitivity is obtained by fitting.

[0023] In some embodiments, the method further includes:

[0024] The preset denoising network model is trained based on the noise image and the acquisition image, to obtain a trained denoising network model, which is used for denoising a captured shooting image.

[0025] According to a second aspect of the embodiments of the present disclosure, an image processing method is provided, at least including:

[0026] An acquisition image is acquired, which is captured under the condition that the ambient illumination is less than a preset illumination;

[0027] The shooting image is denoised based on a noise image or a denoising network model, to obtain a denoised image;

[0028] The noise image is obtained based on an analog gain corresponding to a plurality of sensitivities and a dark field frame image, the dark field frame image being captured under the condition that there is no incident ambient light; and the denoising network model is trained based on the noise image.

[0029] According to a third aspect of the embodiments of the present disclosure, an image processing apparatus is provided, at least including:

[0030] A first acquisition module is configured to acquire an acquisition image, which is captured under the condition that the ambient illumination is less than a preset illumination;

[0031] A second acquisition module is configured to acquire an analog gain corresponding to a plurality of sensitivities and a dark field frame image corresponding to the plurality of sensitivities, the dark field frame image being captured under the condition that there is no incident ambient light;

[0032] A noise image generation module is configured to obtain a noise image corresponding to the acquisition image under the plurality of sensitivities based on the analog gain, the dark field frame image, and the acquisition image.

[0033] According to a fourth aspect of the embodiments of the present disclosure, an image processing apparatus is provided, at least including:

[0034] A third acquisition module is configured to acquire a shooting image, which is captured under the condition that the ambient illumination is less than a preset illumination;

[0035] A denoising processing module is configured to denoise the shooting image based on a noise image or a denoising network model, to obtain a denoised image;

[0036] The noise image is obtained based on an analog gain corresponding to a plurality of sensitivities and a dark field frame image, the dark field frame image being captured under the condition that there is no incident ambient light; and the denoising network model is trained based on the noise image.

[0037] According to a fifth aspect of the embodiments of the present disclosure, an electronic device is provided, at least including:

[0038] a processor;

[0039] a memory for storing processor-executable instructions;

[0040] The processor is configured to perform the image processing method according to the first aspect above; or is configured to perform the image processing method according to the second aspect above.

[0041] According to a sixth aspect of the embodiments of the present disclosure, a storage medium is provided, comprising:

[0042] When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device can perform the image processing method according to the first aspect above; or is configured to perform the image processing method according to the second aspect above.

[0043] The technical solutions provided by the embodiments of the present disclosure can include the following beneficial effects:

[0044] The embodiments of the present disclosure can obtain noise images of different sensitivities based on the collected images of dark light scenes, i.e., the collected images of the environment illumination less than the preset illumination, so that it is more convenient to obtain the noise images of the dark light scenes, and thus the problem of difficulty in obtaining the noise images in the dark light scenes can be solved.

[0045] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0046] The accompanying drawings, which are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure.

[0047] Figure 1 is a flowchart of an image processing method according to an exemplary embodiment Figure One .

[0048] Figure 2 is a gray scale chart for dividing a gray scale region according to an exemplary embodiment.

[0049] Figure 3 is a fitting line diagram under sensitivity A according to an exemplary embodiment.

[0050] Figure 4 is a diagram of fitting of sensitivity and analog gain according to an exemplary embodiment.

[0051] Figure 5 is a diagram of generating a noise image according to an exemplary embodiment.

[0052] Figure 6 is a comparison diagram of a noise image (left) and a noise image corresponding to a captured image (right) according to an example embodiment.

[0053] Figure 7 is a flowchart of an image processing method according to an example embodiment Figure Two .

[0054] Figure 8 is a schematic diagram of an image processing device according to an example embodiment Figure One .

[0055] Figure 9 is a schematic diagram of an image processing device according to an example embodiment Figure Two .

[0056] Figure 10 is a block diagram of an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0057] The example embodiments will be described in detail herein with reference to the attached drawings. In the following description, the same numbers are used to designate the same elements, unless otherwise indicated. The embodiments described in the following example embodiments are not representative of all embodiments consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure, as detailed in the appended claims.

[0058] The embodiments of the present disclosure propose an image processing method, applied to an electronic device. Figure 1 is a flowchart of an image processing method according to an example embodiment Figure One . As shown in Figure 1 , the electronic device implements the image processing method comprising the following steps:

[0059] S101, acquiring a captured image with an ambient illuminance less than a preset illuminance;

[0060] S102, acquiring a plurality of analog gains corresponding to a plurality of sensitivities and a plurality of dark field frame images corresponding to the plurality of sensitivities; the dark field frame images are acquired under the condition of no incident ambient light;

[0061] S103, obtaining a plurality of noise images corresponding to the captured image under the plurality of sensitivities based on the analog gains, the dark field frame images, and the captured image.

[0062] In the embodiments of the present disclosure, the image processing method is applicable to a dark light scene with an environment illumination less than a preset illumination. For example, in the process of obtaining the training set of the preset denoising network model, the electronic device can obtain a noise image corresponding to the collected image in the dark light scene by using the image processing method. In this way, the noise image corresponding to the collected image in the dark light scene is added to the training set, so that the training set is more abundant, and the accuracy of denoising by using the denoising network model trained based on the training set can be improved.

[0063] It should be noted that the electronic device can include a terminal device and a server, and the terminal device can include a mobile phone, a desktop computer, a notebook computer, and the like. In the case where the electronic device is a terminal device, the server can send the collected image, the analog gain, and the dark field frame image to the terminal device, so that the terminal device executes the image processing method to generate the noise information corresponding to the collected object. Of course, the terminal device can also obtain the collected image, the analog gain, and the dark field frame image based on its own database, and the embodiments of the present disclosure do not limit this.

[0064] In step S101, the collected image with an environment illumination less than a preset illumination indicates that the collected image is a dark light image or a low-illumination image. Here, the preset illumination can be set according to design requirements, and for example, the preset illumination can be set to be less than or equal to 0.1 lux.

[0065] In the embodiments of the present disclosure, the electronic device can store a collected image database. The obtained collected image can be used as a sample to establish the collected image database, and the process of obtaining the collected image can be to select one of a plurality of samples in the collected image database as the collected image.

[0066] In the process of establishing the collected image database, because the brightness of the image generated by the image sensor of the electronic device is proportional to the environment illumination under the restriction of the imaging device and the environment in the dark light scene, the collected image can be obtained by adjusting the exposure parameter in the embodiments of the present disclosure.

[0067] In some embodiments, obtaining the collected image with the environment illumination less than the preset illumination can include: obtaining a photographed image under the condition that the environment illumination is greater than or equal to the preset illumination; and adjusting the photographed image based on a preset exposure coefficient to obtain the collected image.

[0068] Here, the photographed image under the condition that the environment illumination is greater than or equal to the preset illumination is a clear image, and the collected image is obtained by adjusting the clear image in the embodiments of the present disclosure. In this way, the embodiments of the present disclosure can obtain a clear and dark light collected image.

[0069] The above-mentioned preset exposure coefficient is set by the electronic device according to requirements, and the embodiments of the present disclosure do not limit this.

[0070] It should be noted that the brightness of the captured image generated by the image sensor of the electronic device is proportional to the ambient illuminance according to the captured image gray value and the ambient illuminance, as shown in the following formula (1):

[0071] f = i * r (1)

[0072] wherein f represents the captured image gray value, i represents the incident component, which depends on the ambient illuminance, and r represents the reflection component, which depends on the characteristics of the imaged object.

[0073] It can be known from formula (1) that the captured image gray value is proportional to the ambient illuminance. In the embodiment of the present disclosure, adjusting the gray value of the captured image corresponds to adjusting the ambient illuminance corresponding to the captured image. That is, by adjusting the gray value corresponding to the captured image, the collected image with the ambient illuminance less than the preset illuminance can be obtained, as shown in formula (2):

[0074] F = (G - b) / a (2)

[0075] wherein F represents the collected image, G represents the captured image, a represents the contrast coefficient, and b represents the brightness coefficient.

[0076] In step S102, the analog gain refers to a value corresponding to the sensitivity, and the sensitivity refers to the sensitivity of the photosensitive element of the electronic device to light. The smaller the sensitivity, the lower the light sensitivity, the less the light absorption, and the darker the image brightness. Conversely, the greater the sensitivity, the higher the light sensitivity, the greater the light absorption, the brighter the image brightness, and the more the image noise.

[0077] In the embodiment of the present disclosure, the electronic device can store analog gains corresponding to a plurality of sensitivities, which can be stored in the electronic device through a mapping relationship between the sensitivity and the analog gain, and the analog gain corresponding to different sensitivities can be found from the mapping relationship between the sensitivity and the analog gain. In this way, the analog gain can be obtained by finding the mapping relationship in the embodiment of the present disclosure, without the need to calculate the analog gain, and the generation efficiency of the noise image can be improved.

[0078] The dark field frame image refers to an image obtained under the condition that there is no incident ambient light, wherein the condition that there is no incident ambient light can include covering the lens of the camera in the electronic device to make the camera have no incident ambient light, so that the image collected based on the camera is a dark field frame image.

[0079] It should be noted that one photosensitivity can correspond to multiple dark field frame images, and here, the dark field frame image corresponding to the obtained photosensitivity can be that multiple dark field frame images are collected under different photosensitivity, and then one frame of dark field frame image is selected from the multiple dark field frame images corresponding to each photosensitivity as the dark field frame image corresponding to each photosensitivity.

[0080] In step S103, in the process of obtaining the noise image corresponding to the collected image under multiple photosensitivity, the analog gain and the dark field frame image under the same photosensitivity can be associated, and the noise image corresponding to the collected image under the same photosensitivity can be obtained based on the analog gain and the dark field frame image under the same photosensitivity and the collected image, and then the noise image corresponding to the collected image under different photosensitivity can be obtained.

[0081] In the embodiments of the present disclosure, since the image noise is divided into noise related to incident ambient light and noise other than noise related to incident ambient light (i.e. noise unrelated to incident ambient light), and the acquisition of the collected image has noise related to incident ambient light, and the acquisition of the dark field frame image has noise other than noise related to incident ambient light. Therefore, in the process of obtaining the noise image, the electronic device can construct the noise image based on the analog gain, the noise existing in the collected image and the noise existing in the dark field frame image.

[0082] In the embodiments of the present disclosure, the noise image and the collected image can constitute a dark-light image pair in a dark-light scene. The dark-light image pair in the dark-light scene can be used as a training set of a preset denoising network model for training the preset denoising network model.

[0083] In some embodiments, the method further comprises: training a preset denoising network model based on the noise image and the collected image to obtain a trained denoising network model, and the trained denoising network model is used for denoising a captured shooting image.

[0084] In the embodiments of the present disclosure, the preset denoising network model is a model for removing noise, and in the process of training the preset denoising network model, the dark-light image pair in a dark-light scene composed of the noise image and the collected image is input into the preset denoising network model for training, which can make the training set more abundant, and thus the denoising effect of the trained denoising network model is better.

[0085] Exemplarily, the preset denoising network model comprises a Feed-forward denoising convolutional neural networks (DnCNN) model or a Residual Encoder-Decoder Network (REDNet) model, and the present disclosure is not limited thereto.

[0086] In the present disclosure, an acquisition image with an ambient illuminance less than a preset illuminance is obtained; a plurality of analog gains corresponding to a plurality of sensitivities and a plurality of dark field frame images corresponding to a plurality of target sensitivities are obtained; the dark field frame images are obtained under the condition of no incident ambient light; and a plurality of noise images corresponding to the acquisition image under a plurality of sensitivities are obtained based on the analog gains, the dark field frame images and the acquisition image. That is, the present disclosure can obtain a corresponding noise image based on an acquisition image with an ambient illuminance less than a preset illuminance, i.e., an acquisition image in a dark light scene, and thus can solve the problem of difficulty in obtaining a noise image in a dark light scene.

[0087] In an embodiment, obtaining a plurality of noise images corresponding to the acquisition image under a plurality of sensitivities based on the analog gains, the dark field frame images and the acquisition image comprises:

[0088] obtaining a plurality of light noise images related to the incident ambient light under a plurality of sensitivities based on the analog gains and the acquisition image;

[0089] obtaining a plurality of noise images corresponding to the acquisition image under a plurality of sensitivities based on the light noise images and the dark field frame images corresponding to the same sensitivity under a plurality of sensitivities.

[0090] In the present disclosure, in the process of obtaining a light noise image, a plurality of acquisition images can be compared first, and then the light noise image can be obtained based on the comparison results of the plurality of acquisition images and the analog gains; or a Poisson distribution acquisition image can be obtained first, and then the light noise image can be obtained based on the analog gains and the Poisson processing image.

[0091] In an embodiment, obtaining a plurality of light noise images related to the incident ambient light under a plurality of sensitivities based on the analog gains and the acquisition image comprises:

[0092] performing Poisson distribution processing on the ratio between the acquisition image and the analog gain to obtain a Poisson processing image;

[0093] obtaining a plurality of light noise images related to the incident ambient light under a plurality of sensitivities based on the product of the Poisson processing image and the analog gain.

[0094] Exemplarily, the process of obtaining the light noise image is shown in formula (3).

[0095]

[0096] wherein Y1 represents the light noise image, K a represents the analog gain, P(·) represents the Poisson distribution, X * represents the captured image.

[0097] In the embodiments of the present disclosure, after obtaining the light noise images related to the incident ambient light under multiple sensitivities, the embodiments of the present disclosure need to obtain the noise images corresponding to the captured image under multiple sensitivities based on the light noise images and the dark field frame images corresponding to the same sensitivity under multiple sensitivities.

[0098] It should be noted that, in the process of obtaining the light noise image and the dark field frame image corresponding to the same sensitivity, since the electronic device stores the analog gain corresponding to multiple sensitivities and the dark field frame image corresponding to multiple sensitivities, the embodiments of the present disclosure can first obtain the sensitivity of the captured image, and then obtain the light noise image and the dark field frame image corresponding to the same sensitivity based on the light noise image and the dark field frame image corresponding to the same sensitivity of the captured image.

[0099] wherein the electronic device stores the dark field frame image corresponding to multiple sensitivities, and the dark field frame image corresponding to multiple sensitivities can be stored in the electronic device through the mapping relationship between the sensitivity and the dark field frame image. Therefore, after obtaining the same sensitivity as the captured image, the mapping relationship between the sensitivity and the dark field frame image can be directly looked up to obtain the dark field frame image corresponding to the same sensitivity as the captured image. In this way, the embodiments of the present disclosure can further reduce the calculation time of the dark field frame image on the basis of not needing to calculate the analog gain, thereby further improving the generation efficiency of the noise image.

[0100] In an embodiment, based on the light noise image and the dark field frame image corresponding to the same sensitivity under multiple sensitivities, the noise image corresponding to the captured image under multiple sensitivities is obtained, comprising:

[0101] Based on the sum of the light noise image and the dark field frame image corresponding to the same sensitivity under multiple sensitivities, the noise image corresponding to the captured image under multiple sensitivities is obtained.

[0102] In the embodiments of the present disclosure, in the process of obtaining the noise image corresponding to the captured image under multiple sensitivities, the pixel points of the light noise image and the pixel points of the dark field frame image corresponding to the same sensitivity under multiple sensitivities can be added to obtain the noise image.

[0103] In the embodiments of the present disclosure, for collecting an image, the probability of a photon landing at a certain place is the energy of the pixel divided by the total energy. For a pixel point, whether a photon can be received can be described by a binomial distribution. When the number of experiments in the binomial distribution exceeds a certain limit value, the probability of an event occurring in one experiment is small, and the binomial distribution is approximately a Poisson distribution. Therefore, using the Poisson distribution to process the collected image can obtain a more accurate light noise image, and a more accurate noise image corresponding to the collected image under different sensitivities can also be obtained.

[0104] In an embodiment, the image processing method further includes:

[0105] performing mean and variance calculation on all pixel points surrounded by a plurality of gray scale regions under different sensitivities to obtain a plurality of means corresponding to the plurality of gray scale regions under each sensitivity and a plurality of variances corresponding to the plurality of gray scale regions under each sensitivity; wherein the plurality of gray scale regions are obtained by dividing a preset gray scale image into different gray scale regions.

[0106] fitting the plurality of means corresponding to the plurality of gray scale regions and the plurality of variances corresponding to the plurality of gray scale regions to obtain a fitting line under each sensitivity.

[0107] obtaining the analog gain corresponding to the plurality of sensitivities based on the slope of the fitting line under each sensitivity.

[0108] In the embodiments of the present disclosure, the preset gray scale image can include a gray scale card image. The plurality of gray scale regions can be obtained by dividing the gray scale card image into different gray scale regions. The gray scale values of the plurality of gray scale images are all different.

[0109] It should be noted that the mean and variance of all pixel points in each gray scale region are calculated to obtain the mean corresponding to each region and the variance corresponding to each region, and then a plurality of means and a plurality of variances corresponding to a plurality of gray scale regions can be obtained.

[0110] Exemplarily, Figure 2 is a gray scale card image for dividing gray scale regions according to an exemplary embodiment. As Figure 2 shown, the gray scale card image can be divided into 9 gray scale regions (corresponding to the 9 square regions in Figure 2 After obtaining the 9 gray scale regions, 9 sets of means and variances corresponding to the 9 gray scale regions can be obtained.

[0111] In the embodiments of the present disclosure, in the process of obtaining the fitting line under each sensitivity, the mean of the pixel points can be taken as the abscissa, and the variance can be taken as the ordinate to perform linear fitting to obtain the fitting line. Alternatively, the mean of the pixel points can be taken as the ordinate, and the variance can be taken as the abscissa to perform linear fitting to obtain the fitting line. In the embodiments of the present disclosure, the units of the abscissa and the ordinate are both DPI, and the present disclosure does not limit this.

[0112] In some embodiments, fitting the plurality of means corresponding to the plurality of gray scale regions and the plurality of variances corresponding to the plurality of gray scale regions to obtain a fitting line under each sensitivity includes:

[0113] associating the mean and the variance corresponding to the same gray scale region;

[0114] taking the mean corresponding to each gray scale region as the abscissa and the variance corresponding to each gray scale region as the ordinate to perform fitting to obtain a fitting line under each sensitivity.

[0115] In the embodiments of the present disclosure, the mean and the variance corresponding to the same gray scale region are associated. For example, there are 9 gray scale regions, and there are 9 groups of means and variances. Linear fitting of the 9 groups of means and variances can obtain one fitting line corresponding to one sensitivity.

[0116] For example, Figure 3 FIG. 1 is a schematic diagram of a fitting line under sensitivity A according to an example embodiment. As shown in FIG. 1, the mean is taken as the abscissa, and the variance is taken as the ordinate. Nine groups of means and variances obtained under sensitivity A are fitted to obtain a fitting line, and the slope of the fitting line is calculated. In this way, the slope of the fitting line under each sensitivity can be obtained in turn. Figure 3

[0117] In the embodiments of the present disclosure, the slope of the fitting line is the analog gain, the slope of the fitting line under each sensitivity is the analog gain under each sensitivity, and the analog gain corresponding to multiple sensitivities can be constructed based on the analog gain under different sensitivities.

[0118] It should be noted that the analog gain under different sensitivities can be fitted to obtain a gain line in the embodiments of the present disclosure. For example, Figure 4 FIG. 2 is a schematic diagram of fitting of sensitivity and analog gain according to an example embodiment. As shown in FIG. 2, a plurality of sensitivities are taken as the abscissa, and a plurality of analog gains calculated from a plurality of sensitivity conditions are taken as the ordinate. A straight line can be fitted, and the straight line is a gain line. Figure 4

[0119] In order to better understand the one or more embodiments described above, the embodiments of the present disclosure are illustrated as follows:

[0120] ​​Figure 5 is a flow chart of a noise image generation according to an exemplary embodiment. As shown in Figure 5 the image generation process, the image noise can be divided into noise related to incident ambient light and noise unrelated to incident ambient light. Exemplarily, as shown in equation (4):

[0121] Y = K a (I + N p ) + K a N1 + N2 + N q (4)

[0122] wherein Y represents a digital signal stored by the acquired image, N p represents photon incident noise, N q represents quantization noise, N1 and N2 represent all other noises before and after applying analog gain. Wherein K a (I + N p ) represents noise related to incident ambient light, K a N1 + N2 + N q represents noise unrelated to incident ambient light. Due to the quantum nature of light, the number of photons collected by the acquired image obeys a Poisson distribution with parameter I, as shown in equation (5):

[0123]

[0124] wherein, represents a Poisson distribution.

[0125] In an ideal case without noise, the signal value of the acquired image obeys the following equation (6):

[0126] X = K a I (6)

[0127] Substituting equation (6) into equation (5), equation (7) can be obtained:

[0128]

[0129] Poisson distribution processing the ratio between the acquired image and the analog gain is performed on equation (7), and equation (7) is substituted into equation (4), to obtain equation (8):

[0130]

[0131] wherein Y2 represents a generated noise image, Y1 represents a photon noise image, N others represents a dark field frame image.

[0132] In the process of obtaining the light noise image, different gray regions can be obtained based on the first acquisition module, and a fitting line under each photosensitivity can be obtained based on the fitting module; a preset mapping relationship between the photosensitivity and the analog gain can be obtained based on the photosensitivity and analog gain mapping module, and then the light noise image Y1 can be obtained based on the preset mapping relationship between the photosensitivity and the analog gain, the target photosensitivity, and the acquisition image.

[0133] In the process of obtaining the dark field frame image, the dark field frame images under different photosensitivity can be obtained based on the second acquisition module, and a preset mapping relationship between the photosensitivity and the dark field frame image can be obtained based on the photosensitivity and dark field frame image mapping module, and then the dark field frame image N can be obtained based on the preset mapping relationship between the photosensitivity and the dark field frame image and the target photosensitivity. others .

[0134] Finally, the light noise image Y1 and the dark field frame image N others are input into formula (8) to obtain the noise image Y2 corresponding to the acquisition image.

[0135] Figure 6 is a comparison diagram of the noise image (left) and the noise image corresponding to the shooting image (right) according to an example embodiment, as Figure 6 shown, where the left image is the obtained noise image corresponding to the acquisition image, and the right image is the noise image corresponding to the shooting image. It can be seen that the noise image corresponding to the acquisition image obtained by the embodiment of the disclosure and the noise image corresponding to the shooting image are very close. In this way, the noise image corresponding to the acquisition image in the dark light scene can be used to guide the training of the denoising network model.

[0136] The embodiment of the disclosure also proposes an image processing method. Figure 7 is a flowchart of an image processing method according to an example embodiment Figure Two . As Figure 7 shown, the image processing method comprises:

[0137] S201, obtaining a shooting image collected under the condition that the ambient illuminance is less than a preset illuminance;

[0138] S202, performing denoising processing on the shooting image based on a noise image or a denoising network model to obtain a denoised image;

[0139] The noise image is obtained based on the analog gain and the dark field frame image corresponding to a plurality of photosensitivity, and the dark field frame image is obtained under the condition that there is no incident ambient light. The denoising network model is obtained based on the noise image.

[0140] In the embodiments of the present disclosure, the image processing method is applicable to a camera shooting process in a dark light scene where the ambient illumination is less than a preset illumination. For example, when the electronic device is in a dark light scene where the ambient illumination is less than a preset illumination, the electronic device can directly obtain a noise image by denoising a shooting image collected in the dark light scene through the image processing method, and thus a better image in the dark light scene can be obtained in the shooting process.

[0141] In step S201, acquiring the shooting image collected in the case where the ambient illumination is less than the preset illumination means that the electronic device acquires a shooting image in a dark light scene. Here, the shooting image is a non-clear image containing noise, which can be directly obtained by the electronic device through a camera, or can be sent to the electronic device by another electronic device, and the present disclosure does not limit this. In the embodiments of the present disclosure, the shooting image can be a photographing image obtained in a photographing process, or a video image frame obtained in a video recording process.

[0142] In step S202, the noise image can be obtained based on the analog gain corresponding to the sensitivity of the shooting image and the dark field frame image, so that the sensitivity of the noise image is the same as that of the shooting image, and the noise image can be used to denoise the shooting image to obtain a denoised image.

[0143] In the embodiments of the present disclosure, the way of obtaining the denoised image can include two kinds. One is to denoise the shooting image based on the noise image to obtain the denoised image, that is, to guide the shooting image to denoise based on the noise image to obtain the denoised image, and not to use the denoising network model to denoise. The other is to denoise the shooting image based on the denoising network model to obtain the denoised image.

[0144] In one embodiment, the denoising processing of the shooting image to obtain the noise image can include:

[0145] Based on the shooting image and the noise parameter corresponding to the shooting image, a denoised image is obtained.

[0146] It should be noted that obtaining the noise parameter corresponding to the shooting image includes: obtaining the sensitivity of the shooting image; obtaining the analog gain corresponding to the sensitivity of the shooting image; obtaining the variance of the dark field frame corresponding to the sensitivity of the shooting image; and obtaining the noise parameter based on the analog gain corresponding to the sensitivity of the shooting image, the variance of the dark field frame and the shooting image, as shown in formula (9):

[0147]

[0148] wherein, denotes the noise parameter, K a denotes the analog gain, Y denotes the shooting image, and var denotes the variance of the dark field frame.

[0149] In the process of obtaining the denoised image, the denoising is performed on the photographed image based on the noise parameter corresponding to the photographed image, and the denoised image is obtained.

[0150] Exemplarily, the method for denoising the photographed image based on the noise parameter includes a three-dimensional block matching filtering (BM3D), and the embodiments of the present disclosure are not limited thereto.

[0151] In an embodiment, the denoising processing on the photographed image based on the denoising network model includes inputting the photographed image into the denoising network model to obtain the denoised image corresponding to the photographed image. The input of the denoising network model is the photographed image, and the output is the denoised image corresponding to the photographed image.

[0152] In the embodiments of the present disclosure, the electronic device can obtain the denoised image by denoising the photographed image through the noise image or the denoising network model, so that the electronic device can directly obtain the denoised image in the dark light scene of the camera shooting, improve the user's shooting experience, and make the acquisition of the denoised image more flexible. Moreover, the embodiments of the present disclosure can obtain the noise image corresponding to the photosensitivity of the photographed image in the case that the environmental illumination is less than the preset illumination, and can solve the problem of difficulty in obtaining the noise image in the dark light scene.

[0153] The embodiments of the present disclosure further provide an image processing apparatus. Figure 8 Fig. 1 is a schematic diagram of an image processing apparatus according to an exemplary embodiment of the present disclosure. Figure One As shown in Fig. 1, the image processing apparatus 1000 includes: Figure 8

[0154] The first obtaining module 1001 is configured to obtain an acquisition image with an environmental illumination less than a preset illumination.

[0155] The second obtaining module 1002 is configured to obtain a plurality of analog gains corresponding to a plurality of photosensitivities and a plurality of dark field frame images corresponding to the plurality of photosensitivities. The dark field frame image is obtained under the condition of no incident ambient light.

[0156] The noise image generation module 1003 is configured to obtain a plurality of noise images corresponding to the acquisition image under the plurality of photosensitivities based on the analog gain, the dark field frame image, and the acquisition image.

[0157] ​In some embodiments, the noise image generation module is further configured to obtain, based on the analog gain and the acquisition image, a plurality of light noise images related to the incident ambient light under a plurality of ISO sensitivities; and obtain, based on the light noise images and the dark frame image corresponding to a same ISO sensitivity under a plurality of ISO sensitivities, the noise images corresponding to the acquisition image under a plurality of ISO sensitivities.

[0158] In some embodiments, the noise image generation module is further configured to perform Poisson distribution processing on a ratio between the acquisition image and the analog gain to obtain a Poisson processing image; and obtain, based on a product of the Poisson processing image and the analog gain, the light noise images related to the incident ambient light under a plurality of ISO sensitivities.

[0159] In some embodiments, the noise image generation module is further configured to obtain, based on a sum of the light noise images and the dark frame image corresponding to a same ISO sensitivity under a plurality of ISO sensitivities, the noise images corresponding to the acquisition image under a plurality of ISO sensitivities.

[0160] In some embodiments, the apparatus further comprises:

[0161] The calculation module is configured to perform mean value and variance calculation on all pixel points surrounded by a plurality of gray scale regions under different ISO sensitivities to obtain a plurality of mean values corresponding to the plurality of gray scale regions under each ISO sensitivity and a plurality of variances corresponding to the plurality of gray scale regions under each ISO sensitivity; wherein the plurality of gray scale regions are obtained by dividing a preset gray scale image into different gray scale regions.

[0162] The fitting line obtaining module is configured to perform fitting on the plurality of mean values corresponding to the plurality of gray scale regions and the plurality of variances corresponding to the plurality of gray scale regions to obtain a fitting line under each ISO sensitivity.

[0163] The mapping relationship obtaining module is configured to obtain a mapping relationship between the preset ISO sensitivity and the analog gain based on a slope of the fitting line under each ISO sensitivity.

[0164] In some embodiments, the fitting line obtaining module is further configured to associate the mean value and the variance corresponding to a same gray scale region; and take the mean value corresponding to each gray scale region as an abscissa and the variance corresponding to each gray scale region as an ordinate to obtain the fitting line under each ISO sensitivity.

[0165] In some embodiments, the apparatus further comprises:

[0166] The training module is configured to train a preset denoising network model based on the noise image and the acquisition image, to obtain a trained denoising network model, and the trained denoising network model is used for denoising the acquired shooting image.

[0167] The present disclosure also provides an image processing device. Figure 9 An image processing device is shown according to an example embodiment Figure Two As shown in Figure 9 The image processing device 2000 includes:

[0168] The third acquisition module 2001 is configured to acquire a shooting image acquired under the condition that the ambient illumination is less than a preset illumination;

[0169] The denoising processing module 2002 is configured to perform denoising processing on the shooting image based on a noise image or a denoising network model, to obtain a denoised image.

[0170] The noise image is obtained based on an analog gain corresponding to a plurality of sensitivities and a dark field frame image, and the dark field frame image is acquired under the condition that there is no incident ambient light; and the denoising network model is trained based on the noise image.

[0171] The specific manner in which each module in the above-described embodiment device performs operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

[0172] Figure 10 An electronic device block diagram is shown according to an example embodiment. For example, the device 800 can be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0173] Referring to Figure 10 , the electronic device 800 can include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0174] The processing component 802 generally controls the overall operations of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 can include one or more processors 820 to execute instructions to complete the steps of the methods described above, in whole or in part. Moreover, the processing component 802 can include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0175] The memory 804 is configured to store various types of data to support the operations of the electronic device 800. Examples of these data include instructions to operate any applications or methods on the electronic device 800, contact data, phonebook data, messages, pictures, videos, and the like. The memory 804 can be realized by any type of volatile or non-volatile memory devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disc, or optical disc.

[0176] The power component 806 provides power to the various components of the electronic device 800. The power component 806 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.

[0177] The multimedia component 808 includes a screen to provide an output interface between the electronic device 800 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes the touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensors can not only sense a boundary of a touching or sliding action, but also detect duration and pressure related to the touching or sliding action. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. The front and / or rear camera can receive external multimedia data when the device 800 is in an operating mode, such as a shooting mode or a video mode. Each of the front and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0178] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC) that is configured to receive an external audio signal when the electronic device 800 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0179] The I / O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which can be a keypad, a click wheel, buttons, and the like. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0180] The sensor component 814 includes one or more sensors for providing status assessments of various aspects of the electronic device 800. For example, the sensor component 814 can detect an open / closed position of the device 800, relative positioning of components, such as a display and a keypad of the electronic device 800, a change of position of the electronic device 800 or a component of the electronic device 800, presence or absence of user contact with the electronic device 800, orientation or acceleration / deceleration of the electronic device 800, and a temperature change of the electronic device 800. The sensor component 814 can include a proximity sensor configured to detect presence of a nearby object without any physical touch. The sensor component 814 can further include a light sensor, such as a CMOS or CCD image sensor, for use in an imaging application. In some embodiments, the sensor component 814 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0181] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 3G, or 5G, or a combination thereof. In an example embodiment, the communication component 816 receives broadcast signals or broadcast-related information from an external broadcasting management system via a broadcast channel. In an example embodiment, the communication component 816 further includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technology.

[0182] In exemplary embodiments, the electronic device 800 can be implemented with one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic elements, for performing the above-described methods.

[0183] In exemplary embodiments, a non-transitory computer-readable storage medium including instructions, such as the memory 804 including instructions, is also provided, which can be executed by the processor 820 of the electronic device 800 to accomplish the above-described methods. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0184] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform an image processing method, the method comprising: acquiring an acquisition image with an ambient illuminance less than a preset illuminance; acquiring a plurality of analog gains corresponding to a plurality of sensitivities and a plurality of dark field frame images corresponding to the plurality of sensitivities; the dark field frame images being acquired under a condition without incident ambient light; and based on the analog gains, the dark field frame images, and the acquisition image, obtaining a plurality of noise images corresponding to the acquisition image under the plurality of sensitivities.

[0185] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the present disclosure cover any and all variations of the present disclosure including those variations that can be incorporated into the above detailed description and making use of the general principles of the present disclosure. It is intended that the present disclosure include all such as fall within the scope of the appended claims and their equivalents. The specification and examples given herein are to be considered exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

[0186] It is to be understood that the present disclosure is not limited to the precise construction described and shown in the drawings and that various modifications and changes can be made by those skilled in the art without departing from the scope of the present disclosure. The scope of the present disclosure is limited only by the claims that follow.

Claims

1. An image processing method, characterized by, The method comprises: acquiring a collection image in an environment illumination less than a preset illumination; acquiring analog gains corresponding to a plurality of sensitivities and dark field frame images corresponding to the plurality of sensitivities; the dark field frame image is acquired under the condition of no incident ambient light; based on the analog gains, the dark field frame images and the collection image, a plurality of noise images corresponding to the collection image under the plurality of sensitivities are obtained; the method further comprises: based on the analog gains and the collection image, light noise images related to the incident ambient light under the plurality of sensitivities are obtained; based on the light noise images and the dark field frame images corresponding to the same sensitivity under the plurality of sensitivities, the noise images corresponding to the collection image under the plurality of sensitivities are obtained.

2. The method of claim 1, wherein, the method further comprises: based on the analog gains and the collection image, light noise images related to the incident ambient light under the plurality of sensitivities are obtained; based on the light noise images and the dark field frame images corresponding to the same sensitivity under the plurality of sensitivities, the noise images corresponding to the collection image under the plurality of sensitivities are obtained.

3. The method of claim 1, wherein, the method further comprises: based on the noise images and the collection image, a preset denoising network model is trained to obtain a trained denoising network model, and the trained denoising network model is used for denoising a collected shooting image.

4. The method according to any one of claims 1 to 3, characterized in that, ​ ​ ​ ​ 5. The method of claim 4, wherein, ​ ​ ​ 6. The method according to any one of claims 1 to 3, characterized in that, ​ ​ 7. An image processing method characterized by, ​ acquire a captured image collected in a case where ambient illuminance is less than preset illuminance; perform denoising processing on the captured image based on a noise image or a denoising network model to obtain a denoised image; The noise image is obtained based on light noise images and dark field frame images corresponding to a same ISO under multiple ISOs, the light noise images are obtained based on analog gains corresponding to the multiple ISOs and a collected image, the light noise images are related to incident ambient light, and the dark field frame images are collected in a case where there is no incident ambient light; and the denoising network model is trained based on the noise image.

8. An image processing apparatus characterized by comprising: The apparatus comprises: a first acquisition module configured to acquire a collected image in a case where ambient illuminance is less than preset illuminance; a second acquisition module configured to acquire analog gains corresponding to multiple ISOs and dark field frame images corresponding to the multiple ISOs; the dark field frame images are collected in a case where there is no incident ambient light; a noise image generation module configured to obtain noise images corresponding to the collected image under the multiple ISOs based on the analog gains, the dark field frame images, and the collected image; The noise image generation module is further configured to obtain light noise images related to the incident ambient light under the multiple ISOs based on the analog gains and the collected image, and obtain the noise images corresponding to the collected image under the multiple ISOs based on the light noise images and the dark field frame images corresponding to a same ISO under the multiple ISOs.

9. An image processing apparatus characterized by comprising: The apparatus comprises: a third acquisition module configured to acquire a captured image collected in a case where ambient illuminance is less than preset illuminance; a denoising processing module configured to perform denoising processing on the captured image based on a noise image or a denoising network model to obtain a denoised image; The noise image is obtained based on light noise images and dark field frame images corresponding to a same ISO under multiple ISOs, the light noise images are obtained based on analog gains corresponding to the multiple ISOs and a collected image, the light noise images are related to incident ambient light, and the dark field frame images are collected in a case where there is no incident ambient light; and the denoising network model is trained based on the noise image.

10. An electronic device, comprising: The electronic device at least comprises: a processor; a memory for storing processor-executable instructions; The processor is configured to perform the image processing method according to any one of claims 1 to 6, or is configured to perform the image processing method according to claim 7.

11. A non-transitory computer-readable storage medium, comprising: When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device can perform the image processing method according to any one of claims 1 to 6, or is configured to perform the image processing method according to claim 7.

Images