Image processing method and device, electronic equipment and storage medium

Abstract

The invention relates to an image processing method and device, electronic equipment and a storage medium. The image processing method comprises the following steps: acquiring a collected image of which the ambient illumination is smaller than preset illumination; acquiring analog gains corresponding to a plurality of light sensitivities and dark field frame images corresponding to the plurality of light sensitivities; the dark field frame image is acquired under the condition of no incident ambient light; and based on the analog gain, the dark field frame image and the acquired image, obtaining a plurality of noise images corresponding to the acquired image under the light sensitivity. According to the embodiment of the invention, a noise image in a dark light scene can be obtained.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to an image processing method and device, electronic equipment, and storage medium. Background Art

[0002] Digital images are inevitably disturbed by complex noise during the generation, storage and transmission process. With the development of deep learning image denoising methods, deep learning-based image denoising methods have become the mainstream solution to the image denoising problem. However, deep learning-based image denoising methods require the collection of images in different scenes and the noise images corresponding to the collected images as the support of the data set. However, in dark light scenes where the ambient illumination is less than the preset illumination, it is difficult to obtain the noise images corresponding to the collected images under different exposure parameters. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides an image processing method and device, an electronic device, and a storage medium, which can obtain a noise image in a dark light scene.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, comprising at least:

[0005] Acquire a captured image with an ambient illumination less than a preset illumination;

[0006] Acquire analog gains corresponding to a plurality of sensitivities and a plurality of dark field frame images corresponding to the sensitivities; the dark field frame images are acquired under the condition of no incident ambient light;

[0007] Based on the analog gain, the dark field frame image and the collected image, a plurality of noise images corresponding to the collected image at the sensitivities are obtained.

[0008] In some embodiments, obtaining a plurality of noise images corresponding to the collected image at the sensitivity based on the analog gain, the dark field frame image and the collected image includes:

[0009] Based on the analog gain and the collected image, obtaining a plurality of light noise images related to the incident ambient light at the sensitivities;

[0010] Based on the light noise image and the dark field frame image corresponding to the same sensitivity at multiple sensitivities, the noise images corresponding to the collected images at multiple sensitivities are obtained.

[0011] In some embodiments, obtaining a plurality of light noise images related to the incident ambient light at the sensitivities based on the analog gain and the collected image includes:

[0012] Performing Poisson distribution processing on the ratio between the collected image and the analog gain to obtain a Poisson processed image;

[0013] Based on the product of the Poisson processed image and the analog gain, the optical noise image related to the incident ambient light at multiple sensitivities is obtained.

[0014] In some embodiments, the obtaining of the noise images corresponding to the collected images at multiple sensitivities based on the light noise images and the dark field frame images corresponding to the same sensitivities at multiple sensitivities includes:

[0015] Based on the sum of the light noise image and the dark field frame image corresponding to the same sensitivity under multiple sensitivities, the noise images corresponding to the collected images under multiple sensitivities are obtained.

[0016] In some embodiments, the method further comprises:

[0017] Calculating the mean and variance of all pixel points surrounded by the multiple grayscale areas at different sensitivities to obtain multiple means corresponding to the multiple grayscale areas at each sensitivity and multiple variances corresponding to the multiple grayscale areas at each sensitivity; wherein the multiple grayscale areas are obtained by dividing the preset grayscale image into different grayscale areas;

[0018] Fitting a plurality of the means corresponding to a plurality of the grayscale regions and a plurality of the variances corresponding to a plurality of the grayscale regions to obtain a fitting line at each of the sensitivities;

[0019] Based on the slope of the fitting line at each sensitivity, the analog gains corresponding to the multiple sensitivities are obtained.

[0020] In some embodiments, fitting the multiple means corresponding to the multiple grayscale regions and the multiple variances corresponding to the multiple grayscale regions to obtain a fitting line at each sensitivity includes:

[0021] Associating the mean and the variance corresponding to the same grayscale area;

[0022] The mean value corresponding to each grayscale area is used as the horizontal coordinate, and the variance corresponding to each grayscale area is used as the vertical coordinate, and a fitting line under each sensitivity is obtained by fitting.

[0023] In some embodiments, the method further comprises:

[0024] Based on the noisy image and the captured image, a preset denoising network model is trained to obtain a trained denoising network model, and the trained denoising network model is used to denoise the captured image.

[0025] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing method, comprising at least:

[0026] Acquire a captured image when the ambient illumination is less than a preset illumination;

[0027] Based on the noisy image or the denoising network model, denoising the captured image to obtain a denoised image;

[0028] Among them, the noise image is obtained based on analog gains corresponding to multiple sensitivities and dark field frame image processing, and the dark field frame image is collected under the condition of no incident ambient light; the denoising network model is obtained by training based on the noise image.

[0029] According to a third aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, comprising at least:

[0030] A first acquisition module is configured to acquire a captured image with an ambient illumination less than a preset illumination;

[0031] A second acquisition module is configured to acquire analog gains corresponding to a plurality of sensitivities and a plurality of dark field frame images corresponding to the sensitivities; the dark field frame images are acquired under the condition of no incident ambient light;

[0032] The noise image generation module is configured to obtain noise images corresponding to the collected image at multiple sensitivities based on the analog gain, the dark field frame image and the collected image.

[0033] According to a fourth aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, comprising at least:

[0034] A third acquisition module is configured to acquire a captured image captured when the ambient illumination is less than a preset illumination;

[0035] A denoising processing module, configured to perform denoising processing on the captured image based on a noise image or a denoising network model to obtain a denoised image;

[0036] Among them, the noise image is obtained based on analog gains corresponding to multiple sensitivities and dark field frame image processing, and the dark field frame image is collected under the condition of no incident ambient light; the denoising network model is obtained by training based on the noise image.

[0037] According to a fifth aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising at least:

[0038] processor;

[0039] a memory for storing processor-executable instructions;

[0040] Wherein, the processor is configured to execute the image processing method as described in the first aspect above; or, is configured to execute the image processing method as described in the second aspect above.

[0041] According to a sixth aspect of an embodiment of the present disclosure, there is provided a storage medium, including:

[0042] When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image processing method as described in the first aspect above; or, is configured to execute the image processing method as described in the second aspect above.

[0043] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0044] The disclosed embodiments can obtain corresponding noise images of different sensitivities based on captured images with ambient illumination less than a preset illumination, i.e., captured images of dark-light scenes, thereby making it more convenient to obtain noise images of dark-light scenes, and thus solving the problem of difficulty in obtaining noise images in dark-light scenes.

[0045] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0047] Figure 1 is a schematic diagram of a process of an image processing method according to an exemplary embodiment Figure 1 .

[0048] Figure 2 The figure is a grayscale color chart for dividing grayscale areas according to an exemplary embodiment.

[0049] Figure 3 is a schematic diagram of a fitting line at sensitivity A according to an exemplary embodiment.

[0050] Figure 4 is a schematic diagram showing the fitting of sensitivity and analog gain according to an exemplary embodiment.

[0051] Figure 5 is a schematic diagram showing how to generate a noise image according to an exemplary embodiment.

[0052] Figure 6 is a comparison diagram of a noise image (left) and a noise image (right) corresponding to a captured image according to an exemplary embodiment.

[0053] Figure 7 is a schematic diagram of a process of an image processing method according to an exemplary embodiment Figure 2 .

[0054] Figure 8 A schematic diagram of an image processing device according to an exemplary embodiment is shown Figure 1 .

[0055] Fig. 9 A schematic diagram of an image processing device according to an exemplary embodiment is shown Figure 2 .

[0056] Fig.10 The present invention is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0057] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0058] The embodiment of the present disclosure provides an image processing method, which is applied in an electronic device. Figure 1 is a schematic diagram of a process of an image processing method according to an exemplary embodiment Figure 1 .like Figure 1 As shown, the electronic device implements the following steps of the image processing method:

[0059] S101, acquiring a captured image whose ambient illumination is less than a preset illumination;

[0060] S102, obtaining analog gains corresponding to a plurality of sensitivities and a plurality of dark field frame images corresponding to the sensitivities; the dark field frame images are acquired under a condition where there is no incident ambient light;

[0061] S103, obtaining noise images corresponding to the collected image at multiple sensitivities based on the analog gain, the dark field frame image, and the collected image.

[0062] In the disclosed embodiment, the image processing method is applicable to a dark light scene where the ambient illumination is less than the preset illumination. For example, in the process of obtaining the training set of the preset denoising network model, the electronic device can obtain the noise image corresponding to the collected image in the dark light scene through the image processing method, so that the obtained noise image corresponding to the collected image in the dark light scene is added to the training set, which can make the training set richer, and then training the preset denoising network model based on the training set can improve the accuracy of denoising performed by the denoising network model obtained by the training.

[0063] It should be noted that the electronic device may include a terminal device and a server, and the terminal device includes a mobile phone, a desktop computer or a notebook, etc. Among them, when the electronic device is a terminal device, the server can send the collected image, the analog gain and the dark field frame image to the terminal device, so that the terminal device executes the image processing method to generate noise information corresponding to the collected object. Of course, the terminal device can also obtain the collected image, the analog gain and the dark field frame image based on its own database, and the embodiment of the present disclosure is not limited to this.

[0064] In step S101, if the ambient illumination is less than the preset illumination of the captured image, it indicates that the captured image is a dark light image or a low illumination image. Here, the preset illumination can be set according to design requirements, and for example, the preset illumination can be set to be less than or equal to 0.1 lux.

[0065] In the embodiment of the present disclosure, the electronic device may store a captured image database. The acquired captured image may be used as a sample to establish the captured image database. The process of acquiring the captured image may be to select one of the samples from the captured image database as the captured image.

[0066] In constructing the acquired image database, due to the limitations of the imaging device and the environment in dark light scenes and the fact that the brightness of the image generated by the image sensor of the electronic device is proportional to the ambient illumination, the embodiment of the present disclosure can obtain the acquired image by adjusting the exposure parameters.

[0067] In some embodiments, obtaining a captured image when the ambient illumination is less than a preset illumination may include: obtaining a captured image when the ambient illumination is greater than or equal to the preset illumination; and adjusting the captured image based on a preset exposure coefficient to obtain the captured image.

[0068] Here, the captured image when the ambient illumination is greater than or equal to the preset illumination is a clear image, and the embodiment of the present disclosure adjusts the exposure of the clear image to obtain the captured image. In this way, the embodiment of the present disclosure can obtain a clear and dark light captured image.

[0069] The above-mentioned preset exposure coefficient is set by the electronic device according to demand, and the embodiment of the present disclosure does not limit this.

[0070] It should be noted that the brightness of the captured image generated by the image sensor of the above electronic device is proportional to the ambient illumination and is determined according to the grayscale value of the captured image and the ambient illumination, as shown in the following formula (1):

[0071] f=i*r (1)

[0072] Among them, f represents the grayscale value of the captured image, i represents the incident component, which depends on the ambient illumination, and r represents the reflected component, which depends on the characteristics of the imaged object.

[0073] It can be seen from formula (1) that the grayscale value of the captured image is proportional to the ambient illumination. In the disclosed embodiment, adjusting the grayscale value of the captured image corresponds to adjusting the ambient illumination corresponding to the captured image. In other words, by adjusting the grayscale value corresponding to the captured image, a captured image with an ambient illumination less than a preset illumination can be obtained, as shown in formula (2):

[0074] F=(Gb) / a (2)

[0075] Among them, F represents the collected image, G represents the captured image, a represents the contrast coefficient, and b represents the brightness coefficient.

[0076] In step S102, the above-mentioned analog gain refers to a numerical value corresponding to sensitivity. Sensitivity refers to the sensitivity of the photosensitive element of the electronic device to light. The smaller the sensitivity, the lower the sensitivity to light, the less light is absorbed, and the darker the image brightness. On the contrary, the greater the sensitivity, the higher the sensitivity to light, the more light is absorbed, the brighter the image brightness, and the more image noise.

[0077] In the embodiment of the present disclosure, the electronic device may store multiple analog gains corresponding to sensitivities, and the multiple analog gains corresponding to the sensitivities may be stored in the electronic device through a mapping relationship between the sensitivities and the analog gains. The process of obtaining the analog gains corresponding to different sensitivities may be to find the analog gains corresponding to different sensitivities from the mapping relationship between the sensitivities and the analog gains. In this way, the embodiment of the present disclosure can obtain the analog gain by finding the mapping relationship, and does not need to calculate the analog gain, which can improve the efficiency of generating noise images.

[0078] The above-mentioned dark field frame image refers to an image obtained under the condition of no incident ambient light, wherein setting the condition of no incident ambient light may include covering the lens of the camera in the electronic device so that the camera has no incident ambient light, so that the image acquired based on the camera is a dark field frame image.

[0079] It should be noted that one sensitivity can correspond to multiple dark field frame images. Here, the dark field frame image corresponding to the sensitivity obtained can be obtained by first collecting multiple dark field frame images at different sensitivities, and then selecting one of the dark field frame images from the multiple dark field frame images corresponding to each sensitivity as the dark field frame image corresponding to each sensitivity.

[0080] In step S103, in the process of obtaining noise images corresponding to the captured image at multiple sensitivities, the analog gain and dark field frame image at the same sensitivity can be associated, and based on the analog gain and dark field frame image at the same sensitivity, the noise image corresponding to the captured image at the same sensitivity can be obtained, and then the noise images corresponding to the captured image at different sensitivities can be obtained.

[0081] In the disclosed embodiment, since the image noise is divided into noise related to the incident ambient light and noise other than the noise related to the incident ambient light (i.e., noise unrelated to the incident ambient light), and the acquisition of the captured image has noise related to the incident ambient light, and the acquisition of the dark field frame image has noise other than the noise related to the incident ambient light, the electronic device can construct the noise image based on the analog gain, the noise in the captured image, and the noise in the dark field frame image during the acquisition of the noise image.

[0082] In the disclosed embodiment, the noise image and the collected image can form a dark-light image pair in a dark-light scene. The dark-light image pair in a dark-light scene can be used as a training set of a preset denoising network model for training the preset denoising network model.

[0083] In some embodiments, the method further includes: training a preset denoising network model based on the noisy image and the captured image to obtain a trained denoising network model, and the trained denoising network model is used to denoise the captured image.

[0084] In the embodiment of the present disclosure, the preset denoising network model is a model for removing noise. In the process of training the preset denoising network model, the embodiment of the present disclosure inputs a dark-light image pair in a dark-light scene consisting of a noise image and a captured image into the preset denoising network model for training, which can enrich the training set and thus make the denoising effect of the trained denoising network model better.

[0085] Exemplarily, the preset denoising network model includes: a feed-forward denoising convolutional neural networks (DnCNN) model or a residual encoder-decoder network (REDNet) model, which is not limited in the embodiments of the present disclosure.

[0086] In the disclosed embodiment, a captured image with an ambient illumination less than a preset illumination is obtained; analog gains corresponding to multiple sensitivities and multiple dark field frame images corresponding to the target sensitivities are obtained; the dark field frame images are acquired under the condition of no incident ambient light; based on the analog gains, the dark field frame images and the captured images, noise images corresponding to the captured images at multiple sensitivities are obtained. In other words, the disclosed embodiment can obtain corresponding noise images based on captured images with an ambient illumination less than a preset illumination, i.e., captured images of dark light scenes, thereby solving the problem of difficulty in obtaining noise images in dark light scenes.

[0087] In one embodiment, obtaining noise images corresponding to the collected image at multiple sensitivities based on the analog gain, the dark field frame image, and the collected image includes:

[0088] Based on the analog gain and the collected image, obtaining a plurality of light noise images related to the incident ambient light at the sensitivities;

[0089] Based on the light noise image and the dark field frame image corresponding to the same sensitivity at multiple sensitivities, the noise images corresponding to the collected images at multiple sensitivities are obtained.

[0090] In the embodiment of the present disclosure, in the process of obtaining the optical noise image, the optical noise image can be obtained based on the comparison of multiple acquired images and the comparison results of the multiple acquired images and the simulation gain. It is also possible to first acquire images with Poisson distribution and then obtain the optical noise image based on the simulation gain and Poisson processed images.

[0091] In one embodiment, based on the analog gain and the collected image, a plurality of light noise images related to the incident ambient light at the sensitivity are obtained, including:

[0092] Performing Poisson distribution processing on the ratio between the collected image and the analog gain to obtain a Poisson processed image;

[0093] Based on the product of the Poisson processed image and the analog gain, the optical noise image related to the incident ambient light at multiple sensitivities is obtained.

[0094] Exemplarily, the process of obtaining the optical noise image is shown in formula (3).

[0095]

[0096] Among them, Y1 represents the light noise image, K a represents the analog gain, P(·) represents the Poisson distribution, X * Indicates the acquisition of images.

[0097] In an embodiment of the present disclosure, after obtaining light noise images related to incident ambient light at multiple sensitivities, the embodiment of the present disclosure needs to obtain noise images corresponding to the captured images at multiple sensitivities based on the light noise images corresponding to the same sensitivity at multiple sensitivities and the dark field frame images.

[0098] It should be noted that in the process of acquiring the light noise image and dark field frame image corresponding to the same sensitivity, since the electronic device stores multiple analog gains corresponding to the sensitivities and multiple dark field frame images corresponding to the sensitivities, the embodiment of the present disclosure can first acquire the sensitivity of the captured image, and then obtain the light noise image and dark field frame image corresponding to the same sensitivity based on the light noise image and dark field frame image corresponding to the same sensitivity as the captured image.

[0099] Among them, the electronic device stores dark field frame images corresponding to multiple sensitivities, and the dark field frame images corresponding to the multiple sensitivities can be stored in the electronic device through the mapping relationship between the sensitivity and the dark field frame image. Therefore, after obtaining the same sensitivity as the captured image, a table can be directly looked up in the mapping relationship between the sensitivity and the dark field frame image to obtain the dark field frame image corresponding to the same sensitivity as the captured image. In this way, the embodiment of the present disclosure can also reduce the calculation time of the dark field frame image without calculating the analog gain, thereby further improving the generation efficiency of the noise image.

[0100] In one embodiment, based on the light noise image and the dark field frame image corresponding to the same sensitivity at multiple sensitivities, obtaining the noise image corresponding to the collected image at multiple sensitivities includes:

[0101] Based on the sum of the light noise image and the dark field frame image corresponding to the same sensitivity under multiple sensitivities, the noise images corresponding to the collected images under multiple sensitivities are obtained.

[0102] In the disclosed embodiment, in the process of obtaining the noise image corresponding to the captured image at multiple sensitivities, the pixel points of the light noise image corresponding to the same sensitivity at multiple sensitivities and the pixel points of the dark field frame image can be added to obtain the noise image.

[0103] In the disclosed embodiment, for a captured image, the probability of a photon falling at a certain place is the energy of this pixel divided by the total energy. For a pixel, whether a photon can be received can be described by a binomial distribution. When the number of experiments in the binomial distribution exceeds a certain limit value, the probability of an event occurring in one experiment is small, and the binomial distribution is approximately a Poisson distribution. Therefore, using the Poisson distribution to process the captured image can obtain a more accurate light noise image, and thus can also obtain a more accurate noise image corresponding to the captured image at different sensitivities.

[0104] In one embodiment, the image processing method further includes:

[0105] Calculating the mean and variance of all pixel points surrounded by the multiple grayscale areas at different sensitivities to obtain multiple means corresponding to the multiple grayscale areas at each sensitivity and multiple variances corresponding to the multiple grayscale areas at each sensitivity; wherein the multiple grayscale areas are obtained by dividing the preset grayscale image into different grayscale areas;

[0106] Fitting a plurality of the means corresponding to a plurality of the grayscale regions and a plurality of the variances corresponding to a plurality of the grayscale regions to obtain a fitting line at each of the sensitivities;

[0107] Based on the slope of the fitting line at each sensitivity, the analog gains corresponding to the multiple sensitivities are obtained.

[0108] In the embodiment of the present disclosure, the preset grayscale image may include a grayscale color card diagram. In the embodiment of the present disclosure, different grayscale areas may be divided in the grayscale color card diagram to obtain multiple grayscale areas, and the grayscale values ​​of the multiple grayscale images are all different.

[0109] It should be noted that the variance and mean are calculated for all pixels in each grayscale area to obtain the mean corresponding to each area and the variance corresponding to each area, and then corresponding to multiple grayscale areas, multiple means and multiple variances can be obtained.

[0110] For example, Figure 2 is a grayscale color card diagram for dividing grayscale areas according to an exemplary embodiment. Figure 2 As shown, the grayscale color card can be divided into 9 grayscale areas (corresponding to Figure 2 ), after obtaining 9 grayscale areas, 9 groups of means and variances corresponding to the 9 grayscale areas can be obtained.

[0111] In the embodiment of the present disclosure, in the process of obtaining the fitting line at each sensitivity, the pixel mean can be used as the horizontal coordinate and the variance can be used as the vertical coordinate to perform linear fitting to obtain the fitting line. The pixel mean can also be used as the vertical coordinate and the variance can be used as the horizontal coordinate to perform linear fitting to obtain the fitting line, wherein the units of the horizontal coordinate and the vertical coordinate are both DPI, and the embodiment of the present disclosure does not limit this.

[0112] In some embodiments, fitting the multiple means corresponding to the multiple grayscale regions and the multiple variances corresponding to the multiple grayscale regions to obtain a fitting line at each sensitivity includes:

[0113] Associating the mean and the variance corresponding to the same grayscale area;

[0114] The mean value corresponding to each grayscale area is used as the horizontal coordinate, and the variance corresponding to each grayscale area is used as the vertical coordinate, and a fitting line under each sensitivity is obtained by fitting.

[0115] In the embodiment of the present disclosure, the mean and variance corresponding to the same grayscale region are associated. For example, there are 9 grayscale regions, corresponding to 9 groups of means and variances, and a fitting line corresponding to a sensitivity can be obtained by linear fitting the 9 groups of means and variances.

[0116] For example, Figure 3 is a schematic diagram of a fitting line at a sensitivity A according to an exemplary embodiment. Figure 3 As shown, with the mean as the horizontal coordinate and the variance as the vertical coordinate, the 9 groups of means and variances obtained at sensitivity A are fitted to obtain a fitting line, and the slope of the fitting line is calculated. In this way, the slope of the fitting line at each sensitivity can be obtained in turn.

[0117] In the disclosed embodiment, the slope of the fitting line is the analog gain, and the slope of the fitting line at each sensitivity is the analog gain at each sensitivity. Then, based on the analog gains at different sensitivities, analog gains corresponding to multiple sensitivities can be constructed.

[0118] It should be noted that the embodiment of the present disclosure can fit the analog gain under different sensitivities to obtain a gain line. For example, Figure 4 FIG. 1 is a schematic diagram showing the fitting of sensitivity and analog gain according to an exemplary embodiment. Figure 4 As shown, with multiple sensitivities as the horizontal coordinates and multiple analog gains calculated under the multiple sensitivity conditions as the vertical coordinates, a straight line can be fitted, and the straight line is the gain line.

[0119] In order to better understand one or more of the above embodiments, the embodiments of the present disclosure are exemplified as follows:

[0120] Figure 5 FIG. 1 is a flowchart of generating a noise image according to an exemplary embodiment. Figure 5 As shown, during the image generation process, the image noise can be divided into noise related to the incident ambient light and noise unrelated to the incident ambient light. For example, as shown in formula (4):

[0121] Y=K a (I+N p )+K a N1+N2+N q (4)

[0122] Among them, Y represents the digital signal stored in the acquired image, N p represents the photon incident noise, N q K represents the quantization noise, and N1 and N2 represent all other noise before and after the analog gain is applied. a (I+N p ) represents the noise associated with the incident ambient light, K a N1+N2+N q Represents noise that is unrelated to the incident ambient light. Due to the quantum nature of light, the number of photons collected by the acquired image follows a Poisson distribution with parameter I, as shown in formula (5):

[0123]

[0124] in, represents a Poisson distribution.

[0125] In the ideal case without noise, the signal value of the acquired image obeys the following formula (6):

[0126] X=K a I (6)

[0127] Substituting formula (6) into formula (5), we can get formula (7):

[0128]

[0129] Formula (7) is used to process the ratio between the acquired image and the simulated gain using Poisson distribution, and formula (7) is substituted into formula (4), and formula (8) is obtained:

[0130]

[0131] Among them, Y2 represents the generated noise image, Y1 represents the light noise image, and N others Represents a dark field frame image.

[0132] In the process of acquiring the light noise image, different grayscale areas can be first acquired based on the first acquisition module, and then the fitting line under each sensitivity can be acquired based on the fitting module; then the mapping relationship between the preset sensitivity and the analog gain can be obtained based on the sensitivity and analog gain mapping module, and then the light noise image Y1 can be obtained based on the mapping relationship between the preset sensitivity and the analog gain, the target sensitivity and the acquired image.

[0133] In the process of acquiring the dark field frame image, the dark field frame images under different sensitivities can be first acquired based on the second acquisition module, and then the mapping relationship between the preset sensitivity and the dark field frame image can be obtained based on the sensitivity and dark field frame image mapping module, and then the dark field frame image N can be obtained based on the mapping relationship between the preset sensitivity and the dark field frame image and the target sensitivity. others .

[0134] Finally, the light noise image Y1 and the dark field frame image N others Input into formula (8) to obtain the noise image Y2 corresponding to the collected image.

[0135] Figure 6 is a comparison diagram of a noise image (left) and a noise image (right) corresponding to a captured image according to an exemplary embodiment. Figure 6 As shown in the figure, the left figure is the noise image corresponding to the acquired image, and the right figure is the noise image corresponding to the captured image. It can be seen that the noise image corresponding to the acquired image obtained by the embodiment of the present disclosure is very close to the noise image corresponding to the captured image. In this way, the obtained noise image corresponding to the acquired image in a dark light scene can be used to guide the training of the denoising network model.

[0136] The disclosed embodiment also provides an image processing method. Figure 7 is a flowchart of an image processing method according to an exemplary embodiment. Figure 2 .like Figure 7 As shown, the image processing method includes:

[0137] S201, acquiring a captured image captured when the ambient illumination is less than a preset illumination;

[0138] S202, performing denoising processing on the captured image based on the noise image or the denoising network model to obtain a denoised image;

[0139] Among them, the noise image is obtained by processing the analog gain corresponding to multiple sensitivities and the dark field frame image, and the dark field frame image is collected under the condition of no incident ambient light; the denoising network model is obtained by training based on the noise image.

[0140] In the disclosed embodiment, the image processing method is applicable to the camera shooting process in a dark light scene where the ambient illumination is less than the preset illumination. For example, when the electronic device is in a dark light scene where the ambient illumination is less than the preset illumination, the electronic device can directly denoise the captured image in the dark light scene to obtain a noise image through the image processing method, thereby obtaining a better image in the dark light scene during the shooting process.

[0141] In step S201, obtaining a captured image captured when the ambient illumination is less than a preset illumination means that the electronic device obtains a captured image in a dark light scene. Here, the captured image is a non-clear image containing noise, and the captured image can be obtained by directly capturing the electronic device through a camera, or can be sent to the electronic device by other electronic devices, and the embodiments of the present disclosure do not limit this. In the embodiments of the present disclosure, the captured image can be a photographic image obtained during a photo shooting process, or can be a video image frame obtained during a video recording process.

[0142] In step S202, the noise image can be obtained based on the analog gain corresponding to the sensitivity of the captured image and the dark field frame image. Therefore, the sensitivity of the noise image is the same as the sensitivity of the captured image, and the captured image can be denoised through the noise image to obtain a denoised image.

[0143] In the embodiments of the present disclosure, there are two ways to obtain a denoised image. One is to perform denoising on the captured image based on the noise image to obtain the denoised image, that is, to guide the denoising of the captured image based on the noise image to obtain the denoised image, instead of using a denoising network model for denoising; the other is to perform denoising on the captured image based on the denoising network model to obtain the denoised image.

[0144] In one embodiment, performing denoising on the captured image to obtain a noisy image may include:

[0145] A denoised image is obtained based on the captured image and noise parameters corresponding to the captured image.

[0146] It should be noted that obtaining the noise parameter corresponding to the captured image includes: obtaining the sensitivity of the captured image; obtaining the analog gain corresponding to the sensitivity of the captured image; obtaining the variance of the dark field frame corresponding to the sensitivity of the captured image; and obtaining the noise parameter based on the analog gain corresponding to the sensitivity of the captured image, the variance of the dark field frame and the captured image, as shown in formula (9):

[0147]

[0148] in, represents the noise parameter, K a represents the analog gain, Y represents the captured image, and var represents the dark field frame variance.

[0149] In the process of acquiring the denoised image, the captured image is denoised based on the noise parameters corresponding to the captured image to obtain the denoised image.

[0150] Exemplarily, the method for denoising the captured image based on noise parameters includes three-dimensional block matching filtering (Block-matching and 3D filtering, BM3D), which is not limited in the embodiments of the present disclosure.

[0151] In one embodiment, based on the denoising network model, denoising the captured image includes inputting the captured image into the denoising network model to obtain a denoised image corresponding to the captured image. The denoising network model has the captured image as input and the denoised image as output corresponding to the captured image.

[0152] In the disclosed embodiment, the electronic device can perform denoising processing on the captured image through the noise image or the denoising network model to obtain the denoised image. In this way, not only can the electronic device directly obtain the denoised image in the dark light scene captured by the camera, improving the user's shooting experience, but also making the acquisition of the denoised image more flexible. In addition, the disclosed embodiment can obtain the noise image corresponding to the sensitivity of the captured image when the ambient illumination is less than the preset illumination, which can solve the problem of difficulty in obtaining the noise image in the dark light scene.

[0153] The disclosed embodiment also provides an image processing device. Figure 8 A schematic diagram of an image processing device according to an exemplary embodiment is shown Figure 1 .like Figure 8 As shown, the image processing device 1000 includes:

[0154] The first acquisition module 1001 is configured to acquire a captured image whose ambient illumination is less than a preset illumination;

[0155] The second acquisition module 1002 is configured to acquire analog gains corresponding to a plurality of sensitivities and a plurality of dark field frame images corresponding to the sensitivities; the dark field frame images are acquired under the condition of no incident ambient light;

[0156] The noise image generating module 1003 is configured to obtain noise images corresponding to the collected image at multiple sensitivities based on the analog gain, the dark field frame image and the collected image.

[0157] In some embodiments, the noise image generation module is further configured to obtain, based on the analog gain and the acquired image, a plurality of light noise images related to the incident ambient light at the sensitivities; and to obtain, based on the light noise image and the dark field frame image corresponding to the same sensitivity at the plurality of sensitivities, a plurality of noise images corresponding to the acquired image at the sensitivities.

[0158] In some embodiments, the noise image generation module is further configured to perform Poisson distribution processing on the ratio between the acquired image and the analog gain to obtain a Poisson processed image; and based on the product of the Poisson processed image and the analog gain, obtain the light noise image related to the incident ambient light at multiple sensitivities.

[0159] In some embodiments, the noise image generation module is further configured to obtain the noise images corresponding to the acquired images at multiple sensitivities based on the sum of the light noise images corresponding to the same sensitivity at multiple sensitivities and the dark field frame images.

[0160] In some embodiments, the apparatus further comprises:

[0161] A calculation module, configured to calculate the mean and variance of all pixel points surrounded by multiple grayscale areas at different sensitivities, to obtain multiple means corresponding to the multiple grayscale areas at each sensitivity and multiple variances corresponding to the multiple grayscale areas at each sensitivity; wherein the multiple grayscale areas are obtained by dividing the preset grayscale image into different grayscale areas;

[0162] A fitting line obtaining module, configured to fit a plurality of the means corresponding to the plurality of the grayscale regions and a plurality of the variances corresponding to the plurality of the grayscale regions, to obtain a fitting line at each of the sensitivity;

[0163] The mapping relationship obtaining module is configured to obtain the mapping relationship between the preset sensitivity and the analog gain based on the slope of the fitting line at each sensitivity.

[0164] In some embodiments, the fitting line obtaining module is also configured to associate the mean and the variance corresponding to the same grayscale area; with the mean corresponding to each grayscale area as the horizontal coordinate and the variance corresponding to each grayscale area as the vertical coordinate, a fitting line is obtained at each sensitivity.

[0165] In some embodiments, the apparatus further comprises:

[0166] The training module is configured to train a preset denoising network model based on the noisy image and the captured image to obtain a trained denoising network model, and the trained denoising network model is used to denoise the captured image.

[0167] The disclosed embodiment also provides an image processing device. Fig. 9 A schematic diagram of an image processing device according to an exemplary embodiment is shown Figure 2 .like Fig. 9 As shown, the image processing device 2000 includes:

[0168] The third acquisition module 2001 is configured to acquire a captured image captured when the ambient illumination is less than a preset illumination;

[0169] A denoising processing module 2002 is configured to perform denoising processing on the captured image based on a noise image or a denoising network model to obtain a denoised image;

[0170] Among them, the noise image is obtained based on analog gains corresponding to multiple sensitivities and dark field frame image processing, and the dark field frame image is collected under the condition of no incident ambient light; the denoising network model is obtained by training based on the noise image.

[0171] The specific manner in which each module in the device of the above embodiment performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0172] Fig.10 800 is a block diagram of an electronic device according to an exemplary embodiment. For example, the device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0173] Reference Fig.10 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0174] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0175] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0176] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.

[0177] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0178] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0179] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0180] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0181] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 3G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0182] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0183] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by a processor 820 of an electronic device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0184] A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform an image processing method, the method comprising: obtaining a captured image with an ambient illumination less than a preset illumination; obtaining analog gains corresponding to multiple sensitivities and multiple dark field frame images corresponding to the sensitivities; the dark field frame images are captured under conditions without incident ambient light; based on the analog gains, the dark field frame images and the captured images, obtaining multiple noise images corresponding to the captured images at the sensitivities.

[0185] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0186] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Acquire a captured image with an ambient illumination less than a preset illumination; Acquire analog gains corresponding to a plurality of sensitivities and a plurality of dark field frame images corresponding to the sensitivities; The dark field frame image is acquired under the condition of no incident ambient light; Based on the analog gain, the dark field frame image and the collected image, a plurality of noise images corresponding to the collected image at the sensitivities are obtained.

2. The method according to claim 1, characterized in that: The step of obtaining a plurality of noise images corresponding to the collected images at the sensitivities based on the analog gain, the dark field frame image and the collected images comprises: Based on the analog gain and the collected image, obtaining a plurality of light noise images related to the incident ambient light at the sensitivities; Based on the light noise image and the dark field frame image corresponding to the same sensitivity at multiple sensitivities, the noise images corresponding to the collected images at multiple sensitivities are obtained.

3. The method according to claim 2, characterized in that The step of obtaining a plurality of light noise images related to the incident ambient light at the sensitivity based on the analog gain and the collected image includes: Performing Poisson distribution processing on the ratio between the collected image and the analog gain to obtain a Poisson processed image; Based on the product of the Poisson processed image and the analog gain, the optical noise image related to the incident ambient light at multiple sensitivities is obtained.

4. The method according to claim 2, characterized in that: The obtaining the noise images corresponding to the collected images at the multiple sensitivities based on the light noise images and the dark field frame images corresponding to the same sensitivity at the multiple sensitivities comprises: Based on the sum of the light noise image and the dark field frame image corresponding to the same sensitivity under multiple sensitivities, the noise images corresponding to the collected images under multiple sensitivities are obtained.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Calculating the mean and variance of all pixel points surrounded by the multiple grayscale areas at different sensitivities to obtain multiple means corresponding to the multiple grayscale areas at each sensitivity and multiple variances corresponding to the multiple grayscale areas at each sensitivity; wherein the multiple grayscale areas are obtained by dividing the preset grayscale image into different grayscale areas; Fitting a plurality of the means corresponding to a plurality of the grayscale regions and a plurality of the variances corresponding to a plurality of the grayscale regions to obtain a fitting line at each of the sensitivities; Based on the slope of the fitting line at each sensitivity, the analog gains corresponding to the multiple sensitivities are obtained.

6. The method according to claim 5, characterized in that The fitting of the plurality of means corresponding to the plurality of grayscale regions and the plurality of variances corresponding to the plurality of grayscale regions to obtain a fitting line at each sensitivity includes: Associating the mean and the variance corresponding to the same grayscale area; The mean value corresponding to each grayscale area is used as the horizontal coordinate, and the variance corresponding to each grayscale area is used as the vertical coordinate, and a fitting line under each sensitivity is obtained by fitting.

7. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Based on the noisy image and the captured image, a preset denoising network model is trained to obtain a trained denoising network model, and the trained denoising network model is used to denoise the captured image.

8. An image processing method, characterized in that: include: Acquire a captured image when the ambient illumination is less than a preset illumination; Based on the noisy image or the denoising network model, denoising the captured image to obtain a denoised image; Among them, the noise image is obtained based on analog gains corresponding to multiple sensitivities and dark field frame image processing, and the dark field frame image is collected under the condition of no incident ambient light; the denoising network model is obtained by training based on the noise image.

9. An image processing device, characterized in that: The device comprises: A first acquisition module is configured to acquire a captured image with an ambient illumination less than a preset illumination; A second acquisition module is configured to acquire analog gains corresponding to a plurality of sensitivities and a plurality of dark field frame images corresponding to the sensitivities; the dark field frame images are acquired under the condition of no incident ambient light; The noise image generation module is configured to obtain noise images corresponding to the collected image at multiple sensitivities based on the analog gain, the dark field frame image and the collected image.

10. An image processing device, characterized in that: The device comprises: A third acquisition module is configured to acquire a captured image captured when the ambient illumination is less than a preset illumination; A denoising processing module, configured to perform denoising processing on the captured image based on a noise image or a denoising network model to obtain a denoised image; Among them, the noise image is obtained based on analog gains corresponding to multiple sensitivities and dark field frame image processing, and the dark field frame image is collected under the condition of no incident ambient light; the denoising network model is obtained by training based on the noise image.

11. An electronic device, characterized in that: The electronic device at least comprises: processor; a memory for storing processor-executable instructions; The processor is configured to execute the image processing method according to any one of claims 1 to 7; or, is configured to execute the image processing method according to claim 8.

12. A non-transitory computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image processing method as described in any one of claims 1 to 7; or is configured to execute the image processing method as described in claim 8.

Images