Method, medium and electronic device for obtaining image noise

Abstract

The present application relates to the field of image processing technology and discloses a method, medium, and electronic device for obtaining image noise. The method includes: a processing device determining multiple image noise types in a sensor device connected to the processing device; the processing device classifying the multiple image noise types based on the relationship between the multiple image noise types and grayscale values; and determining a mapping relationship between the total image noise of the sensor device and the image grayscale value based on the classified image noise types and at least one sample image data obtained by the sensor device. Thus, when denoising an image based on the mapping relationship obtained by the above scheme, since the obtained total image noise incorporates multiple types of image noise in the sensor, various types of noise can be accurately removed during image denoising. Furthermore, compared to the prior art method of sequentially denoising an image using multiple image noise functions, this method can shorten the denoising time and reduce the computational complexity of the electronic device capable of performing image denoising.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, medium, and electronic device for obtaining image noise. Background Art

[0002] In fields such as security monitoring, medical image diagnosis, microscopic imaging, autonomous driving, and astronomical observation, images are generally collected through image acquisition devices. The image acquisition devices can be devices such as cameras, and the image acquisition devices generally include image sensors.

[0003] During the image acquisition process, various noises will be introduced due to the influence of the image sensor material properties, working environment, electronic components and circuit structure, such as shot noise, readout noise, transient noise, thermal noise caused by resistance, photon noise, dark current noise and light response non-uniformity noise. As a result, the image captured by the image acquisition device will inevitably deviate from the actual image to a certain extent. This will affect the image quality to a certain extent and cause the image to be distorted to a certain extent.

[0004] To remove noise components from images and improve image quality, existing image denoising methods include various filtering methods such as mean filtering, median filtering, and Gaussian low-pass filtering. These filters limit the noise-related components in the image to achieve the desired effect, but this approach can only remove a portion of the noise. Summary of the Invention

[0005] The embodiments of the present application provide a method for acquiring image noise, a medium, and an electronic device.

[0006] In a first aspect, embodiments of the present application provide a method for obtaining image noise, which is applied to an electronic system. The electronic system includes a sensing device and a processing device. The method includes:

[0007] The processing device determines a plurality of image noise types in the sensing device connected to the processing device;

[0008] The processing device classifies the multiple image noise types according to the relationship between the multiple image noise types and grayscale values;

[0009] Based on the classified types of image noise and at least one sample image data acquired by the sensing device, a mapping relationship between the total image noise of the sensing device and the image grayscale value is determined.

[0010] It is understood that the sensing device can be a camera, and the processing device can be a computer, a server, a mobile phone, etc., but is not limited thereto. "Acquiring image noise" in the embodiment of the present application can be understood as determining the mapping relationship between the total image noise of the sensing device and the image grayscale value.

[0011] In an embodiment of the present application, when denoising an image based on the mapping relationship between the total image noise and the image grayscale value obtained by the above-mentioned scheme, since the total image noise obtained by the above-mentioned scheme integrates multiple types of image noise in the sensor, it is possible to accurately remove various types of noise when denoising the image. Moreover, compared with the prior art method of denoising an image sequentially using multiple image noise functions, the denoising time can be shortened and the computational load of the electronic device capable of denoising the image can be reduced. In the image noise acquisition stage, compared with the method of inputting multiple sample images into multiple image noise functions to obtain each type of image noise in turn, the present application can determine the mapping relationship between the total image noise of the sensor device and the image grayscale value based solely on the sample image data, thereby reducing the computational load of the electronic device.

[0012] In a possible implementation of the first aspect, the processing device classifies the multiple image noise types according to the relationship between the multiple image noise types and grayscale values, including:

[0013] The processing device classifies the type of image noise that varies with grayscale value into a first type of image noise;

[0014] The processing device classifies the image noise type that does not change with the grayscale value into the second type of image noise.

[0015] It is understood that the relationship between image noise type and grayscale value can be the characteristic information described below. Transient noise is the first type of image noise. Shot noise, thermal noise, and readout noise are the second type of image noise.

[0016] In a possible implementation of the first aspect, determining a mapping relationship between total image noise and image grayscale values ​​of the sensing device based on the classified image noise types and at least one sample image data acquired by the sensing device includes:

[0017] Fusing the relationship between image noise and grayscale value in the first type of image noise to obtain unknown terms in the mapping function between total image noise and image grayscale value;

[0018] Fusing the relationship between image noise and grayscale value in the second type of image noise to obtain a constant term in a mapping function between total image noise and image grayscale value;

[0019] Based on at least one sample image data acquired by the sensing device, the unknown term and the constant term, a first function parameter to be determined and a constant term in the unknown term are determined to determine a mapping function between the total image noise and the image grayscale value of the sensing device as a mapping relationship between the total image noise and the image grayscale value.

[0020] In a possible implementation of the first aspect above, the sensing device is a device including an image sensor.

[0021] In a possible implementation of the first aspect, the multiple image noise types include shot noise, thermal noise, readout noise, and transient noise.

[0022] In a possible implementation of the first aspect, the sample image data includes multiple frames of sample images captured from the same shooting angle of view for the same scene.

[0023] In a possible implementation of the first aspect, determining, based on at least one sample image data acquired by the sensing device, the unknown term, and the constant term, a first to-be-determined function parameter and a constant term in the unknown term, so as to determine a mapping function between the total image noise and the image grayscale value of the sensing device as the mapping relationship between the total image noise and the image grayscale value, includes:

[0024] For each sample image data, the following processing is performed:

[0025] Acquire a pixel grayscale value set from the multiple frames of sample images, wherein the pixel grayscale value set includes a plurality of pixel grayscale value subsets, each pixel grayscale value subset consisting of pixel grayscale values ​​at the same pixel position in the multiple frames of sample images;

[0026] Performing average processing on each subset of pixel grayscale values ​​in the pixel grayscale value set to obtain a pixel grayscale average value set;

[0027] Performing sample standard deviation processing on each subset of pixel grayscale values ​​in the pixel grayscale value set to obtain a pixel grayscale sample standard deviation set;

[0028] Based on at least one set of pixel grayscale average values ​​and at least one set of pixel grayscale sample standard deviations corresponding to at least one sample image data, the first function parameter to be determined and the constant term in the unknown number term are determined to determine the mapping function between the total image noise and the image grayscale value of the sensing device, as the mapping relationship between the total image noise and the image grayscale value.

[0029] In a possible implementation of the first aspect above, the sample image data includes sample image data of a first grayscale range and sample image data of a second grayscale range, and the amount of sample image data of the first grayscale range is different from the amount of sample image data of the second grayscale range.

[0030] In a second aspect, an embodiment of the present application provides a readable medium having instructions stored thereon. When the instructions are executed on an electronic device, the electronic device executes the method for obtaining image noise as described in any one of the first aspects.

[0031] In a second aspect, an embodiment of the present application provides an electronic device, characterized by including:

[0032] a memory for storing instructions to be executed by one or more processors of the electronic device, and

[0033] The processor is one of the processors of the electronic device, and is used to execute the method for obtaining image noise as described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 According to some embodiments of the present application, a schematic diagram of the structure and principle of an image sensor in a camera 100 is shown;

[0035] Figure 2 According to some embodiments of the present application, a schematic diagram of a curve for calculating multiple image noise superposition values ​​obtained by computer 200 using MATLAB software based on sample image fitting is shown;

[0036] Figure 3 According to some embodiments of the present application, a schematic diagram of an application scenario of image denoising is shown;

[0037] Figure 4 According to some embodiments of the present application, a schematic diagram of denoising an image B to be denoised is exemplarily shown;

[0038] Figure 5 According to some embodiments of the present application, a flowchart of an image denoising method is shown;

[0039] Figure 6 A schematic diagram of an application scenario of determining the image noise function in step 602 is shown;

[0040] Figure 7 According to some embodiments of the present application, corresponding to Figure 1 and 6 , shows a flow chart of a method for determining an image noise function;

[0041] Figure 8 According to some embodiments of the present application, a schematic diagram of multiple frames of sample images is shown;

[0042] Figure 9 According to some embodiments of the present application, a structural diagram of a camera 100 is shown;

[0043] Figure 10 According to some embodiments of the present application, a schematic structural diagram of the ISP 102 in the camera 100 is shown;

[0044] Figure 11 A schematic diagram of a process of processing image data by a general functional module is shown. DETAILED DESCRIPTION

[0045] Illustrative embodiments of the present application include, but are not limited to, an image denoising method, a medium, and an electronic device.

[0046] The following first introduces the relevant terms and concepts involved in the embodiments of this application.

[0047] (a) Grayscale value. Grayscale value indicates the color depth in a gray image. It is understood that the grayscale value range depends on the number of bits in the imaging system. For example, a 12-bit imaging system can generally provide grayscale values ​​ranging from 0 to 4095, where 4095 represents the grayscale level of white and 0 represents the grayscale level of black. An 8-bit imaging system can generally provide grayscale values ​​ranging from 0 to 255, where 255 represents the grayscale level of white and 0 represents the grayscale level of black.

[0048] (b) Image noise. Image noise is the deviation between the image information of the image captured by the image acquisition device and the image information of the actual image. It will be appreciated that in the embodiments of the present application, image noise is used to calibrate the image to be denoised. Therefore, image noise can also be referred to as a grayscale calibration value or a grayscale denoising value. In the embodiments of the present application, the image information may include grayscale values.

[0049] Image noise can include shot noise, readout noise, thermal noise, and transient noise, which will be introduced below.

[0050] (c) Shot noise: Shot noise is the fluctuation of the image's actual grayscale value caused by the non-uniformity of electron emission in active devices (such as vacuum tubes) within the image sensor. It is also called shot noise. Shot noise is related to the incident photons and dark current of the image sensor and follows a Poisson distribution. Shot noise can be calculated using the following formula:

[0051]

[0052] Where N represents the electron energy of dark current, P N Indicates the image noise value caused by shot noise.

[0053] (d) Readout noise: Readout noise is the fluctuation of the actual grayscale value of the image caused by the readout circuit in the image sensor.

[0054] (e) Thermal noise. Thermal noise is the fluctuation of the actual grayscale value of the image caused by the thermal motion of electrons in the resistor of the image sensor. Thermal noise can be calculated using the following formula:

[0055] Sv (f) = 4kTR(V 2 / Hz)

[0056] Among them, S v (f) represents thermal noise, k is the Boltzmann constant of the image sensor, T is the absolute temperature of the image sensor, and R is the resistance of the image sensor.

[0057] (f) Transient noise. Transient noise can be calculated using the following Gaussian distribution formula:

[0058]

[0059] Where x is the pixel grayscale value, m is the average pixel grayscale value, σ is the standard deviation of the grayscale value samples, σ represents transient noise, and p(x) represents transient noise.

[0060] (g) Image distortion: Image distortion refers to the deviation between the captured image and the actual image.

[0061] In the embodiment of the present application, during the image acquisition process, various noises may be introduced due to the influence of the material properties of the image sensor, the working environment, the electronic components and the circuit structure, so that the acquired image will inevitably deviate from the actual image to a certain extent. This affects the image quality to a certain extent and causes a certain degree of image distortion.

[0062] For example, Figure 1 According to some embodiments of the present application, a schematic diagram of the structure and principle of an image sensor in a camera 100 is shown. Figure 1 As shown, the image sensor includes a photosensitive element 201 , a floating diffusion amplifier (FDA) 202 , and an analog-to-digital converter (ADC) 203 .

[0063] The photosensitive element 201 is used to convert incident photons formed by light reflected from the surface of the photographed scene into electrons. The electrons are then converted into voltage through the floating diffusion amplifier 202. The voltage is amplified and converted into a digital signal through the analog-to-digital converter 203. In this way, the photons incident on the photosensitive element 201 are converted into digital signals and ultimately displayed as images on the computer 200.

[0064] During the above-mentioned image formation process, due to the non-uniform light response noise caused by the thermal motion of free electrons in the resistor in the photosensitive element 201 and the fluctuation of carriers in the semiconductor, the grayscale value of the collected image is greater than the grayscale value of the actual image, that is, the collected image is brighter than the actual image, resulting in a certain degree of distortion in the collected image.

[0065] To remove noise components from images and improve image quality, conventional image denoising methods include various filtering methods, such as mean filtering, median filtering, and Gaussian low-pass filtering. These filters limit the noise-related components in the image, thereby achieving the desired effect. However, this approach has the disadvantage of only partially removing noise.

[0066] To address the aforementioned issues, embodiments of the present application provide a denoising method: Multiple image noise functions are typically deployed in electronic devices, each used to calculate a single type of image noise. For example, a shot noise function, a thermal noise function, a readout noise function, and a transient noise function are used to calculate shot noise, thermal noise, readout noise, and transient noise, respectively. During the image denoising process, the electronic device sequentially inputs multiple sample images into each image noise function to remove various types of noise from the image.

[0067] The above method can remove noise more accurately. However, since it is necessary to sequentially obtain grayscale calibration values ​​for removing image noise through multiple image noise functions, the image noise removal takes a long time and the amount of calculation required by the electronic device is large.

[0068] In order to solve the problem that the above-mentioned denoising method takes a long time to remove image noise and the electronic device has a large amount of calculation. An embodiment of the present application provides another method for obtaining image noise. Specifically, the method determines multiple types of image noise existing in the image sensor, and the electronic device obtains characteristic information of each noise, wherein the characteristic information includes the relationship between the noise and related influencing factors, and the characteristic information of each noise can be represented by a corresponding single image noise function. By fusing the various noise functions in the image sensor, an initial mapping relationship between the total noise and the grayscale value can be obtained, wherein the initial mapping relationship between the total noise and the grayscale value can be represented by an image total noise function, and the image total noise function includes function parameters to be determined. The electronic device then obtains a sample image composed of multiple frames of images shot at the same shooting angle of view for the same scene, obtains determined function parameters based on the sample image and the image total noise function, and thereby obtains an image total noise function with determined function parameters.

[0069] In the embodiments of the present application, when denoising an image based on the image noise function obtained by the above-described method, because the total image noise function obtained by the above-described method incorporates multiple types of image noise in the image sensor, it is possible to accurately remove all types of noise during image denoising, and grayscale calibration values ​​can be obtained using only the total image noise function once. This shortens the denoising time compared to the above-described method of sequentially obtaining grayscale calibration values ​​for removing image noise using multiple single image noise functions. Furthermore, compared to the above-described method of inputting multiple sample images into multiple single image noise functions to sequentially obtain the total image noise function with determined function parameters, the computational effort of the electronic device is reduced.

[0070] The following describes in detail how to fuse various noise functions in the image sensor to obtain the mapping relationship between total noise and grayscale values:

[0071] It is understood that some types of image noise may be affected by the inherent characteristics of the image sensor (such as the resistance of the image sensor, the inherent structure of the image sensor circuit, and the electron energy of the dark current in the image sensor), resulting in a fixed image noise value. However, some types of image noise vary with grayscale values. Electronic devices determine the image noise corresponding to the mapping relationship between total noise and grayscale values ​​based on these two influencing factors.

[0072] First, the electronic device will determine a first type of function set that characterizes how the image noise changes with the grayscale value, and a second type of function set that characterizes how the image noise does not change with the grayscale value but is only affected by the inherent characteristic values ​​of the image sensor (such as the resistance of the image sensor, the inherent structure of the image sensor circuit, and the electron energy of the dark current in the image sensor) based on the characteristic information corresponding to each image noise in the image noise to be fused.

[0073] The characteristic information of each noise in the first type of function set is fused to obtain the unknown term in the mapping function between the total image noise and the image grayscale value; the characteristic information of each noise in the second type of function set is fused to obtain the constant term in the mapping function between the total image noise and the image grayscale value.

[0074] The following uses transient noise, shot noise, readout noise, and thermal noise in image sensors as examples to explain how these four image noise functions are fused to obtain the mapping relationship between total noise and grayscale values:

[0075] As previously mentioned, transient noise follows a Gaussian distribution relative to grayscale value x. Transient noise varies with grayscale value, while shot noise, readout noise, and thermal noise have no relationship to grayscale value x. In other words, shot noise, readout noise, and thermal noise do not change with grayscale value. Therefore, transient noise can be used to determine the unknown terms in the total image noise function, while shot noise, readout noise, and thermal noise can be used to determine the constant terms in the total image noise function.

[0076] Specifically, based on the characteristics of the Gaussian distribution, electronic devices can deduce that transient noise varies with grayscale values. That is, transient noise follows a Gaussian distribution N(μ, σ^2), where σ^2 is the variance. Therefore, the relationship between σ^2 and grayscale value x is determined as σ^2 approximately equals x. Therefore, the relationship between transient noise and grayscale value x is σ approximately equals sqrt(x), where sqrt(x) is the number of unknown terms and σ represents the sample standard deviation.

[0077] To further ensure accuracy, a coefficient a is added before sqrt(x), thereby determining that the transient noise is equal to the product of the coefficient a and the square root function.

[0078] The electronic device can then deduce from the formulas for shot noise, readout noise, and thermal noise that the three image noises have no relationship with the grayscale value x, that is, the three image noises do not change with changes in the grayscale value, so they are determined to be additive noise. Therefore, the sum of the above three image noises is determined to be the constant term of the image function.

[0079] In summary, the electronic device fuses the unknown number of terms corresponding to transient noise and the constant term corresponding to the sum of shot noise, thermal noise, and readout noise to obtain the total image noise function: y = a*sqrt(x) + b, where sqrt(x) represents the square root function, and a and b are the function parameters to be determined.

[0080] Figure 2 According to some embodiments of the present application, a schematic diagram of a curve for calculating multiple image noise superposition values ​​obtained by computer 200 using MATLAB software based on sample image fitting is shown.

[0081] like Figure 2 As shown, the value of the sample horizontal coordinate is the average value of the grayscale values ​​of multiple pixels at the same position in multiple frames of sample images, that is, Figure 1 The camera 100 shown here captures pixel values ​​with image noise.

[0082] The value of the sample ordinate is the sample standard deviation of the grayscale values ​​of multiple pixels at the same position in the multi-frame sample image, that is, the value of the ordinate represents the Figure 1 The deviation between the collected pixel value and the actual pixel value obtained by the camera 100 shown is affected by the physical characteristics of the optical device itself, that is, the image noise.

[0083] Computer 200 acquisition Figure 1 The camera 100 shown captures multiple frames of sample images of the same scene at the same shooting angle. Based on sample data consisting of multiple acquired pixel values ​​and image noise in the multiple frames of sample images, the total image noise function is fitted using MATLAB software on an electronic device: y = a'*sqrt(x)+b', where a'h and b' are determined function parameters.

[0084] It is understandable that the electronic device used to obtain image noise can be a computer, a server, a mobile phone, etc., but is not limited thereto.

[0085] The technical solution of this application will be introduced below in conjunction with specific application scenarios.

[0086] The image total noise function can be applied in the image acquisition process in the fields of security monitoring, medical image diagnosis, microscopy, autonomous driving, and astronomical observation. The following describes the process of denoising an image based on the image total noise function with the function parameters determined above in the security monitoring scenario. For example, Figure 3 According to some embodiments of the present application, a schematic diagram of an application scenario of image denoising is shown, such as Figure 3 As shown, each street in the city road is equipped with a camera 100 ( Figure 4 Only cameras 100-1 to 100-11 are shown schematically. The camera 100 transmits the captured image to the computer 300 in the road monitoring room. The image captured by the camera 100 can be presented in the computer 300 when called by the monitoring personnel, or the camera 100 transmits the captured image to the computer 300 in the road monitoring room, and the image is directly presented in the computer 300.

[0087] During the process of capturing images by the camera 100, it is necessary to denoise these images to obtain higher quality images and transmit them to the computer 300. Specifically, the camera 100 is provided with the above-mentioned total image noise function of the determination function parameter, and the total image noise function is used to calculate the total value of multiple types of image noise. The camera 100 captures images, and these images are subjected to image denoising by the camera 100 as images to be denoised. The camera 100 determines the image noise corresponding to the grayscale value in the image to be denoised based on the mapping relationship between the grayscale value of each pixel in the total image noise function and the grayscale value to be denoised, and subtracts the image noise corresponding to each determined grayscale value from the grayscale value of each pixel in the image to be denoised to obtain the denoised image. For example, Figure 4 According to some embodiments of the present application, a schematic diagram of denoising an image B to be denoised is exemplarily shown. Figure 4 As shown, taking the image B taken by the camera 100 as an example, if the grayscale value of some pixels in the image B is 500, then according to Figure 4 The image noise function in is used to determine that the total value of the multi-class image noise corresponding to the grayscale value 60 containing image noise acquired by the camera 100 is 25. Therefore, the grayscale value of the pixel with a grayscale value of 500 in image B is subtracted by 25 to obtain the denoised image B', and the grayscale value in image B' becomes 58. After the camera 100 transmits the denoised image such as image B' to the computer 300, the computer 300 can then display image B'.

[0088] Figure 5 According to some embodiments of the present application, a flow chart of an image denoising method is shown. The execution subject of the process can be Figure 3 The camera 100 in the flowchart includes the following steps:

[0089] 501: Obtain the image to be denoised.

[0090] It can be understood that the camera 100 is set with the above-obtained total image noise function with the determined function parameters, and the image captured by the camera 100 is the image to be denoised.

[0091] It can be understood that the image to be denoised can be a grayscale image. After step 603, the denoised image data is interpolated to obtain a color image.

[0092] 502: Determine the image noise corresponding to each grayscale value to be denoised in the image to be denoised according to a total image noise function having determined function parameters, wherein the image noise function represents a mapping relationship between the total image noise and the grayscale value to be denoised.

[0093] For example, Figure 4 As shown, taking the image B taken by the camera 100 as an example, if the grayscale value of some pixels in the image B is 500, then according to Figure 2 The image noise function in is used to determine that the image noise corresponding to the grayscale value 60 to be denoised collected by the camera 100 is 25.

[0094] It is understood that in some other embodiments, in order to facilitate Figure 4 While the image noise function is used in the illustrated application scenario, in other embodiments, a mapping table can be generated between total image noise and grayscale values ​​to be denoised. This mapping table is stored in the camera 100, and the camera can use a table lookup method to obtain the total image noise corresponding to the grayscale value to be denoised during the denoising process of the image to be denoised.

[0095] 503: Subtract the determined total image noise corresponding to each grayscale value from each grayscale value in the image to be denoised to obtain denoised image data.

[0096] For example, Figure 4As shown, the camera 100 subtracts 25 from the grayscale value of the pixel to be denoised in image B with a grayscale value of 500 to obtain a denoised image B', and the grayscale value in image B' becomes 58. After the camera 100 transmits the denoised image such as image B' to the computer 300, the computer 300 can present image B'.

[0097] It is understood that the camera 100 sends the denoised image data to the computer 300, and the computer 300 can then generate a denoised image based on the denoised image data. In this way, images captured by a large number of cameras 100 can be displayed on the computer 300 as denoised images with less distortion when the monitoring personnel call them.

[0098] It is understood that the above embodiment is an application scenario in which the High Dynamic Range (HDR) mode of the camera 100 is not enabled. In other embodiments, after the HDR mode is enabled, the camera 100 will capture images of the same scene at different exposure levels from the same shooting angle. The images captured at that moment are then combined to prevent overexposure in bright areas and provide some fill light in dark areas. All the images are then combined to produce an image with clear contours and layers.

[0099] Assuming only two frames are fused, each frame has its own total image noise function. The total image noise function corresponding to the fused image is the fusion of the total image noise functions corresponding to the two frames, and the weights in front of the total image noise function corresponding to each frame can be adjusted in a timely manner based on the exposure ratio and exposure gain. For example, taking the fusion of two frames, one is a long frame (i.e., long exposure time) and the other is a short frame (i.e., short exposure time), if the exposure ratio is 4:1, then if the NP curve of the short frame has been calculated, then the total image noise function of the long frame (e.g., y = a*sqrt(x) + b) can be directly multiplied by 4.

[0100] Exposure gain (gain) is generally calculated as the total image noise function for multiples of 2, such as 1x, 2x, 4x, 8x, and 16x. If the exposure gain of image sensor 101 is 12x, it can be obtained by linear interpolation of the 8x and 16x exposure gains. The total image noise functions of the long and short frames are then calculated separately, and the total noise functions are then fused.

[0101] Figure 6 FIG. 6 is a schematic diagram showing an application scenario of determining the image noise function in step 602. Figure 6As shown, the camera 100 is the device to be tested, and the computer 200 can use the method for determining the image noise function provided in the embodiment of the present application to determine the image noise function of the camera 100.

[0102] In which, the computer 200 obtains multiple frames of sample images taken by the camera 100 at the same shooting angle of view of the same scene, each frame of the sample image includes the pixel values ​​of multiple pixels, and the data after the average value and sample standard deviation processing of the grayscale values ​​of multiple pixels at the same position in the multiple frames of the sample image are used as sample data. The sample data is fitted with a preset image noise function containing the function parameters to be determined to obtain function parameters that meet the fitting conditions, thereby obtaining an image noise function with determined function parameters. In this way, compared with the computational complexity of inputting multiple sample images into multiple image noise functions separately to obtain multiple image noise functions, inputting multiple sample images into an image noise function provided by an embodiment of the present application can obtain an image noise function that can be used to calculate the superposition value of multiple image noises, and the computational complexity during the function training process is relatively small.

[0103] Figure 7 According to some embodiments of the present application, corresponding to Figure 1 and 6 , shows a flow chart of a method for determining an image noise function. The execution subject of the flow chart may be a computer 200, and the flow chart includes the following steps:

[0104] 701 : Determine the type of image noise present in the image sensor 101 .

[0105] It is understood that image sensors contain shot noise, readout noise, transient noise, thermal noise caused by resistors, photon noise, dark current noise, and light response non-uniformity noise. Computer 200 can determine several types of image noise from the various types of image noise present in image sensor 101, or all types of image noise present in image sensor 101.

[0106] 702: Obtain feature information corresponding to each image noise type.

[0107] It is understood that the characteristic information includes the relationship between noise and related influencing factors. The following uses transient noise, shot noise, readout noise and thermal noise in the image sensor 101 as examples to illustrate the characteristic information corresponding to each type of image noise.

[0108] Transient noise follows a Gaussian distribution relative to grayscale values. Its characteristic information is that it changes with grayscale values. Shot noise, readout noise, and thermal noise have no relationship to grayscale values. They do not change with grayscale values.

[0109] 703 : Perform fusion processing on the feature information corresponding to each image noise type to obtain an initial mapping relationship between the total image noise and the image grayscale value in the image sensor 101 .

[0110] It is understood that computer 200 executes step 701 to obtain the image noise to be fused. Based on the characteristic information corresponding to each image noise in the image noise to be fused, computer 200 determines a first-type function set representing how the image noise varies with grayscale value, and a second-type function set representing how the image noise does not vary with grayscale value, but is only affected by the inherent characteristics of the image sensor (e.g., the resistance of the image sensor, the inherent structure of the image sensor circuit, and the electron energy of the dark current in the image sensor). The characteristic information of each noise in the first-type function set is fused to obtain the unknown term in the mapping function between the total image noise and the image grayscale value; the characteristic information of each noise in the second-type function set is fused to obtain the constant term in the mapping function between the total image noise and the image grayscale value.

[0111] The following uses transient noise, shot noise, readout noise, and thermal noise in image sensors as examples to explain how these four image noise functions are fused to obtain the mapping relationship between total noise and grayscale values:

[0112] As previously mentioned, transient noise follows a Gaussian distribution relative to grayscale value x. Transient noise varies with grayscale value, while shot noise, readout noise, and thermal noise have no relationship to grayscale value x. In other words, shot noise, readout noise, and thermal noise do not change with grayscale value. Therefore, transient noise can be used to determine the unknown terms in the total image noise function, while shot noise, readout noise, and thermal noise can be used to determine the constant terms in the total image noise function.

[0113] Specifically, based on the characteristics of the Gaussian distribution, electronic devices can deduce that transient noise varies with grayscale values. That is, transient noise follows a Gaussian distribution N(μ, σ^2), where σ^2 is the variance. Therefore, the relationship between σ^2 and grayscale value x is determined as σ^2 approximately equals x. Therefore, the relationship between transient noise and grayscale value x is σ approximately equals sqrt(x), where sqrt(x) is the number of unknown terms and σ represents the sample standard deviation.

[0114] To further ensure accuracy, a coefficient a is added before sqrt(x), thereby determining that the transient noise is equal to the product of the coefficient a and the square root function.

[0115] The electronic device can then deduce from the formulas for shot noise, readout noise, and thermal noise that the three image noises have no relationship with the grayscale value x, that is, the three image noises do not change with changes in the grayscale value, so they are determined to be additive noise. Therefore, the sum of the above three image noises is determined to be the constant term of the image function.

[0116] In summary, the electronic device fuses the unknown number of terms corresponding to transient noise and the constant term corresponding to the sum of shot noise, thermal noise, and readout noise to obtain the total image noise function: y = a*sqrt(x) + b, where sqrt(x) represents the square root function, and a and b are the function parameters to be determined.

[0117] 704: Acquire sample image data, and determine a mapping relationship between total image noise and image grayscale values ​​in the image sensor 101 based on the sample image data and the initial mapping relationship.

[0118] It can be understood that the computer 200 can obtain multiple frames of sample images captured by the camera 100 of the device under test at the same shooting angle of view of the same scene.

[0119] It can be understood that in some embodiments, the computer 200 obtains a pixel grayscale value set from multiple frames of images, the pixel grayscale value set includes multiple pixel grayscale value subsets, and each pixel grayscale value subset is composed of pixel grayscale values ​​at the same pixel position in the multiple frames of images; average value processing is performed on each pixel grayscale value subset in the pixel grayscale value set to obtain a pixel grayscale average value set; sample standard deviation processing is performed on each pixel grayscale value subset in the pixel grayscale value set to obtain a pixel grayscale sample standard deviation set; based on the pixel grayscale average value set and the pixel grayscale sample standard deviation set, the mapping function of the total image noise and the image grayscale value is fitted to determine the first function parameter to be determined and the constant term in the unknown number term, so as to determine the determination mapping function of the total image noise and the image grayscale value in the sensor device.

[0120] The following takes the image total noise function y = a*sqrt(x) + b as an example to specifically describe the process of determining the function parameters a and b:

[0121] The computer 200 obtains multiple frames of sample images captured by the camera 100 at the same shooting angle of view for the same scene, and first pre-processes the multiple frames of sample images. The process is as follows:

[0122] It can be understood that since the sample standard deviation can represent the degree of dispersion of the grayscale values ​​of multiple pixels at the same position in multiple frames of sample images from the average value, that is, the deviation between the grayscale value containing image noise and the actual grayscale value collected by the camera 100, the total image noise can be obtained by calculating the sample standard deviation. The sample standard deviation formula is as follows:

[0123]

[0124] in, represents the average value of the grayscale values ​​of multiple pixels at the same position in the multi-frame sample image, n is the number of images in the sample image, S represents the sample standard deviation of the grayscale values ​​of multiple pixels at the same position in the multi-frame sample image (pixel grayscale sample standard deviation), x i Indicates the grayscale value of a pixel at a certain position in the i-th frame of the sample image.

[0125] For example, Figure 8 According to some embodiments of the present application, a schematic diagram of a multi-frame sample image is shown, where the grayscale values ​​of the three pixels 475, 500 and 525 in the upper left corner of the sample images A-1, A-2 and A-3 are subtracted from the sum of the squares of their average value 500, 1250, divided by 2, and the square root of the resulting value 625 is taken to obtain 25. This value 25 is the sample standard deviation of the grayscale values ​​of multiple pixels at the same position in the multi-frame sample images, that is, the sample standard deviation 25 can be used as the deviation value between the captured pixel value and the actual pixel value due to the influence of the physical properties of the optical device of the camera 100 itself.

[0126] It can be understood that since the average value can represent the central tendency of the grayscale values ​​of multiple pixels at the same position, that is, the central tendency of the grayscale values ​​containing image noise collected by the camera 100, the collected pixel values ​​with image noise collected by the camera 100 can be obtained by calculating the average value of the grayscale values ​​of multiple pixels at the same position in multiple frames of sample images. The grayscale values ​​of multiple pixels at the same position in multiple frames of sample images can be simply referred to as the pixel grayscale average value. For example, Figure 2 As shown, the average grayscale value of the three pixels in the upper left corner of the sample images A-1, A-2 and A-3, 500, is used as the collected pixel value with image noise collected by the camera 100.

[0127] It can be understood that the total image noise function y = a*sqrt(x) + b can be called NP (noise profile) function. Figure 3 As can be seen from the curve shown, the function in the low grayscale region (e.g., 0 to 60) is steeper and has a greater impact on the parameters a and b to be determined. To improve fitting accuracy, the sampling points can be increased. In contrast, the function in the high grayscale region (e.g., 60 to 255) is flatter and has a lower impact on the parameters a and b to be determined. Therefore, the sampling points can be reduced. Thus, non-uniform sampling can achieve higher fitting accuracy than uniform sampling, improving accuracy while reducing the computational effort of the computer 200.

[0128] After the above preprocessing process, a set of pixel grayscale average values ​​and a set of pixel grayscale sample standard deviations are obtained. Then, the computer 200 determines the function parameters a and b based on the grayscale average value set and the grayscale sample standard deviation set. The process is as follows:

[0129] It is understood that the function parameters a and b can be obtained by fitting the image noise function containing the function parameters to be determined by combining the grayscale mean value obtained by the above preprocessing and the grayscale sample standard deviation set. The fitting criterion (fitting condition) can adopt the least squares method, but is not limited thereto. The fitting software can be Matlab software.

[0130] Figure 9 According to some embodiments of the present application, a structural diagram of a camera 100 is shown. Figure 9 As shown, the camera 100 may be a fixed-focus lens, a zoom lens, a fisheye lens, a panoramic lens, etc. The camera 100 includes an image sensor 101, an ISP 102, a central processing unit (CPU) 103, a memory 104, an interface module 105, a communication module 106, and a bus 107. The image sensor 101, the ISP 102, the central processing unit (CPU) 103, the memory 104, the interface module 105, and the communication module 106 are coupled via the bus 107. The image sensor 101, the ISP 102, the central processing unit (CPU) 103, the memory 104, the interface module 105, and the communication module 106 may be coupled via the bus 107 to form a system on chip (SOC). In other embodiments, the image sensor 101, the ISP 102, the central processing unit (CPU) 103, the memory 104, the interface module 105, and the communication module 106 may be independent devices.

[0131] The image sensor 101 is used to convert the collected light signals reflected by the scene into digital electrical signals to generate raw image (RAW) data, for example, data in a Bayer format.

[0132] ISP102 is an application-specific integrated circuit (ASIC) for image data processing, which is used to further process the image data generated by image sensor 101 to obtain better image quality. In an embodiment of the present application, camera 100 can pass the raw format image data collected by image sensor 101 to ISP102, which performs a series of processing on the raw format image data to obtain better image data. For example, ISP102 will use the image denoising method provided in this application to process the raw image data obtained from image sensor 101 to obtain a denoised image.

[0133] The CPU 103 may include one or more processing units, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a microprocessor (MCU), an artificial intelligence (AI) processor, or a processing module or processing circuit such as a programmable logic device (FPGA). Different processing units may be independent devices or integrated into one or more processors.

[0134] The memory 104 can be used to store data, software programs, and modules, and can be a volatile memory, such as a random-access memory (RAM) or a double data rate synchronous dynamic random access memory (DDRSDRAM).

[0135] The interface module 105 includes an external memory interface, a universal serial bus (USB) interface, and the like. The external memory interface can be used to connect to an external non-volatile memory (NVM), such as a read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); or a combination of these types of memory, or a removable storage medium, such as a secure digital (SD) memory card, to expand the storage capacity of the camera 100.

[0136] The communication module 106 , such as a WIFI module, a Universal Serial Bus (USB), a 4G or 5G module, etc., is used for the camera 100 to communicate with other electronic devices via the communication module 106 .

[0137] The bus 107 is used to couple the image sensor 101, the ISP 102, the central processing unit (CPU) 103, the memory 104, the interface module 105, and the communication module 106. The bus 107 may be an advanced high-performance bus (AHB) or another type of data bus.

[0138] I understand. Figure 9 The structure of the camera 100 shown is only an example and does not constitute a specific limitation on the camera 100. In other embodiments, the camera 100 may include more or fewer modules, and some modules may be combined or split, which is not limited in the embodiments of the present application.

[0139] further, Figure 10 According to some embodiments of the present application, a schematic diagram of the structure of the ISP 102 in the camera 100 is shown. Figure 10 As shown, ISP 102 includes a processor 1031 , an image transmission interface 1032 , a general peripheral device 1033 , an image recognition module 1034 and a general function module 1035 .

[0140] Processor 1031 is used for logic control and scheduling in ISP 102 .

[0141] The image transmission interface 1032 is used for transmitting image data.

[0142] The general peripheral devices 1033 include but are not limited to: a bus for coupling the various modules of ISP102 and its controller, a bus for coupling with other devices, such as the advanced high-performance bus (AHB), which enables the ISP to communicate with other devices (such as DSP, CPU, etc.) with high performance; and a watchdog unit (WATCHDOG) for monitoring the working status of the ISP.

[0143] The padding module 1034 is used to perform padding operations on the image data according to the requirements of the image processing model in the NPU, such as the deep learning model, for the input data.

[0144] The general function module 1035 is used to process the image input to the ISP 102, including but not limited to bad pixel correction (BPC), black level correction (BLC), automatic white balance (AWB), gamma correction, color correction, noise reduction, edge enhancement, brightness, contrast, and chroma adjustment. When the image sensor transmits RAW image data to the image signal processor 1030, it is first processed by the function module.

[0145] I understand. Figure 10 The structure of ISP102 shown is only an example. Those skilled in the art should understand that it may include more or fewer modules, and may also combine or split some modules, and the embodiments of the present application are not limited thereto.

[0146] The general function modules can include RAW domain processing module, YUV domain processing module and RGB domain processing module. Figure 11 A schematic diagram of a process of processing image data by a general functional module is shown, which includes the following steps.

[0147] The RAW domain processing module performs bad pixel correction, black level correction and automatic white balance on image data.

[0148] The image data processed in the RAW domain is interpolated to obtain the image data in the RGB domain, and then the RGB domain processing module performs gamma correction and color correction on the image data in the RGB domain.

[0149] The image data processed in the RGB domain undergoes color gamut conversion to obtain image data in the YUV domain. The YUV domain processing module then performs noise reduction, edge enhancement, and brightness / contrast / chroma adjustments on the YUV domain image data. In the embodiments of the present application, noise reduction of the YUV domain image data can be performed using the image noise reduction method provided in the embodiments of the present application.

[0150] The various embodiments of the mechanism disclosed in the application can be implemented in hardware, software, firmware, or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0151] Program code can be applied to input instructions to perform the functions described herein and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0152] Program code can be implemented with a high-level programming language or an object-oriented programming language to communicate with the processing system. Where necessary, program code can also be implemented in assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0153] In some cases, the disclosed embodiments can be implemented in hardware, firmware, software or any combination thereof. The disclosed embodiments can also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which can be read and executed by one or more processors. For example, instructions can be distributed over a network or by other computer-readable media. Therefore, machine-readable media can include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including but not limited to, floppy disks, optical disks, optical discs, read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROM), random access memories (RAM), erasable programmable read-only memories (EPROM), electrically erasable programmable read-only memories (EEPROM), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in electrical, optical, acoustic or other forms of propagation signals. Accordingly, machine-readable media includes any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).

[0154] In the accompanying drawings, some structural or method features may be shown in a particular arrangement and / or order. However, it should be understood that such a particular arrangement and / or order may not be required. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of a structural or method feature in a particular figure does not imply that such feature is required in all embodiments, and in some embodiments, such features may not be included or may be combined with other features.

[0155] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems raised by this application. In addition, in order to highlight the innovative part of this application, the above-mentioned device embodiments of this application do not introduce units / modules that are not closely related to solving the technical problems raised by this application. This does not mean that other units / modules do not exist in the above-mentioned device embodiments.

[0156] It should be noted that in the examples and description of this patent, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a" does not exclude the presence of other identical elements in the process, method, article or apparatus comprising the element.

[0157] Although the present application has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the application.

Claims

1. A method for obtaining image noise, applied to an electronic system, wherein the electronic system includes a sensing device and a processing device, characterized in that: The method comprises: The processing device determines a plurality of image noise types in the sensing device coupled to the processing device, the plurality of image noise types comprising shot noise, thermal noise, readout noise, and transient noise; The processing device classifies the image noise type that varies with the grayscale value into a first type of image noise, the transient noise belongs to the first type of image noise, and the transient noise obeys a Gaussian distribution relative to the grayscale value; The processing device classifies the image noise type that does not change with the grayscale value into a second type of image noise, wherein the shot noise, the thermal noise and the readout noise belong to the second type of image noise; determining a mapping relationship between total image noise and image grayscale values ​​of the sensing device based on the classified first type of image noise, the second type of image noise, and at least one sample image data acquired by the sensing device; The mapping relationship between the total image noise and the image grayscale value of the sensing device includes a mapping function between the total image noise and the image grayscale value of the sensing device, and determining the mapping relationship between the total image noise and the image grayscale value of the sensing device includes: Determining a first type function set that characterizes that image noise varies with grayscale values, and a second type function set that characterizes that image noise does not vary with grayscale values; The characteristic information of each noise in the first type of function set is fused to obtain the unknown term in the mapping function between the total image noise and the image grayscale value; the characteristic information of each noise in the second type of function set is fused to obtain the constant term in the mapping function between the total image noise and the image grayscale value.

2. The method according to claim 1, characterized in that Determining a mapping relationship between the total image noise and the image grayscale value of the sensing device based on the classified first-category image noise, the second-category image noise, and at least one sample image data acquired by the sensing device includes: Fusing the relationship between image noise and grayscale value in the first type of image noise to obtain unknown terms in the mapping function between total image noise and image grayscale value; Fusing the relationship between image noise and grayscale value in the second type of image noise to obtain a constant term in a mapping function between total image noise and image grayscale value; Based on at least one sample image data acquired by the sensing device, the unknown term and the constant term, a first function parameter to be determined and a constant term in the unknown term are determined to determine a mapping function between the total image noise and the image grayscale value of the sensing device as a mapping relationship between the total image noise and the image grayscale value.

3. The method according to claim 1, characterized in that The sensing device is a device including an image sensor.

4. The method according to claim 2, characterized in that The sample image data includes multiple frames of sample images shot at the same scene at the same shooting angle.

5. The method according to claim 4, characterized in that The determining, based on at least one sample image data acquired by the sensing device, the unknown term, and the constant term, a first to-be-determined function parameter and a constant term in the unknown term, so as to determine a mapping function between the total image noise and the image grayscale value of the sensing device as a mapping relationship between the total image noise and the image grayscale value, includes: For each sample image data, the following processing is performed: Acquire a pixel grayscale value set from the multiple frames of sample images, wherein the pixel grayscale value set includes a plurality of pixel grayscale value subsets, each pixel grayscale value subset consisting of pixel grayscale values ​​at the same pixel position in the multiple frames of sample images; Performing average processing on each subset of pixel grayscale values ​​in the pixel grayscale value set to obtain a pixel grayscale average value set; Performing sample standard deviation processing on each subset of pixel grayscale values ​​in the pixel grayscale value set to obtain a pixel grayscale sample standard deviation set; Based on at least one set of pixel grayscale average values ​​and at least one set of pixel grayscale sample standard deviations corresponding to at least one sample image data, the first function parameter to be determined and the constant term in the unknown number term are determined to determine the mapping function between the total image noise and the image grayscale value of the sensing device, as the mapping relationship between the total image noise and the image grayscale value.

6. The method according to claim 1, characterized in that The sample image data includes sample image data of a first grayscale range and sample image data of a second grayscale range, and an amount of the sample image data of the first grayscale range is different from an amount of the sample image data of the second grayscale range.

7. A readable medium, characterized in that The readable medium stores instructions, which, when executed on an electronic device, enable the electronic device to execute the method for obtaining image noise according to any one of claims 1 to 6.

8. An electronic device, characterized in that: include: a memory for storing instructions to be executed by one or more processors of the electronic device, and The processor is one of the processors of the electronic device, and is used to execute the method for obtaining image noise according to any one of claims 1 to 6.

Images