Device, board, method and readable storage medium for denoising a camera

Abstract

The present invention relates to a device, a board, a method and a readable storage medium for denoising a camera. The computing device of the present invention is included in an integrated circuit device, which includes a general-purpose interconnect interface and other processing devices. The computing device interacts with other processing devices to jointly complete the computing operations specified by the user. The integrated circuit device may further include a storage device, which is respectively connected to the computing device and other processing devices for data storage of the computing device and other processing devices.

Description

Technical Field

[0001] The present invention generally relates to the field of image signal processing. More specifically, the present invention relates to an apparatus, a board, a method, and a readable storage medium for denoising a camera. Background Art

[0002] Cameras and image sensors are highly mature products. The market is flooded with various cameras with different optical characteristics and sensors with different electrical characteristics, resulting in the inability of an image signal processor (ISP) to automatically uniformly compensate for these hardware characteristics, making the image processing effect less than expected. To solve this problem, suppliers of cameras and image sensors provide a set of calibration tools, using the images obtained by existing instruments and equipment as inputs, and demonstrating adjustment procedures to make the output better approximate the desired effect. Most of the prior arts adopt the Bayer denoising algorithm, that is, using deep learning technology to obtain a better correction effect.

[0003] When performing black level correction, first calibrate the mean value of a set of black frames as a reference value, and subtract this reference value in subsequent algorithms to remove background noise. However, in the case of black frames, the pixels actually follow a Gaussian distribution, and factors such as photo response non-uniformity (PRNU), dark current, and fixed pattern noise will all have a certain impact, so it is difficult to guarantee the accuracy of black level correction.

[0004] When performing noise calibration, since the existing denoising algorithms use a set of filters, the output of noise calibration is only the gain size or the dark frame variance as the filtering intensity, without considering a more accurate model of noise and the distribution describing noise, resulting in inaccurate noise calibration and naturally unsatisfactory denoising effects.

[0005] Therefore, a solution with good denoising effect is urgently needed. Summary of the Invention

[0006] To at least partially solve the technical problems mentioned in the background art, the solution of the present invention provides an apparatus, a board, a method, and a readable storage medium for denoising a camera.

[0007] In one aspect, the present invention discloses a method for denoising a camera, including: obtaining a gray value model of pixels; obtaining multiple black frames, dark frames, and bright frame images captured by the camera at multiple sensitivities; substituting the black frame images into the gray value model to obtain black frame parameters; substituting the dark frame and bright frame images and the black frame parameters into the gray value model to obtain non-black frame parameters; using the black frame parameters and non-black frame parameters as prior inputs to train a neural network model; and inputting the image data captured by the camera into the trained neural network model for image signal processing to obtain a denoised output.

[0008] In another aspect, the present invention discloses a computer-readable storage medium having stored thereon computer program code for denoising a camera, which, when run by a processing device, executes the foregoing method.

[0009] In another aspect, the present invention discloses an integrated circuit device for denoising a camera, including a processing device and a computing device. The processing device is configured to: obtain a gray value model of pixels; obtain multiple black frames, dark frames, and bright frame images captured by the camera at multiple sensitivities; substitute the black frame images into the gray value model to obtain black frame parameters; and substitute the dark frame and bright frame images and the black frame parameters into the gray value model to obtain non-black frame parameters. The computing device is configured to: use the black frame parameters and non-black frame parameters as prior inputs to train a neural network model; and input the image data captured by the camera into the trained neural network model for image signal processing to obtain a denoised output.

[0010] In another aspect, the present invention discloses a board card including the foregoing integrated circuit device.

[0011] By using the black frame parameters and non-black frame parameters as prior inputs to train a neural network model, the present invention can more accurately calibrate noise and optimize the denoising effect of image signal processing. Description of the Drawings

[0012] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features, and advantages of the exemplary embodiments of the present invention will become readily understood. In the drawings, several embodiments of the present invention are shown by way of example and not limitation, and like or corresponding reference numerals indicate like or corresponding parts, wherein:

[0013] Figure 1 is a structural diagram of a board card showing an embodiment of the present invention;

[0014] Figure 2 is a structural diagram of an integrated circuit device showing an embodiment of the present invention;

[0015] Figure 3 is an internal structural schematic diagram of a computing device showing an embodiment of the present invention;

[0016] Figure 4 is a schematic diagram showing the internal structure of a processor core according to an embodiment of the present invention;

[0017] Figure 5 is a schematic diagram showing when a processor core wants to write data to a processor core in another cluster;

[0018] Figure 6 is a schematic diagram showing a single pixel of a camera according to an embodiment of the present invention converting photons into analog - to - digital units (ADUs); and

[0019] Figure 7 is a flowchart showing a method for denoising a camera according to another embodiment of the present invention. Detailed implementation manners

[0020] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present invention.

[0021] It should be understood that the terms "first", "second", "third", and "fourth", etc. in the claims, the description, and the drawings of the present invention are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" used in the description and claims of the present invention indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0022] It should also be understood that the terms used in the description of the present invention herein are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. As used in the description and claims of the present invention, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms. It should be further understood that the term "and / or" used in the description and claims of the present invention refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0023] As used in the description and claims of this specification, the term "if" can be interpreted as "when", "once", "in response to determining", or "in response to detecting" according to the context.

[0024] Next, the detailed implementation manners of the present invention will be described in detail in conjunction with the accompanying drawings.

[0025] The present invention proposes a noise reduction solution in a neural network model, which is originally used to perform specific tasks related to image signal processing, such as image recognition, object detection, semantic segmentation, video understanding, image generation, denoising, demosaicing, etc. The present invention substitutes black frame parameters and non-black frame parameters as prior inputs into the neural network model for training, so that during the training process of this neural network model, the influence of noise on image output is suppressed by updating the parameters.

[0026] An embodiment of the present invention is applied to a network architecture with a camera to achieve computer vision. The network architecture has a board 10 as shown in Figure 1 Figure. As shown in Figure 1 Figure, the board 10 includes a chip 101, which is a System on Chip (SoC), or a system-on-chip, integrated with one or more combined processing devices. The combined processing device is an artificial intelligence computing unit used to support various deep learning and machine learning algorithms to meet the intelligent processing requirements in complex scenarios in the field of computer vision. In particular, deep learning technology is widely used in the field of cloud intelligence. A significant feature of cloud intelligence applications is a large amount of input data, which requires high storage and computing capabilities of the platform. The board 10 of this embodiment is applicable to cloud intelligence applications and has a large off-chip storage, on-chip storage, and powerful computing capabilities.

[0027] The chip 101 is connected to an external device 103 through an external interface device 102. In this embodiment, the external device 103 is a camera. The image data to be processed can be transmitted from the external device 103 to the chip 101 through the external interface device 102. According to different application scenarios, the external interface device 102 can have different interface forms, such as a PCIe interface, etc.

[0028] The board 10 further includes a storage device 104 for storing data, which includes one or more storage units 105. The storage device 104 is connected to the controller device 106 and the chip 101 through a bus for data transmission. The controller device 106 in the board 10 is configured to control the state of the chip 101. For this purpose, in one application scenario, the controller device 106 may include a Micro Controller Unit (MCU).

[0029] Figure 2 Figure is a structural diagram of the combined processing device in the chip 101 of this embodiment. As shown in Figure 2 Figure, the combined processing device 20 includes a computing device 201, an interface device 202, a processing device 203, and a DRAM 204.

[0030] The computing device 201 is configured to perform user-specified operations, mainly implemented as a single-core intelligent processor or a multi-core intelligent processor for performing deep learning or machine learning computations. It can interact with the processing device 203 through the interface device 202 to jointly complete computer vision operations.

[0031] The interface device 202 is used to transfer data and control instructions between the computing device 201 and the processing device 203. For example, the computing device 201 can obtain input data from the processing device 203 via the interface device 202 and write it into the storage device on the chip of the computing device 201. Further, the computing device 201 can obtain control instructions from the processing device 203 via the interface device 202 and write them into the control cache on the chip of the computing device 201. Alternatively or optionally, the interface device 202 can also read the data in the storage device of the computing device 201 and transfer it to the processing device 203.

[0032] The processing device 203, as a general-purpose processing device, performs basic controls including but not limited to data transfer, starting and / or stopping of the computing device 201. Depending on the implementation, the processing device 203 can be a central processing unit (CPU), a graphics processing unit (GPU), or one or more types of processors such as other general-purpose and / or dedicated processors, including but not limited to digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and the number of them can be determined according to actual needs. As mentioned above, only for the computing device 201 of the present invention, it can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when considering the computing device 201 and the processing device 203 integrated together, the two are regarded as forming a heterogeneous multi-core structure.

[0033] The DRAM 204 is used to store the data to be processed. It is a DDR memory with a size usually of 16G or larger, used to save the data of the computing device 201 and / or the processing device 203, including the training sample set for training the neural network model.

[0034] Figure 3Figure 2 shows a schematic diagram of the internal structure of computing device 201. Computing device 201 is used to process computer vision input data. The computing device 201 in the figure adopts a multi-core hierarchical design. Computing device 201, as a system on a chip, includes multiple clusters, each of which includes multiple processor cores. In other words, computing device 201 is constructed in a hierarchy of system on a chip, clusters, and processor cores.

[0035] At the system-on-chip level, Figure 3 As shown, the computing device 201 includes an external storage controller 301 , a peripheral communication module 302 , an on-chip interconnect module 303 , a synchronization module 304 and multiple clusters 305 .

[0036] There can be multiple external storage controllers 301, and two are shown in the figure as an example. They are used to respond to access requests issued by the processor core and access external storage devices, such as Figure 2 The DRAM 204 in the chip is used to read image data from outside the chip or write data. The peripheral communication module 302 is used to receive control signals from the processing device 203 through the interface device 202 to start the computing device 201 to perform tasks. The on-chip interconnect module 303 connects the external storage controller 301, the peripheral communication module 302 and multiple clusters 305 to transmit data and control signals between each module. The synchronization module 304 is a global synchronization barrier controller (GBC) used to coordinate the work progress of each cluster and ensure information synchronization. Multiple clusters 305 are the computing cores of the computing device 201. Four are shown as an example in the figure. With the development of hardware, the computing device 201 of the present invention can also include 8, 16, 64, or even more clusters 305. The clusters 305 are used to efficiently execute deep learning algorithms.

[0037] At the cluster level, Figure 3 As shown, each cluster 305 includes multiple processor cores (IPU cores) 306 and a memory core (MEM core) 307 .

[0038] The figure shows four processor cores 306 as an example, but the present invention does not limit the number of processor cores 306. Figure 4 Each processor core 306 includes three modules: a control module 41 , a calculation module 42 and a storage module 43 .

[0039] The control module 41 is used to coordinate and control the operations of the arithmetic module 42 and the storage module 43 to complete deep learning tasks. It includes an instruction fetch unit (IFU) 411 and an instruction decode unit (IDU) 412. The instruction fetch unit 411 is used to obtain instructions from the processing device 203, and the instruction decode unit 412 decodes the obtained instructions and sends the decoding results as control information to the arithmetic module 42 and the storage module 43.

[0040] The arithmetic module 42 includes a vector arithmetic unit 421 and a matrix arithmetic unit 422. The vector arithmetic unit 421 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and non-linear transformation; the matrix arithmetic unit 422 is responsible for the core calculations of deep learning algorithms, namely matrix multiplication and convolution.

[0041] The storage module 43 is used to store or transfer relevant data, including a neuron storage unit (neuron RAM, NRAM) 431, a weight storage unit (weight RAM, WRAM) 432, an input / output direct memory access module (input / output direct memory access, IODMA) 433, and a move direct memory access module (move direct memory access, MVDMA) 434. The NRAM 431 is used to store the feature maps for the processor core 306 to calculate and the intermediate results after calculation; the WRAM 432 is used to store the weights of the deep learning network; the IODMA 433 controls the memory access of the NRAM 431 / WRAM 432 and the DRAM 204 through the broadcast bus 309; the MVDMA 434 is used to control the memory access of the NRAM 431 / WRAM 432 and the SRAM 308.

[0042] Return to Figure 3 , the storage core 307 is mainly used for storage and communication, that is, to store the shared data or intermediate results between the processor cores 306, and to perform the communication between the execution cluster 305 and the DRAM 204, the communication between the execution clusters 305, and the communication between the processor cores 306, etc. In other embodiments, the storage core 307 has the ability to perform scalar operations and is used to perform scalar operations.

[0043] The storage core 307 includes a shared storage unit (SRAM) 308, a broadcast bus 309, a cluster direct memory access module (CDMA) 310, and a global direct memory access module (GDMA) 311. The SRAM 308 acts as a high-performance data transfer station. The data reused between different processor cores 306 within the same cluster 305 does not need to be obtained by each processor core 306 from the DRAM 204 separately. Instead, it is transferred between the processor cores 306 via the SRAM 308. The storage core 307 only needs to quickly distribute the reused data from the SRAM 308 to multiple processor cores 306, so as to improve the inter-core communication efficiency and greatly reduce the on-chip and off-chip input / output access.

[0044] The broadcast bus 309, CDMA 310, and GDMA 311 are respectively used to perform communication between processor cores 306, communication between clusters 305, and data transfer between the cluster 305 and the DRAM 204. The following will be described separately.

[0045] The broadcast bus 309 is used to complete high-speed communication between processor cores 306 within the cluster 305. The inter-core communication modes supported by the broadcast bus 309 in this embodiment include unicast, multicast, and broadcast. Unicast is a point-to-point (i.e., from a single processor core to a single processor core) data transfer. Multicast is a communication mode that transfers a piece of data from the SRAM 308 to specific several processor cores 306. Broadcast is a communication mode that transfers a piece of data from the SRAM 308 to all processor cores 306, which is a special case of multicast.

[0046] The CDMA 310 is used to control the access to the SRAM 308 between different clusters 305 within the same computing device 201. Figure 5 A schematic diagram is shown when a processor core wants to write data to a processor core in another cluster to illustrate the working principle of the CDMA 310. In this application scenario, the same computing device includes multiple clusters. For the convenience of description, only cluster 0 and cluster 1 are shown in the figure. Cluster 0 and cluster 1 respectively include multiple processor cores. Also for the convenience of description, only processor core 0 in cluster 0 and only processor core 1 in cluster 1 are shown in the figure. Processor core 0 wants to write data to processor core 1.

[0047] First, processor core 0 sends a unicast write request to write data into the local SRAM 0. CDMA 0 acts as the master end, and CDMA 1 acts as the slave end. The master end pushes the write request to the slave end, that is, the master end sends the write address AW and the write data W to transfer the data to SRAM 1 in cluster 1. Then the slave end sends a write response B as a reply. Finally, processor core 1 in cluster 1 sends a unicast read request to read the data from SRAM 1.

[0048] Back to Figure 3 , GDMA 311 cooperates with the external storage controller 301 to control the memory access from SRAM 308 in cluster 305 to DRAM 204, or to read data from DRAM 204 into SRAM 308. As can be seen from the above, the communication between DRAM 204 and NRAM431 or WRAM 432 can be achieved through two channels. The first channel is to directly connect DRAM 204 with NRAM 431 or WRAM 432 through IODAM 433; the second channel is to first make the data transfer between DRAM 204 and SRAM 308 through GDMA 311, and then make the data transfer between SRAM 308 and NRAM 431 or WRAM 432 through MVDMA 434. Although seemingly the second channel requires more components to participate and the data flow is longer, in fact, in some embodiments, the bandwidth of the second channel is much larger than that of the first channel. Therefore, the communication between DRAM 204 and NRAM 431 or WRAM 432 may be more efficient through the second channel. The embodiments of the present invention can select the data transfer channel according to its own hardware conditions.

[0049] In other embodiments, the functions of GDMA 311 and IODMA 433 can be integrated into the same component. For the convenience of description, the present invention regards GDMA 311 and IODMA 433 as different components. For those skilled in the art, as long as the functions implemented and the technical effects achieved are similar to those of the present invention, they belong to the protection scope of the present invention. Further, the functions of GDMA 311, IODMA 433, CDMA 310, and MVDMA 434 can also be implemented by the same component.

[0050] This embodiment models various sources of camera noise and proposes a set of noise calibration methods, which can accurately describe any camera (including internal sensors) to generate data for denoising in the Bayer domain. These data are added as parameters to the training of the neural network model, so that the trained model directly produces a denoising effect on these cameras.

[0051] The camera in this embodiment is connected to an analog-to-digital converter, which is used to convert the image signal (analog signal) into a RAW image signal (digital signal). A RAW image is the original data when the camera converts the captured optical signal into a digital signal. A RAW file is a record of the original information of the camera, including some metadata generated by the camera shooting, such as ISO settings, shutter speed, aperture value, white balance, etc.

[0052] Each photosite of each camera is only sensitive to one primary color component of RGB (red, green, blue). Figure 6 A schematic diagram showing a single pixel of the camera converting photons into analog-to-digital units (ADUs). The camera includes a color filter 601 and multiple pixels 602. The color filter 601 is composed of color filters of three primary colors RGB. Each color filter of the primary color only allows the light of that color to pass through to the pixel 602. For example, the red color filter only allows red light to pass through and reach the pixel 602. Each pixel 602 includes three sensors 603, that is, three photosites, and each sensor 603 corresponds to a primary color filter.

[0053] When the light 605 of an object enters the camera, the color filter 601 filters out light of other wavelengths. As shown in the figure, for example, the green color filter only allows the green light wave in the light 605 of the object to pass through and enter the sensor 603. The light irradiating on the sensor 603 during exposure can be described by photons. When a photon reaches a specific layer on the sensor 603, each photon will excite an electron. The unit of the number of electrons is "e-", that is, the number of electrons excited by the sensor 603 reflects the number of photons reaching the sensor 603. After the exposure time ends, all the excited electrons are received by the analog-to-digital converter 606. The analog-to-digital converter 606 calculates the number of electrons and converts it into an ADU value.

[0054] In the aforementioned process of electron quantization, the analog-to-digital converter 606 uses a conversion factor "M" (from an internal multiplier), in units of "ADU / e-", multiplies it by the number of electrons to obtain the corresponding ADU value. The magnitude of "M" only depends on the ISO setting of the camera. The higher the ISO value, the higher "M" will be.

[0055] The photon flux hitting each photosite on the sensor 603 is proportional to the intensity of the light. A darker environment scene will generate fewer electrons, resulting in a lower ADU value, while a brighter area will generate a higher ADU value. The model of the sensor 603 can be expressed by the following formula (1):

[0056] ADU = M × EL

[0057] Where EL represents the number of electrons in each image.

[0058] However, even under completely uniform illumination, the number of photons received by each photosensitive point is not the same. The number of photons is a random variable rather than a constant. The different photon values generated under completely uniform illumination are the so-called photon shot noise. Photon shot noise is a random variable that follows a Poisson distribution. The Poisson process is described as "the given number of discrete events occurring in a fixed time and / or space, which occur at a known average rate, independent of the time since the last event." In the aforementioned scenario, the discrete event refers to the instant when a photon generates an electron, and "in a fixed time and space" corresponds to the exposure time on the surface of each photosensitive point, and its rate is actually independent of any photons that arrived previously.

[0059] Since photon shot noise follows a Poisson distribution, the expected value and variance of the ADU value can be expressed using the following Equation 2 and Equation 3:

[0060] Eva(ADU) = Eva(M × EL)

[0061] = Eva(M) × Eva(EL)

[0062] = M × Eva(EL)

[0063] = M × λ

[0064] Var(ADU) = Var(M × EL)

[0065] = M 2 × Var(EL)

[0066] = M × Eva(EL)

[0067] = M 2 × λ

[0068] = M × Eva(ADU)

[0069] Where Eva(ADU) is the expected value or average value of the ADU value, Var(ADU) is the variance of the ADU value, and λ is the average number of electrons in each image.

[0070] In Figure 6 the sensor 603, each photosensitive point converts the charge of the electrons generated by photon impact into a voltage and transmits it to the analog-to-digital converter 606. In fact, the output of the analog-to-digital converter 606 is not exactly proportional to the number of electrons emitted by the photons. This is because when the camera is covered and photographed (i.e., the black frame image), in order to retain the dark area signal, a black level signal output will be generated in the analog-to-digital converter 606. Therefore, even in a completely black state, the output of the analog-to-digital converter 606 always has a deviation or baseline, and this kind of noise is called read noise.

[0071] Adding the read noise to the model gives Equation Four as follows:

[0072] ADU = M × EL + RNS

[0073] Where RNS is the read noise. Substituting Equation Four into Equation Three gives Equation Five as follows:

[0074] Eva(ADU) = M × Eva(EL) + Eva(RNS)

[0075] = M × λ + Eva(RNS)

[0076] = M × λ + μ RNS

[0077] Where μ RNS is the mean of the read noise.

[0078] It can be seen from this that in an image with read noise, the average pixel value is the sum of the optical signal and the read noise. By subtracting μ RNS , the signal of the RAW image can be restored, which is the so-called black level correction. The corrected signal can be expressed by Equation Six and Equation Seven:

[0079] ADU RNC = M × EL + RNS - μ RNS

[0080] Eva(ADU RNC ) = M × λ + Eva(RNS) - μ RNS

[0081] = M × λ + μ RNS - μ RNS

[0082] = M × λ

[0083] Where ADU RNC represents the corrected ADU value. After the read noise is corrected, the average value of the read noise can be subtracted from the ADU value. Although it cannot be completely removed, the influence of the read noise can be basically ignored. In other words, the read noise consists of two parts. One part is a fixed black level, that is, the mean of the read noise μ RNS , and the other part fluctuates randomly and follows a normal distribution. Equation Six removes the mean of the read noise μ RNS , that is, the corrected ADU value has excluded the influence of the mean of the read noise. For simplicity of notation, the ADU values in the following equations and descriptions are all the corrected ADU values ADU RNC .

[0084] Taking the variance of Equation Six gives Equation Eight as follows:

[0085] Var(ADU) = Var(M × EL) + Var(RNS)

[0086] = M 2 × λ + RN 2

[0087] Wherein, RN is the standard deviation of the noise signal.

[0088] In fact, due to the differences in the microlenses, analog-to-digital converters, and other sensor circuit elements at each photosensitive point during the manufacturing process, the light sensitivity of each photosensitive point will not be exactly the same. Therefore, the M term in the above model is not exactly the same value for each image. On the contrary, the M value is an attribute of each pixel, and this situation is called photo response non-uniformity (PRNU). Considering PRNU, this embodiment replaces the factor "M" with "M p ", where the subscript p refers to the p-th pixel. Equation six is further adjusted to Equation nine as follows:

[0089] ADU p = M p × EL + RNS - μ RNS

[0090] Each pixel has the same M p value, which means that M p For each image, when keeping the same ISO speed (the sensitivity of the photosensitive film changes with the ISO speed), it is a constant. The average value of all Mp factors corresponding to all light spots on the surface of the sensor 603 is also a constant, which is called the expected multiplier EM here, that is, the average gain of each pixel of the camera. EM can be represented by Equation ten:

[0091] EM = Eva(M p )

[0092] Furthermore, the M p value is regarded as composed of two components. One is the expected multiplier EM, and the other is the photosensitivity deviation MD p relative to the expected multiplier EM, that is, the difference between the gain of the p-th pixel and the average gain. Then M p can be represented by Equation eleven:

[0093] M p = EM + MD p

[0094] Substituting Equation eleven into Equation nine, Equation twelve can be obtained as follows:

[0095] ADUp =(EM + MD p ) × EL p + RNS - μ RNS

[0096] where ADU p is the RAW image value of the p-th pixel, and EL p is the number of photons received by the p-th pixel.

[0097] Taking the mean of Equation (12), we get Equation (13) as follows:

[0098] Eva(ADU p ) = Eva((EM + MD p ) × EL p ) + Eva(RNS) - μ RNS

[0099] = Eva((EM + MD p )) × Eva(EL p ) + Eva(RNS) - μ RNS

[0100] = Eva(EM) × λ + μ RNS - μ RNS

[0101] = EM × λ

[0102] Taking the variance of Equation (12), we get Equation (14) as follows:

[0103] Var(ADU p ) = Var((EM + MD p ) × EL p ) + Var(RNS)

[0104] = PRNU 2 × λ 2 + (EM 2 + PRNU 2 ) × λ + RN 2

[0105] Deriving from the sensor model to Equation (14) explains the distribution of various noises, that is, the variance of ADU p is a quadratic polynomial of λ. From the above equations, it can be seen that PRNU is a very strong noise source.

[0106] Next, further separate PRNU. For a given pixel, PRNU does not change across different images. That is, the average value of the images will highlight the PRNU noise. Using Equation (12) to calculate the average value, we can get Equation (15) as follows:

[0107]

[0108] Among them, n represents the average number of images. It can be seen from Equation XV that except for PRNU noise, other types of noise are reduced after averaging. Further, taking the variance of both ends of Equation XV, Equation XVI is obtained as follows:

[0109]

[0110] In the above equations, it can be seen that the quadratic term of PRNU is not affected by averaging, while the influence of other types of noise decreases with the increase of the n value of the average number of images. Since PRNU is caused by MD p and for the same camera, MD p remains unchanged, the influence of PRNU can be eliminated by comparing the differences of multiple images taken by the same camera. Equation XVII can be obtained from Equation XII as follows:

[0111] ΔADU p =(EM + MD p )×ΔEL p +ΔRNS

[0112] Among them, "Δ" represents the difference in pixel values of two images taken by the same pixel of the same camera in the same scene. Since MD p is the characteristic of the photosensitive point and does not change during the shooting process, the influence of MD p can be excluded when taking the variance of Equation XVII, as shown in Equation XVIII:

[0113]

[0114] Comparing with Equation XIV, Equation XVIII does not include the quadratic term of the PRNU variance. Although it still includes the linear term of PRNU, PRNC 2 is usually much smaller than EM 2 , and the PRNC 2 term can be ignored in Equation XVIII. In other words, it can be seen from Equation XVIII that only read noise and shot noise are the main influencing factors.

[0115] So far, this embodiment completes the modeling, and the processing device 203 obtains the gray value model of the pixel. More specifically, the gray value model of this embodiment includes Equation XII, Equation XIII, Equation XIV and Equation XVIII, where Equation XII corresponds to the ADU value, Equation XIII corresponds to the expected value of the ADU value, Equation XIV corresponds to the variance of the ADU value, and Equation XVIII corresponds to the variance of the difference between two pictures.

[0116] Next, start preparing the data. To obtain the noise model of the sensor 603, several sets of standard color charts need to be photographed in a light box environment within a certain ISO gradient range. In this embodiment, a 24-color chart can be used as the standard color chart. The 24-color chart contains six levels of gray-scale color blocks, the additive primary colors, the subtractive primary colors, as well as skin color and the true colors of simulated natural objects, with a total of 24 color blocks. Fix the 24-color chart in front of the camera and try to take pictures with the central area of the camera to avoid vignetting affecting the image clarity. In this embodiment, the following ISO values can be used to photograph the 24-color chart exemplarily: 100, 200, 400, 800, 1600, 3200, 6400, 12800, 25600, 51200, 102400, 204800, 409600, and 819200. The ISO value range should be as wide as possible and not limited by the aforementioned range.

[0117] The processing device 203 acquires multiple black frame, dark frame, and bright frame images captured by the camera at the aforementioned multiple sensitivities. The black frame image is obtained by photographing the 24-color chart in a lightless environment. Specifically, a group of images without light entering can be directly captured by covering the camera lens cap. For the dark frame images, first, one of the aforementioned multiple sensitivities is set in sequence, and then the camera aperture, exposure time, and shutter speed of the camera are adjusted. The 24-color chart is photographed at different camera apertures, exposure times, and shutter speeds to obtain multiple dark frame images. Then, another sensitivity is set and the aforementioned steps are repeated until all sensitivities have been photographed. For the bright frame images, similarly, one of the aforementioned multiple sensitivities is set in sequence, then the camera aperture of the camera is adjusted to the maximum, and then the exposure time is adjusted so that the brightness of the white color block of the 24-color chart is a specific percentage of the maximum brightness value. In this embodiment, the specific percentage is 80%, that is, the brightness of the white color block is 80% of the maximum brightness value, to avoid overexposure. Finally, based on the aforementioned set sensitivity and exposure time, the 24-color chart is photographed to obtain the bright frame images. Then, another sensitivity is set and the aforementioned steps are repeated until all sensitivities have been photographed. To ensure there are enough sample numbers, the total number of black frame, dark frame, and bright frame images in this embodiment is not less than 100.

[0118] After obtaining the data, enter the calibration procedure. In the calibration procedure, the processing device 203 substitutes the values of various prepared images into the corresponding gray-scale value model to obtain various parameters.

[0119] First, the processing device 203 substitutes the captured black frame image into the gray-scale value model to obtain the black frame parameters. Since the black frame image represents no light entering the lens, Equation XII can be simplified as follows:

[0120] ADU p = RNS - μ RNS

[0121] That is, the ADU value of the black frame image is only related to the read noise signal and the mean value of the read noise signal, where the mean value of the read noise signal reflects the effects of black level offset, dark current, and fixed pattern noise. Specifically, the processing device 203 averages the ADU values of a group of black frame images to obtain the black frame parameter μ RNS . Then, the ADU value of each black frame is subtracted by the black frame parameter to obtain an array that may be positive or negative, that is, the ADU of the black frame p . The processing device 203 analyzes the ADU p of all black frames and can obtain a Gaussian distribution. The processing device 2031 further calculates the variance of the obtained Gaussian distribution, which is the variance of the read noise

[0122] Next, the processing device 203 substitutes the dark frame, bright frame image, and black frame parameter into the gray value model to obtain the non-black frame parameter. There are two ways to calibrate the bright frame and dark frame images. The first way is to subtract the average of the black frame from the ADU value of each frame, and then calculate the mean and variance of 24 color blocks for each channel. More specifically, according to Equation 13 and Equation 14, the processing device 203 can obtain Equation 19 as follows

[0123] Var(ADU p ) = u2 × Eva(ADU p ) 2 + u1 × Eva(ADU p ) + u0

[0124] where

[0125]

[0126]

[0127] u0 = RN 2

[0128] According to u2, u1, and u0, it can be deduced that

[0129]

[0130]

[0131]

[0132] According to the foregoing derivation, the processing device 2031 can obtain the non-black frame parameter, and the non-black frame parameter includes the standard deviation of the noise signal (RN), the standard deviation of the photosensitive point (PRNR), and the pixel average gain (EM).

[0133] The second method is to calculate the variance of two adjacent bright-frame images, then calculate the mean and variance of 24 color blocks for each channel, and then use a linear regression of a linear function to calculate the semi-mean of the variance. More specifically, according to Equation (13) and Equation (18), the processing device 203 can obtain Equation (20) as follows:

[0134]

[0135] Wherein,

[0136]

[0137] z0 = RN 2

[0138] Based on the foregoing derivation, by using linear regression of a linear function, the processing device 2031 can also obtain the foregoing non-black frame parameters.

[0139] So far, the processing device 203 has obtained the black frame parameters and non-black frame parameters, and the black frame parameters and non-black frame parameters are stored in the DRAM 204 as the prior inputs for training the neural network model. At the beginning of the training, the computing device 201 retrieves the black frame parameters and non-black frame parameters from the DRAM 204, and uses the black frame parameters and non-black frame parameters as the prior inputs to substitute into the neural network model for training to optimize various parameters in the neural network model. After the training is completed, the neural network model has removed Figure 6 the possible noise of the camera.

[0140] When the board 10 or the combined processing device 20 uses the Figure 6 image data captured by the camera, the computing device 201 can use the trained neural network model to perform specific tasks related to image signal processing, such as image recognition, object detection, semantic segmentation, video understanding, image generation, denoising, demosaicing, etc., to obtain a denoised output.

[0141] Another embodiment of the present invention is a method for denoising a camera. This method can generate data for denoising in the Bayer domain, and these data are added as parameters to the training of the neural network model, so that the trained model directly produces a denoising effect on these cameras. Figure 7 The flowchart showing the embodiment of the present invention is shown.

[0142] In step 701, a grayscale value model of pixels is obtained. The grayscale value model of this embodiment is the same as that of the previous embodiment and is rearranged as follows:

[0143] ADU p = (EM + MD p ) × EL p + RNS - μ RNS

[0144] Eva(ADU p ) = EM × λ

[0145] Var(ADU p ) = PRNU 2 × λ 2 +(EM 2 +PRNU 2 ) × λ + RN 2

[0146]

[0147] In step 702, multiple black frames, dark frames, and bright frame images captured by the camera at multiple sensitivities are obtained. To obtain the noise model of the sensor, several sets of standard color cards need to be captured in a light box environment within a certain ISO gradient range. In this embodiment, the 24-color card is also used as the standard color card. The 24-color card is fixed in front of the camera and captured at different ISO values to obtain multiple black frames, dark frames, and bright frame images.

[0148] The black frame image is obtained by capturing the 24-color card in a lightless environment. Specifically, a set of images without light input can be directly captured by covering the camera lens cap. For the dark frame image, first, one of the aforementioned multiple sensitivities is set in sequence. Then, the camera aperture, exposure time, and shutter speed of the camera are adjusted, and the 24-color card is captured at different camera apertures, exposure times, and shutter speeds to obtain multiple dark frame images. After that, another sensitivity is set and the aforementioned steps are repeated until all sensitivities have been captured. For the bright frame image, first, one of the aforementioned multiple sensitivities is set in sequence. Then, the camera aperture of the camera is adjusted to the maximum, and then the exposure time is adjusted so that the brightness of the white color block of the 24-color card is a specific percentage of the maximum brightness value. In this embodiment, the specific percentage is 80%, that is, the brightness of the white color block is 80% of the maximum brightness value to avoid overexposure. Finally, based on the aforementioned set sensitivity and exposure time, the 24-color card is captured to obtain the bright frame image. After that, another sensitivity is set and the aforementioned steps are repeated until all sensitivities have been captured. To ensure there are enough sample numbers, the total number of black frames, dark frames, and bright frame images in this embodiment is not less than 100.

[0149] In step 703, the black frame image is substituted into the gray value model to obtain the black frame parameter. Since the black frame image represents that no light enters the lens, that is, the ADU value of the black frame image is only related to the read noise signal and the mean value of the read noise signal, where the mean value of the read noise signal reflects the effects of black level offset, dark current, and fixed pattern noise. Specifically, in this embodiment, the ADU values of a group of black frame images are averaged to obtain the black frame parameter μ RNS。Then subtract the black frame parameter from the ADU value of each black frame to obtain an array that may be positive or negative, i.e., the ADU of the black frame p 。In this step, analyze the ADU of all black frames p to obtain a Gaussian distribution, and further calculate the variance of the obtained Gaussian distribution, which is the variance of the read noise.

[0150] In step 704, substitute the dark frame, bright frame images, and black frame parameters into the grayscale value model to obtain non-black frame parameters. There are also 2 ways to calibrate the bright frame and dark frame images in this embodiment. The first way is to subtract the average of the black frames from the ADU value of each frame, and then calculate the mean and variance of 24 color blocks for each channel. According to Equation XIX, the standard deviation of the noise signal (RN), the standard deviation of the photosensitive points (PRNR), and the average pixel gain (EM) can be obtained as follows:

[0151]

[0152]

[0153]

[0154] The standard deviation of the noise signal (RN), the standard deviation of the photosensitive points (PRNR), and the average pixel gain (EM) are the non-black frame parameters.

[0155] The second way is to calculate the variance of two adjacent bright frame images, then calculate the mean and variance of 24 color blocks for each channel, and then use the semi-mean of the linear regression variance of a linear function to also obtain the aforementioned non-black frame parameters.

[0156] So far, this embodiment has obtained the black frame parameters and non-black frame parameters as the prior inputs for training the neural network model.

[0157] In step 705, use the black frame parameters and non-black frame parameters as prior inputs and substitute them into the neural network model for training. At the beginning of the training, this embodiment takes out the black frame parameters and non-black frame parameters, and substitutes the black frame parameters and non-black frame parameters into the neural network model as prior inputs for training to optimize various parameters in the neural network model. After the training is completed, the neural network model has removed the possible noise of the camera.

[0158] In step 706, input the image data captured by the camera into the trained neural network model for image signal processing to obtain a denoised output. When using the image data captured by the same camera, this embodiment uses the trained neural network model to perform specific tasks related to image signal processing, such as image recognition, object detection, semantic segmentation, video understanding, image generation, denoising, demosaicing, etc., in order to obtain a denoised output.

[0159] Another embodiment of the present invention is a computer-readable storage medium, on which computer program code for denoising a camera is stored. When the computer program code is run by a processor, the methods of the foregoing embodiments are executed. In some implementation scenarios, the above integrated unit may be implemented in the form of a software program module. If it is implemented in the form of a software program module and sold or used as an independent product, the integrated unit may be stored in a computer-readable memory. Based on this, when the solution of the present invention is embodied in the form of a software product (such as a computer-readable storage medium), the software product may be stored in a memory, which may include several instructions for causing a computer device (such as a personal computer, a server, or a network device, etc.) to execute some or all of the steps of the method described in the embodiments of the present invention. The foregoing memory may include, but is not limited to, various media capable of storing program code such as a USB flash drive, a flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disc.

[0160] By substituting the black frame parameters and non-black frame parameters as prior inputs into the neural network model for training, the present invention can more accurately calibrate the noise of the camera and optimize the denoising effect of image signal processing.

[0161] According to different application scenarios, the electronic device or apparatus of the present invention may include a server, a cloud server, a server cluster, a data processing device, a robot, a computer, a printer, a scanner, a tablet computer, a smart terminal, a PC device, an Internet of Things terminal, a mobile terminal, a mobile phone, a driving recorder, a navigator, a sensor, a camera, a camera, a video camera, a projector, a watch, headphones, a mobile storage device, a wearable device, a vision terminal, an autonomous driving terminal, a vehicle, a household appliance, and / or a medical device. The vehicle includes an airplane, a ship, and / or a vehicle; the household appliance includes a television, an air conditioner, a microwave oven, a refrigerator, a rice cooker, a humidifier, a washing machine, an electric light, a gas stove, an oil fume extractor; the medical device includes a nuclear magnetic resonance instrument, a B-ultrasound instrument, and / or an electrocardiogram instrument. The electronic device or apparatus of the present invention can also be applied to fields such as the Internet, the Internet of Things, a data center, energy, transportation, public management, manufacturing, education, a power grid, telecommunications, finance, retail, a construction site, and medicine. Further, the electronic device or apparatus of the present invention can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing such as the cloud, the edge, and the terminal. In one or more embodiments, the electronic device or apparatus with high computing power according to the solution of the present invention can be applied to a cloud device (such as a cloud server), while the electronic device or apparatus with low power consumption can be applied to a terminal device and / or an edge device (such as a smart phone or a camera). In one or more embodiments, the hardware information of the cloud device is compatible with the hardware information of the terminal device and / or the edge device, so that appropriate hardware resources can be matched from the hardware resources of the cloud device according to the hardware information of the terminal device and / or the edge device to simulate the hardware resources of the terminal device and / or the edge device, so as to complete the unified management, scheduling, and collaborative work of the terminal-cloud integration or the cloud-edge-terminal integration.

[0162] It should be noted that, for the purpose of simplicity, some methods and their embodiments of the present invention are expressed as a series of actions and their combinations. However, those skilled in the art can understand that the solution of the present invention is not limited by the order of the described actions. Therefore, according to the disclosure or teaching of the present invention, those skilled in the art can understand that some of the steps can be executed in other orders or simultaneously. Further, those skilled in the art can understand that the embodiments described in the present invention can be regarded as optional embodiments, that is, the actions or modules involved are not necessarily required for the implementation of a certain or some solutions of the present invention. In addition, according to different solutions, the present invention focuses on the descriptions of some embodiments. In view of this, those skilled in the art can understand that the parts not detailed in a certain embodiment of the present invention can also refer to the relevant descriptions of other embodiments.

[0163] In terms of specific implementation, based on the disclosure and teachings of the present invention, those skilled in the art can understand that several embodiments disclosed in the present invention can also be implemented in other ways not disclosed herein. For example, with respect to the various units in the electronic device or device embodiments described above, this article splits them based on the consideration of logical functions, and there may be other ways of splitting them in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features or functions in the units or components can be selectively disabled. With respect to the connection relationship between different units or components, the connection discussed above in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0164] In the present invention, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network elements. In addition, according to actual needs, some or all of the units may be selected to achieve the purpose of the solutions described in the embodiments of the present invention. In addition, in some scenarios, multiple units in the embodiments of the present invention may be integrated into a single unit or each unit may exist physically separately.

[0165] In some other implementation scenarios, the above integrated units can also be implemented in the form of hardware, i.e., a specific hardware circuit, which can include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit can include, but is not limited to, physical devices, and the physical devices can include, but are not limited to, devices such as transistors or memristors. In view of this, various devices described in this article (such as computing devices or other processing devices) can be implemented by appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs, etc. Further, the aforementioned storage unit or storage device can be any appropriate storage medium (including magnetic storage media or magneto-optical storage media, etc.), which can be, for example, Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), ROM, and RAM, etc.

[0166] The foregoing can be better understood in accordance with the following terms:

[0167] Clause A1. A method for denoising a camera, comprising: obtaining a grayscale value model of pixels; obtaining multiple black frame, dark frame, and bright frame images captured by the camera at multiple sensitivities; substituting the black frame image into the grayscale value model to obtain black frame parameters; substituting the dark frame, bright frame images, and the black frame parameters into the grayscale value model to obtain non-black frame parameters; using the black frame parameters and non-black frame parameters as prior inputs to train a neural network model; and inputting the image data captured by the camera into the trained neural network model for image signal processing to obtain a denoised output.

[0168] Clause A2. The method according to Clause A1, wherein the total number of the black frame, dark frame, and bright frame images is not less than 100.

[0169] Clause A3. The method according to Clause A1, wherein the black frame, dark frame, and bright frame images are from capturing a standard color card.

[0170] Clause A4. The method according to Clause A3, wherein the step of obtaining multiple black frames, dark frames and bright frames captured by the camera at multiple sensitivities includes: sequentially setting one of the multiple sensitivities and performing the following steps: adjusting the camera aperture of the camera to the maximum; adjusting the exposure time so that the brightness of the white color block of the standard color card is a specific percentage of the maximum brightness value; and based on the sensitivity and the exposure time, capturing the standard color card to obtain a bright frame image.

[0171] Clause A5. The method according to claim 4, wherein the specific percentage is 80%.

[0172] Clause A6. The method according to Clause A3, wherein the step of obtaining multiple black frames, dark frames and bright frames captured by the camera at multiple sensitivities includes: sequentially setting one of the multiple sensitivities and performing the following steps: adjusting the camera aperture, exposure time and shutter speed of the camera, and capturing the standard color card at different camera apertures, exposure times and shutter speeds to obtain the dark frame image.

[0173] Clause A7. The method according to Clause A3, wherein the capturing step includes: capturing the standard color card in a lightless environment to obtain the black frame image.

[0174] Clause A8. The method according to Clause A1, wherein the black frame parameter includes the mean value of the read noise signal.

[0175] Clause A9. The method according to Clause A8, wherein the mean value of the read noise signal reflects the effects of black level offset, dark current and fixed pattern noise.

[0176] Clause A10. The method according to Clause A1, wherein the non-black frame parameters include the standard deviation of the noise signal, the standard deviation of the photosensitive points and the pixel average gain.

[0177] Clause A11. A computer-readable storage medium, on which computer program code for denoising a camera is stored, and when the computer program code is run by a processing device, the method according to any one of Clauses A1 to A10 is executed.

[0178] Clause A12. An integrated circuit device for denoising a camera, comprising: a processing device configured to: obtain a grayscale value model of pixels; obtain multiple black frames, dark frames, and bright frame images captured by the camera at multiple sensitivities; substitute the black frame images into the grayscale value model to obtain black frame parameters; and substitute the dark frame and bright frame images and the black frame parameters into the grayscale value model to obtain non-black frame parameters; a calculation device configured to: use the black frame parameters and non-black frame parameters as prior inputs to be substituted into a neural network model for training; and input the image data captured by the camera into the trained neural network model for image signal processing to obtain a denoised output.

[0179] Clause A13. The integrated circuit device according to Clause A12, wherein the total number of the black frames, dark frames, and bright frame images is not less than 100.

[0180] Clause A14. The integrated circuit device according to Clause A12, wherein the black frames, dark frames, and bright frame images are from capturing a standard color card.

[0181] Clause A15. The integrated circuit device according to Clause A14, wherein the bright frame image is obtained by performing the following steps: sequentially setting one of the multiple sensitivities, and performing the following steps: adjusting the camera aperture of the camera to the maximum; adjusting the exposure time so that the brightness of the white color block of the standard color card is a specific percentage of the maximum brightness value; and based on the sensitivity and exposure time, capturing the standard color card to obtain the bright frame image.

[0182] Clause A16. The integrated circuit device according to claim 15, wherein the specific percentage is 80%.

[0183] Clause A17. The integrated circuit device according to Clause A14, wherein the dark frame image is obtained by performing the following steps: sequentially setting one of the multiple sensitivities, and performing the following steps: adjusting the camera aperture, exposure time, and shutter speed of the camera, and capturing the standard color card at different camera apertures, exposure times, and shutter speeds to obtain the dark frame image.

[0184] Clause A18. The integrated circuit device according to Clause A14, wherein the black frame image is obtained by performing the following steps: capturing the standard color card in a lightless environment to obtain the black frame image.

[0185] Clause A19. The integrated circuit device according to Clause A12, wherein the black frame parameters include the mean value of the read noise signal.

[0186] Clause A20. The integrated circuit device according to Clause A19, wherein the mean value of the read noise signal reflects the effects of black level bias, dark current, and fixed pattern noise.

[0187] Clause A21. The integrated circuit device according to Clause A12, wherein the non-black frame parameters include the standard deviation of the noise signal, the standard deviation of the photosensitive points, and the pixel average gain.

[0188] Clause A22. A board card comprising the integrated circuit device according to any one of Clauses A12 to 21.

[0189] The embodiments of the present invention have been introduced in detail above. Specific examples are used herein to elaborate on the principles and implementation manners of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A method for denoising a camera, comprising: Obtaining a grayscale value model of pixels, wherein the grayscale value model models various sources of camera noise, and the camera is connected to an analog-to-digital converter, and the grayscale value model is related to the ADU value output by the analog-to-digital converter; Obtaining multiple black frame, dark frame, and bright frame images captured by the camera at multiple sensitivities; Substituting the black frame image into the grayscale value model to obtain black frame parameters, wherein the black frame parameters include the mean value of the read noise signal, and the mean value of the read noise signal reflects the effects of black level offset, dark current, and fixed pattern noise; Substituting the dark frame, bright frame images, and the black frame parameters into the grayscale value model to obtain non-black frame parameters, wherein the non-black frame parameters include the standard deviation of the noise signal, the standard deviation of the photosensitive points, and the pixel average gain; Using the black frame parameters and non-black frame parameters as prior inputs to train a neural network model; And Inputting the image data captured by the camera into the trained neural network model for image signal processing to obtain a denoised output.

2. The method according to claim 1, wherein the total number of the black frame, dark frame, and bright frame images is not less than 100.

3. The method according to claim 1, wherein the black frame, dark frame, and bright frame images are from photographing a standard color card.

4. The method according to claim 3, wherein the step of obtaining multiple black frame, dark frame, and bright frame images captured by the camera at multiple sensitivities comprises: Sequentially setting one of the multiple sensitivities and performing the following steps: Adjusting the camera aperture of the camera to the maximum; Adjusting the exposure time so that the brightness of the white color block of the standard color card is a specific percentage of the maximum brightness value; And Based on the sensitivity and exposure time, photographing the standard color card to obtain a bright frame image.

5. The method according to claim 4, wherein the specific percentage is 80%.

6. The method according to claim 3, wherein the step of obtaining multiple black frame, dark frame, and bright frame images captured by the camera at multiple sensitivities comprises: Sequentially setting one of the multiple sensitivities and performing the following steps: Adjusting the camera aperture, exposure time, and shutter speed of the camera, and photographing the standard color card under different camera apertures, exposure times, and shutter speeds to obtain the dark frame image.

7. The method according to claim 3, wherein the photographing step comprises: Photographing the standard color card in a lightless environment to obtain the black frame image.

8. A computer-readable storage medium having stored thereon computer program code for denoising a camera, which, when run by a processing device, executes the method according to any one of claims 1 to 7.

9. An integrated circuit device for denoising a camera, comprising: A processing device for: Obtaining a grayscale value model of pixels, wherein the grayscale value model models various sources of camera noise, and the camera is connected to an analog-to-digital converter, and the grayscale value model is related to the ADU value output by the analog-to-digital converter; Obtain multiple black frames, dark frames, and bright frame images captured by the camera at multiple sensitivities; Substitute the black frame images into the grayscale value model to obtain black frame parameters, where the black frame parameters include the mean value of the read noise signal, and the mean value of the read noise signal reflects the effects of black level offset, dark current, and fixed pattern noise; and Substitute the dark frame and bright frame images and the black frame parameters into the grayscale value model to obtain non-black frame parameters, where the non-black frame parameters include the standard deviation of the noise signal, the standard deviation of the photosensitive points, and the pixel average gain; A computing device for: Use the black frame parameters and non-black frame parameters as prior inputs and substitute them into the neural network model for training; And Input the image data captured by the camera into the trained neural network model for image signal processing to obtain a denoised output.

10. The integrated circuit device according to claim 9, wherein the total number of the black frames, dark frames, and bright frame images is not less than 100.

11. The integrated circuit device according to claim 9, wherein the black frames, dark frames, and bright frame images are from capturing a standard color card.

12. The integrated circuit device according to claim 11, wherein the bright frame image is obtained by performing the following steps: Sequentially set one of the multiple sensitivities and perform the following steps: Adjust the camera aperture of the camera to the maximum; Adjust the exposure time so that the brightness of the white color block of the standard color card is a specific percentage of the maximum brightness value; and Based on the sensitivity and exposure time, capture the standard color card to obtain the bright frame image.

13. The integrated circuit device according to claim 12, wherein the specific percentage is 80%.

14. The integrated circuit device according to claim 11, wherein the dark frame image is obtained by performing the following steps including: Sequentially set one of the multiple sensitivities and perform the following steps: Adjust the camera aperture, exposure time, and shutter speed of the camera, and capture the standard color card at different camera apertures, exposure times, and shutter speeds to obtain the dark frame image.

15. The integrated circuit device according to claim 11, wherein the black frame image is obtained by performing the following steps including: Capture the standard color card in a lightless environment to obtain the black frame image.

16. A board card, comprising the integrated circuit device according to any one of claims 9 to 15.

Images