Integrated circuit device for camera denoising based on foreground data

Abstract

The present invention relates to an integrated circuit device for denoising a camera based on foreground data, wherein a processing device obtains the foreground data of a front image, sets a reference exposure value according to the foreground data, and then sets the exposure values ​​of a dark frame image and a bright frame image according to the reference exposure value, and substitutes the black frame image, the dark frame image and the bright frame image into a grayscale value model of a pixel to obtain black frame parameters and non-black frame parameters. A computing device substitutes the black frame parameters and non-black frame parameters as prior inputs into a neural network model for training to obtain a denoised output.

Description

Technical Field

[0001] The present invention generally relates to the field of image signal processing and more particularly to an integrated circuit device for denoising a camera based on foreground data. Background Art

[0002] Cameras and image sensors are highly mature products. The market is flooded with cameras with different optical characteristics and sensors with different electrical characteristics, which makes it impossible for image signal processors (ISPs) to automatically compensate for these hardware characteristics, resulting in image processing effects that are not as good as expected. To solve this problem, camera and image sensor suppliers provide a set of calibration tools that use images obtained from existing instruments and equipment as inputs to demonstrate adjustment procedures so that the output can better approximate the expected effect. Most existing technologies use the Bayer denoising algorithm, which uses deep learning technology to achieve better correction effects.

[0003] When performing black level correction, the average value of a group of black frames is first calibrated as a reference value, and this reference value is subtracted in the subsequent algorithm to remove background noise. However, in the case of black frames, the pixels actually obey the Gaussian distribution, and factors such as photo response non-uniformity (PRNU), dark current, and fixed pattern noise will have a certain impact, so the accuracy of black level correction is difficult to guarantee.

[0004] When performing noise calibration, since the existing denoising algorithm uses a set of filters, the output of the noise calibration is only the gain size or the dark frame variance as the filtering strength, and does not take into account a more accurate model of the noise and the distribution of the noise, resulting in inaccurate noise calibration and naturally unsatisfactory denoising effect.

[0005] Therefore, a scheme with good denoising effect is urgently needed. Summary of the invention

[0006] In order to at least partially solve the technical problem mentioned in the background technology, the solution of the present invention provides an integrated circuit device for camera denoising based on foreground data.

[0007] In one aspect, the present invention discloses an integrated circuit device for denoising a camera based on foreground data, comprising a processing device and a computing device.

[0008] The processing device is used to: obtain the foreground data of the front image; set a reference exposure according to the foreground data; set the exposure of the dark frame and the light frame according to the reference exposure value; obtain multiple black frame images taken by the camera; when the sensitivity is less than or equal to the reference exposure, obtain multiple dark frame images taken by the camera at multiple sensitivities; when the sensitivity is greater than the reference exposure, sequentially obtain multiple light frame images taken by the camera at multiple sensitivities; substitute the black frame, dark frame and light frame images into the gray value model of the pixel to obtain black frame parameters and non-black frame parameters.

[0009] The bright frame image is obtained by executing the following steps: adjusting the lens aperture of the camera to the maximum; adjusting the exposure time so that the brightness of the white block of the standard color card is a specific percentage of the maximum brightness; and photographing the standard color card to obtain the bright frame image.

[0010] The computing device is used to: substitute the black frame parameters and non-black frame parameters as prior inputs into a neural network model for training; and input the image data captured by the camera into the trained neural network model for image signal processing to obtain a denoised output.

[0011] The present invention substitutes black frame parameters and non-black frame parameters as priori inputs into a neural network model for training, thereby more accurately calibrating noise and optimizing the denoising effect of image signal processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0013] Figure 1 is a structural diagram of a board card according to an embodiment of the present invention;

[0014] Figure 2 is a structural diagram showing an integrated circuit device according to an embodiment of the present invention;

[0015] Figure 3 is a schematic diagram showing the internal structure of a computing device according to an embodiment of the present invention;

[0016] Figure 4 is a schematic diagram showing the internal structure of a processor core according to an embodiment of the present invention; and

[0017] Figure 5 is a schematic diagram showing how a single pixel of a camera according to an embodiment of the present invention converts photons into analog-to-digital units (ADUs). DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0019] It should be understood that the terms "first", "second", "third" and "fourth" etc. in the claims, specifications and drawings of the present invention are used to distinguish different objects rather than to describe a specific order. The terms "include" and "comprise" used in the specification and claims of the present invention indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections.

[0020] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the claims, the singular forms of "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in the specification of the present invention and the claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0021] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context.

[0022] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0023] The present invention proposes a denoising scheme in a neural network model, which is originally used to perform specific tasks related to image signal processing, such as image recognition, target detection, semantic segmentation, video understanding, image generation, denoising, demosaicing, etc. The present invention substitutes black frame parameters and non-black frame parameters as prior inputs into the neural network model for training, so that the neural network model can suppress the influence of noise on image output by updating parameters during the training process.

[0024] An embodiment of the present invention is a network architecture for realizing computer vision with a camera, wherein the network architecture has the following features: Figure 1 The board 10 shown. Figure 1As shown, the board 10 includes a chip 101, which is a system-on-chip (SoC), or a system on a chip, integrated with one or more combined processing devices, which are an artificial intelligence computing unit to support various deep learning and machine learning algorithms to meet the intelligent processing needs in complex scenarios in the field of computer vision. In particular, deep learning technology is widely used in the field of cloud intelligence. A significant feature of cloud intelligence applications is the large amount of input data, which places high demands on the storage capacity and computing power of the platform. The board 10 of this embodiment is suitable for cloud intelligence applications and has huge off-chip storage, on-chip storage and powerful computing power.

[0025] The chip 101 is connected to the external device 103 via the external interface device 102. In this embodiment, the external device 103 is a camera. The image data to be processed can be transmitted from the external device 103 to the chip 101 via the external interface device 102. According to different application scenarios, the external interface device 102 can have different interface forms, such as a PCIe interface.

[0026] The board 10 also includes a storage device 104 for storing data, which includes one or more storage units 105. The storage device 104 is connected to the control device 106 and the chip 101 through a bus and transmits data. The control device 106 in the board 10 is configured to control the state of the chip 101. To this end, in an application scenario, the control device 106 may include a microcontroller (MCU).

[0027] Figure 2 1 is a block diagram showing the combined processing device in the chip 101 of this embodiment. Figure 2 As shown in , the combined processing device 20 includes a computing device 201 , an interface device 202 , a processing device 203 and a DRAM 204 .

[0028] The computing device 201 is configured to execute user-specified operations, and is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete computer vision operations.

[0029] The interface device 202 is used to transmit data and control instructions between the computing device 201 and the processing device 203. For example, the computing device 201 can obtain input data from the processing device 203 via the interface device 202 and write it into the storage device on the computing device 201 chip. Further, the computing device 201 can obtain control instructions from the processing device 203 via the interface device 202 and write them into the control cache on the computing device 201 chip. Alternatively or optionally, the interface device 202 can also read data in the storage device of the computing device 201 and transmit it to the processing device 203.

[0030] The processing device 203 is a general processing device that performs basic controls including but not limited to data handling, starting and / or stopping the computing device 201. Depending on the implementation, the processing device 203 can be a central processing unit (CPU), a graphics processing unit (GPU) or one or more types of processors in other general and / or special processors, which include but are not limited to digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, only with respect to the computing device 201 of the present invention, it can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 201 and the processing device 203 are integrated and considered together, the two are regarded as forming a heterogeneous multi-core structure.

[0031] DRAM 204 is used to store data to be processed. It is a DDR memory, usually 16G or larger in size, and is used to save data of the computing device 201 and / or the processing device 203, including a training sample set for training a neural network model.

[0032] Figure 3 The figure shows a schematic diagram of the internal structure of the computing device 201. The computing device 201 is used to process the input data of computer vision. The computing device 201 in the figure adopts a multi-core hierarchical structure design. The computing device 201 is a system on chip, which includes multiple clusters, and each cluster includes multiple processor cores. In other words, the computing device 201 is composed of a hierarchy of system on chip-cluster-processor core.

[0033] At the system-on-chip level, Figure 3 As shown, the computing device 201 includes an external storage controller 301 , a peripheral communication module 302 , an on-chip interconnect module 303 , a synchronization module 304 , and multiple clusters 305 .

[0034] There may be multiple external storage controllers 301, two of which are shown in the figure as an example, and are used to respond to access requests issued by the processor core and access external storage devices, such as Figure 2 DRAM 204 in the chip, thereby reading image data from outside the chip or writing data. The peripheral communication module 302 is used to receive control signals from the processing device 203 through the interface device 202 to start the computing device 201 to perform tasks. The on-chip interconnect module 303 connects the external storage controller 301, the peripheral communication module 302 and multiple clusters 305 to transmit data and control signals between each module. The synchronization module 304 is a global synchronization barrier controller (GBC) used to coordinate the work progress of each cluster and ensure information synchronization. Multiple clusters 305 are the computing cores of the computing device 201. Four are shown as examples in the figure. With the development of hardware, the computing device 201 of the present invention can also include 8, 16, 64, or even more clusters 305. Clusters 305 are used to efficiently execute deep learning algorithms.

[0035] At the cluster level, Figure 3 As shown, each cluster 305 includes a plurality of processor cores (IPU cores) 306 and a memory core (MEM core) 307 .

[0036] The figure shows four processor cores 306 by way of example, but the present invention does not limit the number of processor cores 306. Figure 4 Each processor core 306 includes three modules: a control module 41 , a computing module 42 and a storage module 43 .

[0037] The control module 41 is used to coordinate and control the operation of the computing module 42 and the storage module 43 to complete the deep learning task, and includes an instruction fetch unit (IFU) 411 and an instruction decode unit (IDU) 412. The instruction fetch unit 411 is used to obtain instructions from the processing device 203, and the instruction decode unit 412 decodes the obtained instructions and sends the decoding results to the computing module 42 and the storage module 43 as control information.

[0038] The operation module 42 includes a vector operation unit 421 and a matrix operation unit 422. The vector operation unit 421 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformation; the matrix operation unit 422 is responsible for the core calculation of the deep learning algorithm, namely matrix multiplication and convolution.

[0039] The storage module 43 is used to store or move related data, including a neuron RAM (NRAM) 431, a weight RAM (WRAM) 432, an input / output direct memory access module (IODMA) 433, and a move direct memory access module (MVDMA) 434. NRAM 431 is used to store feature maps and intermediate results calculated by the processor core 306; WRAM 432 is used to store the weights of the deep learning network; IODMA 433 controls the memory access between NRAM 431 / WRAM 432 and DRAM 204 through the broadcast bus 309; MVDMA 434 is used to control the memory access between NRAM 431 / WRAM 432 and SRAM 308.

[0040] Back to Figure 3 The storage core 307 is mainly used for storage and communication, that is, to store shared data or intermediate results between the processor cores 306, and to perform communication between the cluster 305 and the DRAM 204, communication between the clusters 305, and communication between the processor cores 306. In other embodiments, the storage core 307 has the capability of scalar operation and is used to perform scalar operation.

[0041] The storage core 307 includes a shared memory unit (SRAM) 308, a broadcast bus 309, a cluster direct memory access module (cluster direct memory access, CDMA) 310 and a global direct memory access module (global direct memory access, GDMA) 311. The SRAM 308 plays the role of a high-performance data transfer station. The data reused between different processor cores 306 in the same cluster 305 does not need to be obtained from the DRAM 204 by each processor core 306, but is transferred between the processor cores 306 via the SRAM 308. The storage core 307 only needs to quickly distribute the reused data from the SRAM 308 to multiple processor cores 306, so as to improve the efficiency of inter-core communication and greatly reduce on-chip and off-chip input / output access.

[0042] The broadcast bus 309, CDMA 310 and GDMA 311 are respectively used to perform communication between processor cores 306, communication between clusters 305 and data transmission between clusters 305 and DRAM 204. They will be described below.

[0043] The broadcast bus 309 is used to complete high-speed communication between the processor cores 306 in the cluster 305. The broadcast bus 309 of this embodiment supports inter-core communication modes including unicast, multicast and broadcast. Unicast refers to point-to-point (i.e., single processor core to single processor core) data transmission, multicast is a communication mode of transmitting a copy of data from SRAM 308 to specific processor cores 306, and broadcast is a communication mode of transmitting a copy of data from SRAM 308 to all processor cores 306, which is a special case of multicast.

[0044] The CDMA 310 is used to control the memory access of the SRAM 308 between different clusters 305 in the same computing device 201 .

[0045] GDMA 311 cooperates with external memory controller 301 to control the access of SRAM 308 of cluster 305 to DRAM 204, or reads data from DRAM 204 to SRAM 308. As can be seen from the above, the communication between DRAM 204 and NRAM 431 or WRAM 432 can be realized through two channels. The first channel is to directly connect DRAM 204 and NRAM 431 or WRAM 432 through IODAM 433; the second channel is to first transmit data between DRAM 204 and SRAM 308 through GDMA 311, and then transmit data between SRAM 308 and NRAM 431 or WRAM 432 through MVDMA 434. Although it seems that the second channel requires more components to participate and the data flow is longer, in fact, in some embodiments, the bandwidth of the second channel is much greater than that of the first channel, so the communication between DRAM 204 and NRAM 431 or WRAM 432 may be more efficient through the second channel. The embodiment of the present invention can select a data transmission channel according to its own hardware conditions.

[0046] In other embodiments, the functions of GDMA 311 and IODMA 433 can be integrated into the same component. For the convenience of description, the present invention regards GDMA 311 and IODMA 433 as different components. For those skilled in the art, as long as the functions implemented and the technical effects achieved are similar to those of the present invention, they belong to the protection scope of the present invention. Furthermore, the functions of GDMA 311, IODMA 433, CDMA 310, and MVDMA 434 can also be implemented by the same component.

[0047] This embodiment models various sources of camera noise and proposes a set of noise calibration methods that can accurately describe any camera (including internal sensors) to generate Bayer domain denoised data. These data are added as parameters to the training of the neural network model, so that the trained model can directly produce a denoising effect on these cameras.

[0048] The camera of this embodiment is connected to an analog-to-digital converter, which is used to convert image signals (analog signals) into RAW image signals (digital signals). RAW images are raw data that the camera converts the captured light signals into digital signals. RAW files are a type of file that records the original information of the camera, including some metadata generated by the camera, such as ISO settings, shutter speed, aperture value, white balance, etc.

[0049] Each photosite of each camera is only sensitive to one primary color RGB (red, green, blue) component. Figure 5 A schematic diagram showing a single pixel of a camera converting photons into analog-to-digital units (ADUs). The camera includes a color filter 501 and multiple pixels 502. The color filter 501 is composed of filters of three primary colors, RGB. Each primary color filter only allows light of that color to pass through to the pixel 502. For example, a red filter only allows red light to pass through to the pixel 502. Each pixel 502 includes three sensors 503, i.e., three photosensing points, and each sensor 503 corresponds to a primary color filter.

[0050] When the object's light 504 enters the camera, the color filter 501 filters out light of other wavelengths, as shown in the figure. For example, the green filter only allows the green light wave in the object's light 504 to pass through and enter the sensor 503. The light irradiated on the sensor 503 during exposure can be described by photons. When the photons reach a specific layer on the sensor 503, each photon will excite an electron. The unit of the number of electrons is "e-", that is, the number of electrons excited by the sensor 503 reflects the number of photons reaching the sensor 503. After the exposure time is over, all the excited electrons are received by the analog-to-digital converter 505, which calculates the number of electrons and converts it into an ADU value.

[0051] In the aforementioned electron quantization process, the analog-to-digital converter 505 uses the conversion factor "M" (from the internal multiplier) to multiply the number of electrons in units of "ADU / e-" to obtain the corresponding ADU value. The size of "M" depends only on the ISO setting of the camera. The higher the ISO value, the higher the "M".

[0052] The photon flux that hits each photosite on sensor 503 is proportional to the intensity of the light. A darker scene will produce fewer electrons, resulting in a lower ADU value, while a brighter area will produce a higher ADU value. The model of sensor 503 can be expressed by the following equation 1:

[0053] ADU=M×EL

[0054] Here, EL represents the number of electrons in each image.

[0055] However, even under completely uniform illumination, the number of photons received by each photosite is not exactly the same. The number of photons is a random variable rather than a constant, and the different values ​​of photons produced under completely uniform illumination is called photon shot noise. Photon shot noise is a random variable that follows a Poisson distribution, and a Poisson process is described as "a given number of discrete events occurring in a fixed time and / or space, which occur at a known average rate, independent of the time since the last event." In the above context, the discrete event refers to the moment when a photon produces an electron, which corresponds to the exposure time of the surface of each photosite "at a fixed time and space", and its rate is actually independent of any photons that arrived before.

[0056] Since the photon shot noise follows a Poisson distribution, the expected value and variance of the ADU value can be expressed using the following equations 2 and 3:

[0057] Eva(ADU)=Eva(M×EL)

[0058] =Eva(M)×Eva(EL)

[0059] =M×Eva(EL)

[0060] =M×λ

[0061] Var(ADU)=Var(M×EL)

[0062] =M 2 ×Var(EL)

[0063] =M×Eva(EL)

[0064] =M 2 ×λ

[0065] =M×Eva(ADU)

[0066] Where Eva(ADU) is the expected value or average of the ADU values, Var(ADU) is the variance of the ADU values, and λ is the average number of electrons in each image.

[0067] exist Figure 5In the sensor 503, each photosensitizing point converts the charge of the electrons generated by the impact of photons into a voltage and transmits it to the analog-to-digital converter 505. In fact, the output of the analog-to-digital converter 505 is not completely proportional to the number of electrons emitted by the photons. This is because when the camera is covered and shooting (i.e., black frame image), in order to retain the dark area signal, a black level signal output is generated in the analog-to-digital converter 505. Therefore, even in a completely black state, the output of the analog-to-digital converter 505 always has a deviation or baseline. This noise is called read noise.

[0068] Adding read noise to the model, we get Equation 4 as follows:

[0069] ADU=M×EL+RNS

[0070] Where RNS is the read noise. Substituting equation 4 into equation 3, we can get equation 5 as follows:

[0071] Eva(ADU)=M×Eva(EL)+Eva(RNS)

[0072] =M×λ+Eva(RNS)

[0073] =M×λ+μ RNS

[0074] Among them, μ RNS is the mean of the read noise.

[0075] It can be seen that in an image with read noise, the average pixel value is the light signal plus the read noise. RNS , the signal of the RAW image can be restored, which is the so-called black level correction. The correction signal can be expressed by equation 6 and equation 7:

[0076] ADU RNC =M×EL+RNS-μ RNS

[0077] Eva(ADU RNC )=M×λ+Eva(RNS)-μ RNS

[0078] =M×λ+μ RNS -μ RNS

[0079] =M×λ

[0080] Among them, ADU RNCIndicates the corrected ADU value. After the read noise is corrected, the read noise can be subtracted from the ADU value by its average value. Although it cannot be completely removed, the influence of the read noise can be basically ignored. In other words, the read noise consists of two parts. One part is a fixed black level, that is, the read noise mean value μ RNS , and the other part is randomly floating and normally distributed. Formula 6 removes the reading noise mean μ RNS , that is, the corrected ADU value has eliminated the influence of the mean reading noise. To simplify the symbols, the ADU values ​​in the following formulas and descriptions are all corrected ADU values ​​ADU RNC .

[0081] By taking the variance of formula 6, we can get formula 8 as follows:

[0082] Var(ADU)=Var(M×EL)+Var(RNS)

[0083] =M 2 ×λ+RN 2

[0084] Where RN is the standard deviation of the noise signal.

[0085] In fact, due to the differences in the manufacturing process of the microlens, analog-to-digital converter and other sensor circuit components on each photosite, the sensitivity of each photosite to light is not exactly the same, so the M term in the above model is not exactly the same value for each image. On the contrary, the M value is a property of each pixel, which is called pixel response non-uniformity (PRNU). Taking PRNU into account, this embodiment replaces the factor "M" with "M p ", the subscript p refers to the pth pixel. Equation 6 is further adjusted to Equation 9 as follows:

[0086] ADU p =M p ×EL+RNS-μ RNS

[0087] Each pixel has the same M p value, which means M p For each image, the time to maintain the same ISO speed (the sensitivity of the photosensitive film changes with the change of ISO speed) is a constant, and the average value of all Mp factors corresponding to all light spots on the surface of the sensor 503 is also a constant, which is called the expected multiplier EM, that is, the average gain of each pixel of the camera. EM can be expressed by formula 10:

[0088] EM=Eva(M p )

[0089] Further, M p The value is considered to be composed of two components, one is the expected multiplier EM, and the other is the photosensitivity deviation MD relative to the expected multiplier EM. p , that is, the difference between the gain of the pth pixel and the average gain, then M p It can be expressed by formula 11:

[0090] M p =EM+MD p

[0091] Substituting equation 11 into equation 9, we can get equation 12 as follows:

[0092] ADU p =(EM+MD p )×EL p +RNS-μ RNS

[0093] Among them, ADU p is the RAW image value of the pth pixel, EL p is the number of photons received by the p-th pixel.

[0094] By taking the mean of equation 12, we get equation 13 as follows:

[0095] Eva(ADU p )=Eva((EM+MD p )×EL p )+Eva(RNS)-μ RNS

[0096] =Eva((EM+MD p ))×Eva(EL p )+Eva(RNS)-μ RNS

[0097] =Eva(EM)×λ+μ RNS -μ RNS

[0098] =EM×λ

[0099] Calculating the variance of equation 12, we get equation 14 as follows:

[0100] Var(ADU p )=Var((EM+MD p )×EL p )+Var(RNS)

[0101] =PRNU 2 ×λ 2 +(EM 2+PRNU 2 )×λ+RN 2

[0102] Equation 14 is derived from the sensor model, which explains the distribution of various noises, namely ADU p The variance of is a quadratic polynomial of λ. From the above formula, we can see that PRNU is a very strong noise source.

[0103] Next, we further separate the PRNU. For a given pixel, the PRNU will not change in different images, that is, the average value of the image will highlight the PRNU noise. Using equation 12 to calculate the average value, we can get equation 15 as follows:

[0104]

[0105] Where n represents the average number of images. From Formula 15, it can be seen that except for PRNU noise, other types of noise are reduced after averaging. Further calculating the variance of both ends of Formula 15, we get Formula 16 as follows:

[0106]

[0107] In the above equation, it can be seen that the PRNU quadratic term is not affected by averaging, while the influence of other types of noise decreases as the number of averaged images n increases. p caused by, while for the same camera, MD p If the value remains unchanged, the influence of PRNU can be eliminated by comparing the differences of multiple images taken by the same camera. From formula 12, formula 17 can be obtained as follows:

[0108] ΔADU p =(EM+MD p )×ΔEL p +ΔRNS

[0109] Here, “Δ” represents the difference in pixel values ​​between two images taken by the same camera and the same pixel in the same scene. p It is the characteristic of the photosite, which will not change during the shooting process, so when calculating the variance of equation 17, MD can be excluded. p The influence of is shown in formula 18:

[0110]

[0111] Compared with Equation 14, Equation 18 does not include the quadratic term of the PRNU variance, although it still includes the linear term of PRNU, but PRNC 2 Usually better than EM 2 is much smaller, and PRNC can be ignored in Equation 182 In other words, it can be seen from Equation 18 that only read noise and shot noise are the main influencing factors.

[0112] At this point, this embodiment completes the modeling, and the processing device 203 obtains the gray value model of the pixel. In more detail, the gray value model of this embodiment includes formula 12, formula 13, formula 14 and formula 18, wherein formula 12 corresponds to the ADU value, formula 13 corresponds to the expected value of the ADU value, formula 14 corresponds to the variance of the ADU value, and formula 18 corresponds to the variance of the difference between the two pictures.

[0113] Then, the scene data is prepared. In order to obtain the noise model of the sensor 503, it is necessary to shoot several sets of standard color cards in a light box environment within a certain ISO gradient range. In this embodiment, a 24-color card can be used as a standard color card. The 24-color card contains six-level grayscale blocks, three primary colors of additive colors, three primary colors of subtractive colors, and skin color and the real color of simulated natural objects, with a total of 24 blocks. The 24-color card is fixed in front of the camera, and the center area of ​​the camera is used as much as possible to take pictures to avoid lens vignetting affecting the image clarity. In this embodiment, the following ISO values ​​can be used to take pictures of the 24-color card: 100, 200, 400, 800, 1600, 3200, 6400, 12800, 25600, 51200, 102400, 204800, 409600 and 819200. The ISO value range should be as wide as possible and not limited by the aforementioned range.

[0114] In this embodiment, a front image is first captured by a camera within a range of middle ISO values ​​(e.g., 1600 to 25600). The processing device 203 performs image processing on the front image to obtain a foreground portion of the front image, and determines a reference exposure value according to brightness information of the foreground portion, and then sets a dark frame exposure value and a bright frame exposure value according to the reference exposure value, wherein the dark frame exposure value is less than or equal to the reference exposure value, and the bright frame exposure value is greater than the reference exposure value.

[0115] For example, assuming that the camera captures the front image with an ISO value of 6400, the processing device 203 performs image processing on the front image to obtain the foreground portion of the image. If the processing device 203 identifies that the exposure is normal based on the foreground portion, the reference exposure can be directly set to 6400, that is, the exposure of the dark frame is less than or equal to the ISO value of 6400, and the exposure of the bright frame is greater than the ISO value of 6400. If the processing device 203 identifies that the foreground portion is overexposed, the processing device 203 adjusts the reference exposure according to the degree of overexposure, for example, setting the reference exposure to 3200, that is, the exposure of the dark frame is less than or equal to the ISO value of 3200, and the exposure of the bright frame is greater than the ISO value of 3200. If the processing device 203 identifies that the foreground portion is underexposed, the processing device 203 adjusts the reference exposure according to the degree of underexposure, for example, setting the reference exposure to 25600, that is, the exposure of the dark frame is less than or equal to the ISO value of 25600, and the exposure of the bright frame is greater than the ISO value of 25600.

[0116] Then the processing device 203 obtains a plurality of black frames, dark frames and bright frames taken by the camera based on the reference exposure. The black frame is obtained by shooting a 24-color card in a dark environment. Specifically, the camera can be directly covered with a lens cap to shoot a group of images without light. The dark frame starts from the ISO value of the reference exposure, and the exposure time of the camera is adjusted at the same time. The 24-color card is shot at different exposure times to obtain a plurality of dark frame images, and then the ISO value is lowered by one level and the above steps are repeated until all ISO values ​​lower than the reference exposure have been shot. The bright frame starts from the ISO value of the reference exposure, and then the exposure time is adjusted so that the brightness of the white color block of the 24-color card is a specific percentage of the maximum brightness. In this embodiment, the specific percentage is 80%, that is, the brightness of the white color block is 80% of the maximum brightness to avoid overexposure. Finally, based on the aforementioned set sensitivity and exposure time, the 24-color card is shot to obtain a bright frame, and then the ISO value is raised by one level and the above steps are repeated until all ISO values ​​higher than the reference exposure have been shot. In order to have a sufficient number of samples, the total number of black frames, dark frames and bright frames in this embodiment is not less than 100.

[0117] After the data is obtained, the calibration procedure is entered. In the calibration procedure, the processing device 203 substitutes the prepared values ​​of various pictures into the corresponding gray value model to obtain various parameters.

[0118] First, the processing device 203 substitutes the captured black frame image into the gray value model to obtain black frame parameters. Since the black frame image indicates that no light enters the lens, the formula 12 can be simplified as follows:

[0119] ADU p =RNS-μ RNS

[0120] That is, the ADU value of the black frame image is only related to the read noise signal and the average of the read noise signal, wherein the average of the read noise signal reflects the effects of black level bias, dark current and fixed pattern noise. Specifically, the processing device 203 averages the ADU values ​​of a group of black frame images to obtain the black frame parameter μ RNS Then subtract the black frame parameter from the ADU value of each black frame to get an array that may be positive or negative, i.e. the ADU of the black frame. p The processing device 203 processes the ADU of all black frames. p After analysis, a Gaussian distribution can be obtained. The processing device 2031 further calculates the variance of the obtained Gaussian distribution, which is the variance of the read noise.

[0121] Next, the processing device 203 substitutes the dark frame, bright frame and black frame parameters into the gray value model to obtain non-black frame parameters. There are two ways to calibrate the bright frame and dark frame. The first way is to subtract the average of the black frame from the ADU value of each frame, and then calculate the mean and variance of the 24 color blocks by channel. In more detail, the processing device 203 can obtain the following formula 19 based on formula 13 and formula 14:

[0122] Var(ADU p )=u2×Eva(ADU p ) 2 +u1×Eva(ADU p )+u0

[0123] in,

[0124]

[0125]

[0126] u0=RN 2

[0127] According to u2, u1 and u0, it can be deduced that:

[0128]

[0129]

[0130]

[0131] According to the above derivation, the processing device 2031 can obtain non-black frame parameters, which include the standard deviation of the noise signal (RN), the standard deviation of the photosites (PRNR) and the average pixel gain (EM).

[0132] The second method is to calculate the variance of two adjacent bright frames, then calculate the mean and variance of 24 color blocks by channel, and then use a linear function to linearly regress the half mean of the variance. In more detail, the processing device 203 can obtain the following equation 20 based on equation 13 and equation 18:

[0133]

[0134] in,

[0135]

[0136] z0=RN 2

[0137] Based on the above derivation, the processing device 2031 can also obtain the above non-black frame parameters by using a linear regression function.

[0138] At this point, the processing device 203 has obtained the black frame parameters and the non-black frame parameters, which are stored in the DRAM 204 as a priori input for training the neural network model. At the beginning of training, the computing device 201 takes out the black frame parameters and the non-black frame parameters from the DRAM 204, substitutes the black frame parameters and the non-black frame parameters as a priori input into the neural network model for training, so as to optimize various parameters in the neural network model. After training, the neural network model has been removed. Figure 5 possible noise from the camera.

[0139] When the board 10 or the combined processing device 20 uses Figure 5 When the image data is captured by a camera, the computing device 201 can use the trained neural network model to perform specific tasks related to image signal processing, such as image recognition, target detection, semantic segmentation, video understanding, image generation, denoising, demosaicing, etc., to obtain a denoised output.

[0140] The present invention substitutes black frame parameters and non-black frame parameters as priori inputs into a neural network model for training, thereby more accurately calibrating the noise of the camera and optimizing the denoising effect of image signal processing.

[0141] According to different application scenarios, the electronic equipment or device of the present invention may include a server, a cloud server, a server cluster, a data processing device, a robot, a computer, a printer, a scanner, a tablet computer, a smart terminal, a PC device, an Internet of Things terminal, a mobile terminal, a mobile phone, a driving recorder, a navigator, a sensor, a camera, a camera, a video camera, a projector, a watch, a headset, a mobile storage, a wearable device, a visual terminal, an automatic driving terminal, a vehicle, a household appliance, and / or a medical device. The vehicle includes an airplane, a ship, and / or a vehicle; the household appliance includes a television, an air conditioner, a microwave oven, a refrigerator, an electric rice cooker, a humidifier, a washing machine, an electric light, a gas stove, and a range hood; the medical device includes a nuclear magnetic resonance apparatus, a B-ultrasound apparatus, and / or an electrocardiograph. The electronic equipment or device of the present invention can also be applied to the Internet, the Internet of Things, a data center, energy, transportation, public administration, manufacturing, education, power grid, telecommunications, finance, retail, construction sites, medical treatment and other fields. Further, the electronic equipment or device of the present invention can also be used in the cloud, edge end, terminal and other application scenarios related to artificial intelligence, big data and / or cloud computing. In one or more embodiments, electronic devices or apparatuses with high computing power according to the solution of the present invention can be applied to cloud devices (such as cloud servers), while electronic devices or apparatuses with low power consumption can be applied to terminal devices and / or edge devices (such as smart phones or cameras). In one or more embodiments, the hardware information of the cloud device and the hardware information of the terminal device and / or edge device are compatible with each other, so that according to the hardware information of the terminal device and / or edge device, appropriate hardware resources can be matched from the hardware resources of the cloud device to simulate the hardware resources of the terminal device and / or edge device, so as to complete the unified management, scheduling and collaborative work of end-to-end or cloud-edge-to-end.

[0142] It should be noted that, for the purpose of simplicity, the present invention describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art will appreciate that the scheme of the present invention is not limited by the order of the described actions. Therefore, based on the disclosure or teaching of the present invention, those skilled in the art will appreciate that some of the steps therein may be performed in other orders or simultaneously. Further, those skilled in the art will appreciate that the embodiments described in the present invention may be regarded as optional embodiments, i.e., the actions or modules involved therein are not necessarily necessary for the implementation of one or some schemes of the present invention. In addition, depending on the different schemes, the present invention also has different emphases on the description of some embodiments. In view of this, those skilled in the art will appreciate that the parts not described in detail in a certain embodiment of the present invention may also refer to the relevant descriptions of other embodiments.

[0143] In terms of specific implementation, based on the disclosure and teachings of the present invention, those skilled in the art can understand that several embodiments disclosed in the present invention can also be implemented in other ways not disclosed herein. For example, with respect to the various units in the electronic device or device embodiments described above, this article splits them on the basis of considering the logical functions, and there may be other ways of splitting them in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features or functions in the units or components can be selectively disabled. In terms of the connection relationship between different units or components, the connection discussed in the above text in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0144] In the present invention, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units. The aforementioned components or units may be located at the same location or distributed on multiple network units. In addition, according to actual needs, some or all of the units may be selected to achieve the purpose of the scheme described in the embodiment of the present invention. In addition, in some scenarios, multiple units in the embodiment of the present invention may be integrated into one unit or each unit may exist physically separately.

[0145] In some other implementation scenarios, the above-mentioned integrated unit can also be implemented in the form of hardware, that is, a specific hardware circuit, which may include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit may include but is not limited to physical devices, and the physical devices may include but are not limited to devices such as transistors or memristors. In view of this, the various devices described herein (such as computing devices or other processing devices) can be implemented by appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs, etc. Further, the aforementioned storage unit or storage device may be any appropriate storage medium (including magnetic storage media or magneto-optical storage media, etc.), which may be, for example, a variable resistive memory (Resistive Random Access Memory, RRAM), a dynamic random access memory (Dynamic Random Access Memory, DRAM), a static random access memory (Static Random Access Memory, SRAM), an enhanced dynamic random access memory (Enhanced Dynamic Random Access Memory, EDRAM), a high bandwidth memory (High Bandwidth Memory, HBM), a hybrid memory cube (Hybrid Memory Cube, HMC), ROM and RAM, etc.

[0146] The embodiments of the present invention are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. An integrated circuit device for denoising a camera based on foreground data, comprising: Processing means for: Obtaining the foreground data of the front image; setting a reference exposure according to the foreground data; Setting the exposure of the dark frame and the bright frame according to the reference exposure; Acquire multiple black frame images taken by the camera; Under the condition that the sensitivity is less than or equal to the reference exposure, obtaining a plurality of dark frame images taken by the camera at a plurality of sensitivities; Under the condition that the light sensitivity is greater than the reference exposure, a plurality of bright frame images captured by the camera at a plurality of light sensitivities are sequentially acquired, wherein the bright frame images are acquired by executing the following steps: Adjusting the lens aperture of the camera to the maximum; Adjusting the exposure time so that the brightness of the white block of the standard color card is a specific percentage of the maximum brightness; and photographing the standard color card to obtain the bright frame image; as well as Substituting the black frame picture, the dark frame picture and the bright frame picture into the gray value model of the pixel to obtain black frame parameters and non-black frame parameters; Computing means for: Substituting the black frame parameters and the non-black frame parameters as prior inputs into a neural network model for training; as well as The image data captured by the camera is input into the trained neural network model for image signal processing to obtain a denoised output.

2. The integrated circuit device according to claim 1, wherein the total number of the black frames, dark frames and bright frames is not less than 100. The integrated circuit device according to claim 1 , wherein the specific percentage is 80%.

4. The integrated circuit device according to claim 1, wherein the dark frame is obtained by executing the following steps: A sensitivity among the plurality of sensitivities is sequentially set, and the following steps are performed: The lens aperture of the camera is adjusted, and the standard color card is photographed under different lens apertures to obtain the dark frame image.

5. The integrated circuit device according to claim 1, wherein the black frame is obtained by executing the following steps: The standard color card is photographed in a dark environment to obtain the black frame image. The integrated circuit device according to claim 1 , wherein the black frame parameter comprises a read noise signal mean.

7. The integrated circuit device of claim 6, wherein the read noise signal mean reflects the effects of black level bias, dark current and fixed pattern noise.

8. The integrated circuit device according to claim 1, wherein the non-black frame parameters include a standard deviation of a noise signal, a standard deviation of a photosensitive cell, and a pixel average gain.

Images