Image signal processor

Abstract

The invention relates to an image signal processor. The image signal processor includes: a bit expansion processor configured to generate expansion data by expanding a number of bits of input data including a Bayer image and a white image by a first number of bits; and an output circuit configured to adjust the first number of bits to the second number of bits after receiving the extended data to generate output data.

Description

Technical Field

[0001] Techniques and implementations disclosed in this disclosure generally relate to image signal processors capable of processing image signals. Background Art

[0002] Recently, as the number of functions of electronic devices has rapidly increased, the demand for improving image capturing functions using such electronic devices has also increased. Therefore, technology for improving the quality of images obtained by electronic devices is desired.

[0003] The image sensor used in the imaging device has a configuration in which color filters that transmit red (R) light, green (G) light, and blue (B) light of specific wavelength bands for each pixel are attached to the surface of the sensor. There are various types of color arrays for color filters that have been widely used, such as the Bayer array, which includes three types of color filters that transmit only light of specific wavelengths such as red, green, and blue.

[0004] Recently, as the size of pixels of image sensors is miniaturized, the amount of light incident on each pixel is reduced, increasing the signal-to-noise ratio (SNR), thereby causing the performance of image sensors to deteriorate rapidly. In order to increase the transmittance of color filters located inside pixels and realize high-sensitivity pixels in response to low-light environments, the use of image sensors including white (W) filters and other color filters that transmit only specific wavelengths of light such as red, green, and blue (RGB) is rapidly increasing. Summary of the invention

[0005] Various embodiments of the disclosed technology are directed to an image signal processor that enables rounding errors to be smaller than random noise in a process of generating a Bayer image with less noise from an image including white pixels, and thus reduces the degree of tone jumps and the degree of fading.

[0006] According to an embodiment of the disclosed technology, the image signal processor may include: a bit extension processor configured to generate extended data by expanding the number of bits of input data including a Bayer image and a white image by a first bit number; and an output circuit configured to adjust the first bit number to a second bit number to generate output data after receiving the extended data.

[0007] According to another embodiment of the disclosed technology, the image signal processor may include: a bit extension processor, which is configured to generate extended data by expanding the number of bits of input data by a first number; a storage circuit, which is configured to determine a register value for calculating the amount of noise based on the gain of the input data; an output bit number determiner, which is configured to adjust the first bit number to a second bit number based on the register value; and an output circuit, which is configured to generate output data based on the extended data and the second bit number.

[0008] According to another embodiment of the disclosed technology, the image signal processor may include: a bit extension processor, which is configured to generate extended data by expanding the number of bits of input data; a storage circuit, which is configured to determine a register value for calculating the amount of noise based on the gain of the input data; a dark area noise measurement circuit, which is configured to measure the dark area noise of the input data; an output bit number determiner, which is configured to determine the output bit number based on the register value and the dark area noise; and an output circuit, which is configured to receive the extended data and the output bit number to generate output data with a different number of bits from the extended data.

[0009] It is to be understood that both the foregoing general description and the following detailed description of the disclosed technology are illustrative and descriptive and are intended to provide further description of embodiments of the disclosure as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features and advantageous aspects of the disclosed technology will become apparent with reference to the following detailed description when considered in conjunction with the accompanying drawings.

[0011] Figure 1 is a block diagram showing an imaging device according to one embodiment of the disclosed technology.

[0012] Figure 2 Some implementations based on the disclosed technology are shown Figure 1 A schematic diagram of the operation of the bit extension processor.

[0013] Figure 3 Some implementations based on the disclosed technology are shown Figure 1 Schematic diagram of a bit extension processor.

[0014] Figure 4 is a block diagram showing an imaging device according to another embodiment of the disclosed technology.

[0015] Figure 5is a block diagram showing an imaging device according to still another embodiment of the disclosed technology.

[0016] Figure 6 is a flowchart showing an image signal processing method according to yet another embodiment of the disclosed technology.

[0017] Figure 7 is a diagram showing some implementations based on the disclosed technology. Figure 1 A block diagram of an example of a computing device of an image signal processor. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure provide implementations and descriptions of image signal processors for processing image signals, which can be used in configurations to substantially solve one or more technical or engineering problems and alleviate limitations or shortcomings encountered in some image signal processors in the prior art. Some implementations of the disclosed technology relate to an image signal processor that enables rounding errors to be less than random noise in the process of generating a Bayer image with less noise from an image including white pixels, and thus reduces the degree of hue jumps and the degree of fading. In identifying the above-mentioned problems, an image signal processor based on some implementations of the disclosed technology can reduce the degree of hue jumps and fading during the remosaic processing of an RGBW pattern, thereby improving the quality of the image signal.

[0019] Reference will now be made in detail to some embodiments of the disclosed technology shown in the accompanying drawings. Whenever possible, the same reference numerals will be used throughout the accompanying drawings to refer to the same or similar parts. Although the present disclosure is susceptible to various modifications and alternative forms, its specific embodiments are shown in the accompanying drawings. However, the present disclosure should not be construed as being limited to the embodiments set forth herein.

[0020] In the following, various embodiments will be described with reference to the accompanying drawings. However, it should be understood that the disclosed technology is not limited to a specific embodiment, but includes various modifications, equivalents and / or substitutions of the embodiments. The embodiments of the disclosed technology can provide various effects that can be directly or indirectly recognized by the disclosed technology.

[0021] Figure 1 is a block diagram illustrating an imaging device including a readout circuit according to some implementations of the disclosed technology.

[0022] refer to Figure 1 , the imaging device 1 may include an image sensing device 10 and an image signal processor ISP.

[0023] Here, the image sensing device 10 may be or include a complementary metal oxide semiconductor image sensor (CIS) for converting an optical signal into an electrical signal. The image sensing device 10 may control overall operations such as on / off, operation mode, operation timing, sensitivity, etc. through a timing controller (not shown).

[0024] Image sensing device 10 may include a pixel array 20 and a readout circuit 30 .

[0025] The pixel array 20 may include a plurality of pixels continuously arranged in a two-dimensional (2D) matrix structure (e.g., continuously arranged in row and column directions). Each of the plurality of pixels of the pixel array 20 may generate a pixel signal obtained by converting an optical signal into an electrical signal and output the converted pixel signal to the readout circuit 30.

[0026] The readout circuit 30 may process the analog pixel signal output from each of the pixels and generate digital data corresponding to the pixel signal. For example, the readout circuit 30 may include an analog-to-digital converter (ADC) for performing ADC conversion of such a pixel signal.

[0027] The image signal processor ISP may perform image processing of the image data received from the readout circuit 30. The image signal processor 200 may reduce the noise of the image data, and may perform various types of image signal processing (e.g., gamma correction, color filter array interpolation, color matrix, color correction, color enhancement, lens distortion correction, etc.) for image quality improvement of the image data. In addition, the image signal processor ISP may compress the image data that has been created by performing the image signal processing for image quality improvement, so that the image signal processor ISP may create an image file using the compressed image data. Alternatively, the image signal processor 200 may restore the image data from the image file. In this case, the scheme for compressing such image data may be a reversible format or an irreversible format. As a representative example of such a compression format, in the case of using a still image, a Joint Photographic Experts Group (JPEG) format, a JPEG 2000 format, etc. may be used. In addition, in the case of using a moving image, a plurality of frames may be compressed according to the Moving Picture Experts Group (MPEG) standard, so that a moving image file may be created. For example, an image file may be created according to the Exchangeable Image File Format (Exif) standard.

[0028] The image data transmitted from the image sensing device 10 to the image signal processor ISP may be an RGBW (red, green, blue, white) pattern in which a Bayer image and a white image are mixed. The image signal processor ISP may perform signal processing to generate a noise-reduced Bayer image using a white image in the RGBW pattern including the Bayer image and the white image.

[0029] The image signal processor ISP may include a bit extension processor 100 , an output circuit 200 , and a storage circuit 300 .

[0030] Here, the bit extension processor 100 can output extended data by expanding the number of bits of the input data so that the extended data represents a value including a fractional part. That is, the bit extension processor 100 can obtain the value of the extended data, the number of which has been expanded to represent a value including a fractional part, and therefore the number of bits of the extended data can become greater than the number of bits of the input data. For example, when the input data is N bits, where N can be an integer greater than zero "0", such as N = 10, the bit extension processor 100 can expand the number of bits of the input data to generate extended data representing a value including a fractional part, and can output extended data with (N+A) bits (e.g., (10+A) bits) instead of the input data. Here, the A bit can represent the fractional part of the extended data, and can be one or more least significant bits (LSBs) of the extended data.

[0031] When image capture (i.e., imaging) is performed at high gain in a relatively dark low-light environment, random noise may increase, and the standard deviation of the noise may further increase after re-mosaic processing of the captured image. In addition, when the output bit number is expanded, the amount of information transmitted from the image signal processor ISP may increase.

[0032] Therefore, the output circuit 200 can output the unchanged output bit number, or can output fewer output bit numbers by limiting the output bit number. That is, the output circuit 200 can receive the extended data and can generate output data based on the output bit number read from the storage circuit 300. Here, the output bit number can represent the number of bits corresponding to the fractional part that should be included in the output data. The output bit number can be set to an arbitrary number, which responds to the amount of noise to be generated after the re-mosaic process without creating false colors and preventing the amount of transmitted information from being excessively increased.

[0033] For example, the output circuit 200 may adjust the value "A" of the extended data having (N+A) bits (e.g., (10+A) bits) to a value "B" (where B may be less than or equal to A) based on the number of output bits provided from the storage circuit 300, and thus may output data having (N+B) bits (e.g., (10+B) bits). Therefore, in a low illumination environment, the image signal processor ISP can prevent an increase in the amount of data caused by meaningless bits of increased random noise by limiting the number of output bits that are not extended.

[0034] In order to adjust the number of bits of the output data, the storage circuit 300 may preset and store the number of output bits corresponding to the gain value of the pixel data. The image signal processor ISP may perform digital operations to calculate the gain of the pixel data during the re-mosaic operation. For example, the image signal processor ISP may perform various operations by adding gain to the pixel data during image signal processing (e.g., noise reduction, white balance adjustment, demosaicing, color correction, gamma correction, color change, etc.).

[0035] Since the amount of noise generated after the re-mosaic process varies according to the amount of noise in the input data, the number of output bits may vary according to the amount of noise in the input data. The amount of noise in the input data may depend primarily on the gain (sensitivity) setting of the image sensing device 10.

[0036] Therefore, the image signal processor ISP may determine the relationship between the gain value and the number of output bits in advance, may specify the number of output bits based on the determined relationship, and may store the specified number of output bits in the storage circuit 300. Here, the relationship between the gain value and the number of output bits may be obtained by performing pixel data processing in advance and converting the processing result into data.

[0037] As an example, the storage circuit 300 may include a register. The image signal processor ISP may investigate the relationship between the gain of the pixel data and the output bit number in advance, and the investigated relationship information may be implemented in software. In addition, when the software is executed in the image signal processor ISP (or a CPU (central processing unit) located inside or outside the ISP), the storage circuit 300 may store the preset output bit number in the register.

[0038] Figure 2 Some implementations based on the disclosed technology are shown Figure 1 Schematic diagram of the operation of the bit extension processor 100.

[0039] refer to Figure 2 , the input data to be applied to the bit extension processor 100 may be an RGBW pattern in which a Bayer image and a white image are mixed. The RGBW pattern may include color pixels of red (R), blue (B), and green (G) pixels and white pixels.

[0040] Each of the color pixels (R, G, and B pixels) may be a pixel on which light of a specific wavelength is incident through a corresponding color filter. In addition, a white pixel (W) may be a pixel without a color filter and may be referred to as a texture pixel, a black and white pixel, or a monochrome pixel. For example, a color pixel (R, G, B) may obtain a color pixel value according to incident light of a specific color (e.g., red, green, blue, etc.) through a color filter that transmits only incident light of a specific wavelength. In contrast, a white pixel W may obtain a white pixel value according to incident light (e.g., visible light) that is not limited to a specific color.

[0041] exist Figure 2 In the embodiment, although an RGBW pattern having a size of 64 pixels arranged in a (8×8) matrix has been described for convenience of description, the size of the RGBW pattern and the arrangement shape of the corresponding pixels are not limited thereto.

[0042] When the pixels of the image sensing device 10 are miniaturized, the amount of incident light for each pixel decreases, resulting in a reduced signal-to-noise ratio (SNR), which will cause rapid degradation in the performance of the image sensing device 10. In order to increase the transmittance of each color filter and realize high-sensitivity pixels in response to a low-illuminance environment, the utilization of the image sensing device 10 including a white pixel and a color filter (e.g., an RGB color filter) designed to transmit only light having a specific wavelength such as red, green, and blue (RGB) is rapidly increasing.

[0043] Compared with RGB images, white images can have relatively strong signal-to-noise ratio (SNR) characteristics. Since white images are designed to receive full-band light compared to RGB images that receive light in specific bands, white images can have relatively low SNR even when acquired in a low-light environment.

[0044] The bit extension processor 100 may separate the Bayer image and the white image from each other by binning the RGBW pattern. Here, "binning" may mean the sum of pixel values ​​of unit pixels having the same color on a sub-pixel array (SP) basis. For example, the bit extension processor 100 may sum (e.g., 4 sums) the pixel values ​​of unit pixels having the same color in the RGBW pattern in units of four (2×2) sub-pixels (i.e., on a sub-pixel basis), thereby resulting in the formation of the Bayer image and the white image.

[0045] The image signal processor ISP can output an image with good SNR by performing a re-mosaic process using a white image. In particular, when a high gain is applied to an image in a low illumination environment, a composite image with better SNR can be output.

[0046] The RGBW pattern may be a single-plate image sensor in which R, G, B, and W color filters are arranged in a mosaic shape. Therefore, when generating a color image, a re-mosaic process needs to be performed as color encoding to generate an RGB pixel value corresponding to each pixel.

[0047] Here, "re-mosaic processing" may be a task of outputting a high-quality image signal using a pixel signal. Alternatively, "re-mosaic processing" may refer to image processing that converts an original image into an image with a Bayer pattern.

[0048] The bit extension processor 100 may perform re-mosaic processing on the original image using a re-mosaic method, resulting in a converted image. In some implementations, the re-mosaic method may be implemented as a method for performing re-mosaic processing based on a preset reference equation. The reference equation may be an equation for calculating the pixel value of each pixel included in the image on which the re-mosaic processing has been performed. For example, the reference equation may be an equation for calculating the pixel value of the re-mosaiced image obtained when weights are applied to the pixel values ​​of adjacent pixels in the original image.

[0049] In some implementations, the bit extension processor 100 may include a re-mosaic processing module (not shown) for performing re-mosaic processing. Here, the configuration of the re-mosaic processing module may be implemented as a software block to be executed by a predetermined processor, or may be implemented as a combination of a dedicated hardware block and a processing unit.

[0050] Typically, an image signal processor ISP may convert an RGBW pattern into a Bayer pattern by performing various types of signal processing on the RGBW pattern during a re-mosaic process. For example, during the re-mosaic process, a digital calculation operation of multiplying pixel data by a gain may be performed. Here, quantization errors may be accumulated, which may result in differences in color sensing between images. Therefore, in order to reduce noise and quickly perform arithmetic calculations when re-mosaicing the RGBW pattern, the image signal processor ISP may round off the decimal part from the bits of the pixel data value without expressing the decimal part, or may limit the decimal part to a preset finite number of bits.

[0051] However, when image capture is performed at low gain in a relatively bright high-illuminance environment, the standard deviation of random noise becomes smaller than a certain value, so that the influence of rounding error may be greater than the influence of random noise. For example, if the amount of random noise becomes less than the least significant bit (LSB) "1" after the re-mosaic process, the influence of rounding error may increase and wrong color may occur. Therefore, when rounding is performed during the re-mosaic process, tone jumps or fading, etc. may occur in the output image due to the influence of such rounding errors.

[0052] Therefore, the bit extension processor 100 based on some implementations of the disclosed technology can extend the number of bits representing the pixel value of the Bayer image to have a fractional part during the re-mosaic process, and thus can extend the number of bits from the number of bits of the input data to output the extended data with the extended number of bits. For example, when the number of bits of the input data to be input to the bit extension processor 100 is N, but the number of bits of the extended data to be finally output from the bit extension processor 100 after the re-mosaic process can be extended to (N+A) bits. Therefore, the image signal processor ISP based on some implementations of the disclosed technology can prevent tone jumping or fading in the output image by making the rounding error smaller than the random noise.

[0053] Figure 3 Some implementations based on the disclosed technology are shown Figure 1 Schematic diagram of a bit extension processor 100.

[0054] Reference Figure 3 A specific method of extending bits by the bit extension processor 100 is described.

[0055] Figure 3 Pixels to be used by the bit extension processor 100 performing the bit extension process are shown. For example, a Bayer pattern having a size of 81 pixels arranged in a (9x9) matrix and a white pattern having a size of 81 pixels arranged in a (9x9) matrix will be described below as examples.

[0056] exist Figure 3 In the embodiment, although each of the Bayer pattern and the white pattern has a size of 81 pixels arranged in a (9×9) matrix for convenience of description, the size of the Bayer pattern and the white pattern and the arrangement shape of these pixels are not limited thereto.

[0057] In the Bayer pattern, a bit extension process may be performed using pixels of the same color as the target pixel (TP) among the (9x9) pixels based on the target pixel (TP) to be processed in the Bayer pattern. In the Bayer pattern, the target pixel (TP) may be a green pixel. The Bayer pattern may include 41 green pixels and the target pixel (TP). In order to reduce noise during the re-mosaic process, 41 white pixels located at the same position as the green pixel may also be used.

[0058] First, the bit extension processor 100 may calculate a statistic (r) of pixels required to perform a bit extension process, as represented by the following Equation 1. In some implementations, the statistic (r) may be calculated with 1 bit indicating an integer part and 13 bits indicating a fractional part.

[0059] [Equation 1]

[0060]

[0061] In Equation 1, a statistical value (r) may be calculated by representing a pixel value of a selected green pixel as 'Gi' and a pixel value of a selected white pixel as 'Wi'.

[0062] The similarity index (Si) between the target pixel (TP) to be processed and the selected pixel may be calculated as shown in Equation 2 below.

[0063] [Equation 2]

[0064]

[0065] In Equation 2, the similarity index (Si) may be calculated using the pixel value of the green pixel to be processed as "Gc" and the pixel value of the white pixel to be processed as "Wc." As the similarity between the pixel value (Gc) of the green pixel and the pixel value (Wc) of the white pixel increases, the similarity index (Si) becomes larger.

[0066] In Equation 2, an estimated value V(Gc, Wc) of the noise variance value of the pixel to be processed may be calculated as in Equation 3 below.

[0067] [Equation 3]

[0068] V(G c ,W c )=k(G c +W c )+V0

[0069] In Equation 3, "k" is a parameter related to the amount of shot noise. The higher the gain (sensitivity) during shooting (image capture), the higher the estimated value V (Gc, Wc) of the noise variance value. In addition, "V0" may represent the noise variance value of the image sensing device 10 when light is blocked.

[0070] Thereafter, the bit extension processor 100 may calculate a weighted average value (Z) of white pixels as shown in Equation 4 below.

[0071] [Equation 4]

[0072]

[0073] In Equation 4, a weighted average value (Z) of white pixels may be calculated based on pixel values ​​(Wi) of a total of 5 selected white pixels including a target pixel (TP) to be processed and four pixels located above, below, on the left, and on the right of the target pixel (TP). In some implementations, the weighted average value (Z) may be set to 10 bits indicating an integer part and 2 bits indicating a fractional part.

[0074] Although the similarity index (S i ) is used to calculate the weighted average value (Z) represented by equation 4, but the method of calculating the weighted average value (Z) based on some implementations of the disclosed technology is not limited thereto. In some other implementations, edge determination may be performed, and pixel values ​​may be averaged in the edge direction so that a weighted average value may be obtained.

[0075] In addition, the pixel value (G) to be obtained after the mosaicking process can be calculated as shown in Equation 5. out ).

[0076] [Equation 5]

[0077] G out = rZ

[0078] In Equation 5, the pixel value (G) can be obtained based on the statistical value (r) calculated by Equation 1 and the weighted average value (Z) calculated by Equation 4. out ). Here, the statistical value (r) may include 1 bit indicating the integer part and 13 bits indicating the fractional part, and the weighted average (Z) may include 10 bits indicating the integer part and 2 bits indicating the fractional part. In this case, the value "rZ" may include 11 bits indicating the integer part and 15 bits indicating the fractional part. For example, under high illumination and minimum sensitivity conditions, if the random noise generated after the mosaic process is reduced to about 1 / 8, about 4 bits are required to represent the fractional part and no more than 4 bits are required to fully reduce rounding errors and prevent pseudo colors. Therefore, among the total 15 bits, the fractional part can be represented by the upper 4 bits, and the remaining 11 bits except the upper 4 bits may be unnecessary and can be discarded. If the most significant bit (MSB) of the integer part is represented by "1", that is, if the number of bits corresponding to the integer part exceeds 1023, the integer part can be clipped to 1023. As a result, a total of 14 bits can be obtained by summing 10 bits of the integer part and 4 bits of the fractional part, and the pixel value (G out ) can be a 14-bit integer.

[0079] Figure 4 1 is a block diagram showing an imaging device (1_1) according to another embodiment of the disclosed technology.

[0080] Reference Figure 4 , the imaging device (1_1) may include an image sensing device 10 and an image signal processor ISP. Here, the image signal processor ISP may include a bit extension processor 100, an output circuit 200, an output bit number determiner 250, and a storage circuit 300_1.

[0081] and Figure 1The image signal processor ISP is different. Figure 4 The image signal processor ISP shown may further include an output bit number determiner 250 located between the storage circuit 300_1 and the output circuit 200, but in addition to Figure 4 The remaining components other than the output bit determiner 250 are Figure 1 The components of Figure 4 In, with Figure 1 The same elements are denoted by the same reference numerals, and thus redundant descriptions thereof will be omitted.

[0082] The storage circuit 300_1 may include a register. The image signal processor ISP may store a register value for calculating the amount of noise from the gain value of the pixel data in the storage circuit 300_1. Here, the register value may include a parameter (k) related to the amount of shot noise as shown in Equation 3, a noise variance value (V0) during light blocking as shown in Equation 6 below, etc.

[0083] The output bit number determiner 250 may adjust the bit value of the output data to be output through the output circuit 200 based on the register value stored in the storage circuit 300_1. Figure 4 In some implementations, the output bit number determiner 250 can determine whether to expand the bit value of the output data to be output by the output circuit 200. For example, the output bit number determiner 250 can calculate the output bit number by calculating the noise standard deviation based on the register value stored in the storage circuit 300_1. In some implementations, the output bit number determiner 250 can calculate the output bit number. The calculated output bit number can be included in the output data as one or two least significant bits (LSBs) of the output data.

[0084] The output circuit 200 can adjust the extended data value "A" having (N+A) bits (e.g., (10+A) bits) to a value "B" (where B may be equal to or less than A) based on the output bit number determined by the output bit number determiner 250, and thus can output data having (N+B) bits (e.g., (10+B) bits).

[0085] Figure 5 This is a block diagram showing an imaging device (1-2) according to still another embodiment of the disclosed technology.

[0086] Reference Figure 5 The imaging device 1_2 may include an image sensing device 10 and an image signal processor ISP. Here, the image signal processor ISP may include a bit extension processor 100, an output circuit 200, an output bit number determiner 250_1, a storage circuit 300_1, and a dark area noise measurement circuit 400.

[0087] and Figure 4 The image signal processor ISP is different. Figure 5 The image signal processor ISP shown may further include a dark area noise measurement circuit 400, and the output bit number determiner 250_1 may receive output data of the dark area noise measurement circuit 400, but in addition to Figure 5 The remaining components other than the circuit 400 and the determiner 250_1 are Figure 4 The components of Figure 5 In, with Figure 4 The same elements are denoted by the same reference numerals, and thus such redundant descriptions thereof will be omitted.

[0088] The storage circuit 300_1 may include a register. The image signal processor ISP may store a register value for calculating the amount of noise from the gain value of the pixel data in the storage circuit 300_1. Here, the register value may include a parameter (k) related to the amount of shot noise as shown in Equation 3.

[0089] Noise may occur in a dark area according to an exposure time or temperature of the image sensing device 10. That is, the pixel array 20 of the image sensing device 10 may include optical black pixels capable of compensating for dark signals of active pixels.

[0090] The dark area noise measurement circuit 400 may measure the noise of the dark area of ​​the optical black pixel among the input data, and may output the measured noise to the output bit number determiner 250_1. As an example, a noise variance value (V0) of the dark area noise may be obtained as shown in the following equation 6. For example, the dark area noise calculated by the dark area noise measurement circuit 400 may represent a noise variance value (V0) when light is blocked, which will be described with reference to equation 7 described later.

[0091] [Equation 6]

[0092]

[0093] In Equation 6, “I i " may represent a pixel value, and "N" may represent the number of pixels to be used in the variance calculation. Here, "I" may be an index representing a pixel.

[0094] The output bit number determiner 250_1 can adjust the bit value of the output data to be output through the output circuit 200 based on the register value stored in the storage circuit 300_1 and the dark area noise measured by the dark area noise measurement circuit 400. That is, the output bit number determiner 250_1 can adjust the bit value of the output data by considering not only the register value but also the dark area noise.

[0095] exist Figure 5In some implementations, the output bit determiner 250_1 may determine whether to expand the bit value of the output data output from the output circuit 200. For example, the output bit determiner 250_1 may calculate the output bit by calculating the noise standard deviation based on the register value and the dark area noise. In some implementations, the output bit determiner 250_1 may calculate the output bit. The calculated output bit may be included in the output data as one or two least significant bits (LSBs) of the output data.

[0096] The output circuit 200 can adjust the A value of the extended data having (N+A) bits (e.g., (10+A) bits) to a B value (where B can be equal to or less than A) based on the output bit number determined by the output bit number determiner 250_1, and thus can output data having (N+B) bits (e.g., (10+B) bits).

[0097] Figure 6 is a flowchart showing an image signal processing method according to yet another embodiment of the disclosed technology.

[0098] The following will refer to Figure 6 A method for determining the number of output bits of output data based on some implementations of the disclosed technology is described. In the following description, a method for adjusting the number of output bits by the output bit determiner 250 will be described as an example.

[0099] First, the noise variance value V(I,g) of the image sensing device 10 may be modeled as shown in Equation 7 below.

[0100] [Equation 7]

[0101] V(I,g)=k(g)(I-I0)+V0(g)

[0102] In Equation 7, "g" may represent a gain obtained when performing image capture (shooting), parameter (k) may represent a parameter proportional to gain (g), and "V0(g)" may represent a noise variance value when light is blocked. In addition, "I0" may represent a pixel value when light is blocked, and "I" may represent a pixel value.

[0103] In this manner, the image signal processor ISP may preset register values ​​(ie, gain 'G', parameter 'k', and value 'V0') for calculating the amount of noise, and may store the current register values ​​in the storage circuit 300_1 (operation S1).

[0104] When the standard deviation of noise increases by 1 / R times, a noise variance value VA(I,g) obtained after the re-mosaic process may be calculated as shown in the following Equation 8 (operation S2).

[0105] [Equation 8]

[0106] VA(I,g)={k(g)(I-I0)+V0(g)} / R 2

[0107] As can be seen from the above equation 8, when the noise variance value VA(I,g) obtained after the re-mosaic process reaches about 1 LSB, a tone jump or fading may occur in a dark area. Therefore, the output bit number determiner 250 may calculate the output bit number. The calculated output bit number may be included in the output data as one or two least significant bits (LSBs) of the output data.

[0108] The output bit number determiner 250 based on some implementations of the disclosed technology may calculate the output bit number based on the value of the gain (g), the value of the parameter (k), and the dark area noise value generated when light is blocked (operation S3).

[0109] [Equation 9]

[0110] VA(I,g)={g(S)(I-I0)+V00+g 2 V01} / R 2

[0111] When the parameter (k) is proportional to the gain (g), V0(g) can be modeled as, for example, “V00+g 2 V01". Then, as shown in Equation 9, the noise variance value VA(I,g) obtained after the re-mosaic process can be estimated from the gain (g) and the pixel value (I).

[0112] Here, the pixel value (I) may be set to zero "0" resulting in minimum variance. In some implementations, the pixel value (I) may also use the value of a pixel having a brightness level at which unexpected problems may be identified when the gamma curve is applied to the development process. Thus, the need for alignment expansion may vary depending on the shape of the gamma curve or tone curve.

[0113] As described above, the output data output from the output circuit 200 may have (N+B) bits, where B may be equal to or less than A (operation S4). As a result, the output bit number determiner 250 may determine the noise variance value VA(I, g). In this case, if the number of bits needs to be extended, the output bit number determiner 250 may set the B value in the same manner as the A value, and may output the result value. On the other hand, the output bit number determiner 250 may determine the noise variance value VA(I, g). In this case, if the number of bits of the output data needs to be limited, the output bit number determiner 250 may set the B value to be less than the A value, and may output the result value.

[0114] For example, if the noise variance value VA(I,g) becomes smaller than each of 0.25 and 1, it may be determined that the bit extension is required. That is, when the gain (g) value is small, the output bit number determiner 250 may determine that bit extension is required.

[0115] [Equation 10]

[0116] B~[VA(I,g)] -0.5

[0117] In Equation 10, the extended number of bits (B) may be determined as shown in Equation 10. For ease of description, although the embodiments of the disclosed technology have disclosed a method for determining the noise variance value VA(I,g) based on the gain (g) to calculate the extended output number of bits (B), the scope or spirit of the method for calculating the extended output number of bits (B) based on some implementations of the disclosed technology is not limited thereto, and the method for calculating the extended output number of bits (B) may be sufficiently changed.

[0118] As described above, according to an embodiment of the disclosed technology, the optimal number of output bits can be set to one or two least significant bits (LSBs) of the output data. The optimal number of output bits can represent the standard deviation of the noise after mosaic processing. For ease of description, although the embodiment of the disclosed technology has disclosed a method for setting the output bit number to one or two LSBs of the output data as the standard deviation of the noise, the scope or spirit of the disclosed technology is not limited thereto, and it should be noted that the output bit number can also be set to another value as needed.

[0119] As an example, the input data is 10 bits, and the standard deviation of the noise is 4.0. When the standard deviation of the noise becomes 1 / 8.0 during the re-mosaic process (not bit extension), the standard deviation of the noise after the re-mosaic process can be achieved with 10-bit conversion to "4.0×(1.0 / 8.0)=0.5". In this case, the bits can be extended because color tones, etc. occur after the re-mosaic process.

[0120] For example, when the output data is 14 bits by expanding 4 bits from 10 bits of input data, the noise standard deviation of the output image can be represented by "0.5×2^4=8.0". That is, in the 14-bit output data of the image, the standard deviation of the noise can be represented by 4 LSBs, which is more output bit numbers than 2 LSBs, and the number of bits of the output data may be excessive. Since the standard deviation of the noise is represented by 1 LSB when the output data is 11 bits, and the standard deviation of the noise is represented by 2 LSBs when the output data is 12 bits, it can be seen that the optimal number of bits of the output data is 11 to 12 bits with one to two LSBs.

[0121] As described above, embodiments of the disclosed technology can pre-investigate (or test and detect) the amount of noise to be generated after re-mosaic processing (in dark areas), and can determine the number of output bits as one or two LSBs in the output data, the determined number of output bits representing the amount of noise.

[0122] Here, the amount of noise to be generated after the re-mosaic process can be estimated from the number of pixels mentioned when the statistical value (r) required for the re-mosaic process is processed as in the above equation 1. According to another embodiment, software for executing the algorithm required for the re-mosaic process as a floating point number is pre-written so that various images are processed with the software. Finally, the relationship between the noise standard deviation of the input image for each gain setting and the noise standard deviation of the output image for each gain setting can be studied and calculated.

[0123] Figure 7 is shown corresponding to Figure 1 Block diagram of a computing device 1000 of an imaging device 1.

[0124] refer to Figure 7 , the computing device 1000 may represent a computer program product for executing Figure 1 An embodiment of a hardware configuration of an operation of the imaging device 1.

[0125] The computing device 1000 may be mounted on a chip independent of a chip on which an image sensing device is mounted. According to one embodiment, a chip on which an image sensing device is mounted and a chip on which the computing device 1000 is mounted may be implemented in one package (e.g., a multi-chip package (MCP)), but the scope of the disclosed technology is not limited thereto.

[0126] in addition, Figure 1 The internal configuration or arrangement of the computing device 1000, the image sensing device 10, and the image signal processor ISP described in the above may vary according to the implementation. For example, at least a portion of the computing device 1000 may be included in the image sensing device 10. Alternatively, at least a portion of the computing device 1000 may be included in the image sensing device 10. In this case, at least a portion of the computing device 1000 may be mounted together on a chip on which the image sensing device 10 is mounted.

[0127] The computing device 1000 may include a processor 1010 , a memory 1020 , an input / output interface 1030 , and a communication interface 1040 .

[0128] Processor 1010 can process execution Figure 1The processor 1010 may be referred to as an image signal processor ISP, but the scope of the disclosed technology is not limited thereto.

[0129] The memory 1020 may store data and / or instructions required to perform operations of the components (100, 200, 300) of the image signal processor ISP, and may be accessed by the processor 1010. For example, the memory 1020 may be a volatile memory (e.g., a dynamic random access memory (DRAM), a static random access memory (SRAM), etc.) or a nonvolatile memory (e.g., a programmable read-only memory (PROM), an erasable PROM (EPROM), an EEPROM (electrically erasable PROM), a flash memory, etc.).

[0130] That is, a computer program for executing the operation of the image signal processor ISP disclosed in the present disclosure is recorded in the memory 1020 and is executed and processed by the processor 1010, thereby realizing the operation of the image signal processor ISP.

[0131] The input / output interface 1030 is an interface that connects an external input device (eg, a keyboard, a mouse, a touch panel, etc.) and / or an external output device (eg, a display) to the processor 1010 to allow data to be transmitted and received.

[0132] The communication interface 1040 is a component capable of transmitting and receiving various data with an external device (eg, an application processor, an external memory, etc.), and may be a device supporting wired or wireless communication.

[0133] It can be seen from the above description that an image signal processor based on some implementations of the disclosed technology can reduce the degree of tone jumping and fading during the re-mosaic processing of the RGBW pattern, thereby improving the quality of the image signal.

[0134] The embodiments of the disclosed technology can provide various effects that can be directly or indirectly recognized through the above-mentioned embodiments of the present disclosure.

[0135] Although a number of illustrative embodiments have been described, it should be understood that modifications and enhancements to the disclosed embodiments and other embodiments may be designed based on what is described and / or shown in this disclosure. In addition, the embodiments may be combined to form additional embodiments.

[0136] CROSS-REFERENCE TO RELATED APPLICATIONS

[0137] This application claims priority to and the benefit of Korean Patent Application No. 10-2023-0111819, filed on August 25, 2023, which is hereby incorporated by reference in its entirety.

Claims

1. An image signal processor, the image signal processor comprising: a bit extension processor that generates extended data by extending the number of bits of input data including the Bayer image and the white image by the first bit; as well as The output circuit adjusts the first bit number to a second bit number to generate output data when receiving the extended data.

2. The image signal processor according to claim 1, wherein: The first bit number represents a fractional portion of the extension data, and the first number of bits are one or more least significant bits (LSBs) of the extension data.

3. The image signal processor according to claim 1, wherein: The second digit is equal to or smaller than the first digit. 4 . The image signal processor according to claim 3 , further comprising a storage circuit configured to store the second bit number as a register value.

5. The image signal processor according to claim 3, wherein: The output circuit also determines the second bit number based on a gain of the input data.

6. The image signal processor according to claim 1, wherein: The bit extension processor also: generating the Bayer image and the white image by merging the input data, and A re-mosaic process is performed based on the white image to generate the Bayer image from which noise has been removed.

7. The image signal processor according to claim 1, wherein: The bit extension processor also determines the first bit number based on like-color pixels in each of the Bayer image and the white image that are located at a specific position relative to a target pixel.

8. The image signal processor according to claim 7, wherein: The bit extension processor determines the first number of bits by: Obtaining the statistical value of the same color pixel according to the similarity index between the target pixel and the same color pixel, and The first digit is calculated based on the statistical value.

9. The image signal processor according to claim 8, wherein: The bit extension processor also: calculating an estimate of the noise variance value in response to a parameter related to the amount of shot noise and a noise variance value generated when light is blocked, and The similarity index is calculated based on the estimated value.

10. The image signal processor according to claim 8, wherein: The bit expansion processor: The pixel value of the extended data after re-mosaic processing is calculated in response to the statistical value and a weighted average value of white pixels.

11. An image signal processor, comprising: a bit extension processor that generates extended data by extending the number of bits of input data by a first digit; a storage circuit that determines a register value for calculating an amount of noise based on a gain of the input data; an output number of bits determiner, the output number of bits determiner adjusting the first number of bits to a second number of bits based on the register value; as well as An output circuit generates output data based on the extended data and the second bit number.

12. The image signal processor according to claim 11, wherein: The storage circuit also determines the register value based on a parameter related to an amount of shot noise and a noise variance value generated when light is blocked.

13. The image signal processor according to claim 11, wherein: The input data includes a Bayer image and a white image.

14. The image signal processor according to claim 11, wherein: The second digit is equal to or smaller than the first digit.

15. The image signal processor according to claim 11, wherein: The output bit number determiner further determines the second bit number by calculating a noise standard deviation based on the register value.

16. The image signal processor according to claim 15, wherein: The output bit number determiner determines the second bit number based on the noise standard deviation, and The second bit number represents a fractional portion of the output data, and the second number of bits is included in the output data as one or two least significant bits (LSBs) of the output data.

17. An image signal processor, comprising: a bit extension processor, the bit extension processor generating extended data by expanding the number of bits of input data; a storage circuit that determines a register value for calculating an amount of noise based on a gain of the input data; a dark area noise measurement circuit, wherein the dark area noise measurement circuit measures the dark area noise of the input data; an output number of bits determiner, the output number of bits determiner determining the output number of bits based on the register value and the dark area noise; as well as An output circuit receives the extended data and the output bit number to generate output data having a different bit number from the extended data.

18. The image signal processor according to claim 17, wherein: The output bit number determiner determines the output bit number by calculating a noise standard deviation in response to the register value and the dark area noise.

19. The image signal processor according to claim 17, wherein: The input data includes a Bayer image and a white image.

20. The image signal processor according to claim 17, wherein: The extension data includes an extension bit number representing a decimal part, and the bits of the extension number are one or more least significant bits (LSBs) of the extension data.

Images