Imaging device

Abstract

An imaging apparatus includes: an image sensing device including a pixel array of sensing pixels, the pixel array including at least one first pixel sensing incident light and having a first dynamic range and at least one second pixel sensing incident light and having a second dynamic range, the first dynamic range being represented by a first measurable high light intensity and a first measurable low light intensity, the second dynamic range being represented by a second measurable high light intensity and a second measurable low light intensity, the second measurable high light intensity being higher than the first measurable high light intensity of the first pixel, wherein the pixel array of sensing pixels makes the ratio of the number of first pixels to all sensing pixels higher than the ratio of the number of second pixels to all sensing pixels; and an image signal processor receiving and processing pixel data from the image sensing device to generate a high dynamic range (HDR) image corresponding to a dynamic range larger than the first dynamic range or the second dynamic range based on the pixel data of the first pixel and the second pixel in the pixel array.

Description

Technical Field

[0001] The technologies and implementations included in this patent document generally relate to an imaging device capable of generating image data by sensing light. Background Technology

[0002] Image sensing devices are devices that capture optical images by converting light into electrical signals using photosensitive semiconductor materials that react to light. With the development of the automotive, medical, computer, and communications industries, the demand for high-performance image sensing devices is constantly increasing in various fields such as smartphones, digital cameras, game consoles, IoT (Internet of Things), robotics, security cameras, and medical miniature cameras.

[0003] Image sensing devices can be broadly categorized into CCD (Charge-Coupled Device) image sensing devices and CMOS (Complementary Metal-Oxide-Semiconductor) image sensing devices. Compared to CMOS image sensing devices, CCD image sensing devices offer better image quality, but they tend to consume more power and are larger. CMOS image sensing devices, on the other hand, are smaller and consume less power. Furthermore, the use of CMOS manufacturing technology allows for the integration of photosensitive elements and other signal processing circuitry onto a single chip, enabling the production of miniaturized image sensing devices at a lower cost. For these reasons, CMOS image sensing devices are being developed for many applications, including mobile devices. Summary of the Invention

[0004] Various embodiments of the disclosed technology relate to an imaging apparatus capable of acquiring high dynamic range (HDR) images.

[0005] According to an embodiment of the disclosed technology, an imaging apparatus may include: an image sensing device comprising a pixel array of sensing pixels, the pixel array including at least one first pixel for sensing incident light and having a first dynamic range and at least one second pixel for sensing incident light and having a second dynamic range, the first dynamic range being represented by a first measurable high light intensity that does not saturate the first pixel and a first measurable low light intensity relative to a pixel noise level of the first pixel, the second dynamic range being represented by a second measurable high light intensity that does not saturate the second pixel and a second measurable low light intensity relative to a pixel noise level of the second pixel, the second measurable high light intensity being higher than the first measurable high light intensity of the first pixel, wherein the pixel array of sensing pixels is configured such that the ratio of the number of first pixels to all sensing pixels is higher than the ratio of the number of second pixels to all sensing pixels; and an image signal processor configured to receive pixel data from the image sensing device and process the pixel data to generate a high dynamic range (HDR) image corresponding to a dynamic range larger than the first dynamic range or the second dynamic range based on the pixel data of the first pixel and the pixel data of the second pixel in the pixel array.

[0006] According to another embodiment of the disclosed technology, an imaging apparatus may include: an image sensing device including at least one first pixel and at least one second pixel with a sensitivity lower than that of the at least one first pixel; and an image signal processor configured to generate a high dynamic range (HDR) image corresponding to a dynamic range greater than that of the at least one first pixel or the dynamic range of the at least one second pixel, based on pixel data of the at least one first pixel and pixel data of the at least one second pixel. The ratio of the at least one first pixel to all pixels included in the image sensing device is higher than the ratio of the at least one second pixel to all pixels included in the image sensing device.

[0007] It will be understood that the above general description and the following detailed description of the disclosed technology are both illustrative and explanatory, and are intended to provide further explanation of the claimed disclosure. Attached Figure Description

[0008] The above and other features and advantages of the disclosed technology will become readily apparent when considered in conjunction with the accompanying drawings and with reference to the following detailed description.

[0009] Figure 1 This is a block diagram illustrating an example of an imaging system based on some implementations of the disclosed technology.

[0010] Figure 2 This demonstrates some implementation methods based on the disclosed technology. Figure 1 A block diagram of an example image sensing device is shown.

[0011] Figure 3 This demonstrates some implementation methods based on the disclosed technology. Figure 1 A schematic diagram of an example pixel array is shown.

[0012] Figure 4 The graph shows an example of the response of some implementations based on the disclosed technology to the intensity of incident light applied to the high-sensitivity pixel and the low-sensitivity pixel.

[0013] Figures 5A to 5D This is a schematic diagram illustrating examples of how high-sensitivity pixels and low-sensitivity pixels are implemented in different ways based on differences in transmittance, according to some implementation methods of the disclosed technology.

[0014] Figure 6A and Figure 6B This diagram illustrates examples of how high-sensitivity pixels and low-sensitivity pixels are implemented in different ways based on differences in circuit structure, using some implementation methods of the disclosed technology.

[0015] Figure 7 It is a graph showing an example of the response of some implementations based on the disclosed technology to the illuminance of incident light applied to each of the low-sensitivity green pixel, low-sensitivity red pixel, and low-sensitivity blue pixel.

[0016] Figure 8 This is a diagram illustrating an example of a frame generated by an image sensing device based on some implementations of the disclosed technology.

[0017] Figure 9 It is a graph showing the comparison results between pixel data and noise for high-sensitivity pixels and low-sensitivity pixels based on some implementations of the disclosed technology.

[0018] Figure 10 This is a diagram illustrating an example of a method for establishing combined weights based on some implementations of the disclosed technology.

[0019] Figure 11 This is another example of a method for establishing combined weights based on some implementations of the disclosed technology.

[0020] Figure 12 This is another example of a method for establishing combined weights based on some implementations of the disclosed technology.

[0021] Figure 13 This is a diagram illustrating an example of the arrangement of high-sensitivity pixels and low-sensitivity pixels based on some implementations of the disclosed technology.

[0022] Figure 14 It is a graph showing the variation of pixel data of shared pixels with the intensity of incident light in some implementations based on the disclosed technology.

[0023] Figure 15 It is a graph showing the comparison results between pixel data of shared pixels and gain-controlled pixel data based on some implementations of the disclosed technology. Detailed Implementation

[0024] This patent document provides implementations and examples of an imaging apparatus capable of generating image data by sensing light, which can be used in configurations to substantially solve one or more technical or engineering problems and mitigate some limitations or drawbacks encountered in other image sensing apparatuses. Some implementations of the disclosed technology relate to an imaging apparatus capable of acquiring high dynamic range (HDR) images. The disclosed technology provides various implementations of an image sensing apparatus configured such that high-sensitivity pixels and low-sensitivity pixels are arranged together, thereby acquiring a high dynamic range (HDR) image by performing only one image capture (i.e., one shooting action).

[0025] Reference will now be made in detail to embodiments of the disclosed technology, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numerals will be used throughout the drawings to denote the same or similar parts. While this disclosure is readily available for various modifications and alternatives, specific embodiments thereof are illustrated in the accompanying drawings by way of example. However, this disclosure should not be construed as limiting itself to the embodiments set forth herein.

[0026] Various embodiments will now be described with reference to the accompanying drawings. However, it should be understood that the disclosed technology is not limited to specific embodiments, but includes various modifications, equivalents, and / or substitutions of the embodiments. Embodiments of the disclosed technology can provide various effects that can be directly or indirectly identified by the disclosed technology.

[0027] Figure 1 This is a block diagram illustrating an example of an imaging system 1 based on some implementations of the disclosed technology. Figure 2 It is shown Figure 1 A block diagram of an example image sensing device is shown. Figure 3 It is shown Figure 1 A schematic diagram of an example pixel array is shown. Figure 4 This is a graph showing an example of the response depending on the intensity of the incident light applied to the high-sensitivity pixel and the low-sensitivity pixel.

[0028] Reference Figure 1Imaging system 1 may refer to, for example, a digital still camera for capturing still images or a digital video camera for capturing moving images. For example, imaging device 10 may be implemented as a digital single-lens reflex (DSLR) camera, a mirrorless camera, or a smartphone. Imaging device 10 may include a device having both a lens and an image pickup element, enabling the device to capture (or photograph) a target object, thereby creating an image of the target object.

[0029] The imaging system 1 may include an imaging device 10 and a host device 20.

[0030] The imaging device 10 may include an image sensing device 100, a line memory 200, an image signal processor (ISP) 300, and an input / output (I / O) interface 400.

[0031] The image sensing device 100 may be a complementary metal-oxide-semiconductor (CIS) image sensor for converting optical signals into electrical signals. The overall operation of the image sensing device 100, such as on / off states, operating modes, operating timing, and sensitivity, can be controlled by the ISP 300. Under the control of the ITSP 300, the image sensing device 100 can provide image data obtained by converting optical signals into electrical signals to the row memory 200.

[0032] Reference Figure 2 The image sensing device 100 may include a pixel array 110, a row driver 120, a correlated dual sampler (CDS) 130, an analog-to-digital converter (ADC) 140, an output buffer 150, a column driver 160, and a timing controller 170. Figure 2 The components of the image sensing device 100 shown are discussed as examples only, and this patent document covers numerous other changes, substitutions, variations, alterations and modifications.

[0033] Pixel array 110 may include a plurality of imaging pixels arranged in rows and columns. In one example, the plurality of imaging pixels may be arranged as a two-dimensional pixel array including rows and columns. In another example, the plurality of imaging pixels may be arranged as a three-dimensional pixel array. The plurality of imaging pixels may convert optical signals into electrical signals based on individual pixels or based on pixel groups, wherein imaging pixels in a pixel group share at least certain internal circuitry. Pixel array 110 may receive pixel control signals including a row selection signal, a pixel reset signal, and a transmission signal from row driver 120. Upon receiving the pixel control signal, a corresponding imaging pixel in pixel array 110 may be enabled to perform an operation corresponding to the row selection signal, pixel reset signal, and transmission signal. Each imaging pixel may generate photocharge corresponding to the intensity of incident light and may generate an electrical signal corresponding to the amount of photocharge, thereby sensing incident light. For ease of description, an imaging pixel may also be referred to as a pixel.

[0034] The row driver 120 can enable the pixel array 110 to perform specific operations on the imaging pixels in the corresponding row based on command and control signals provided by controller circuitry such as timing controller 170. In some implementations, the row driver 120 can select one or more imaging pixels arranged in one or more rows of the pixel array 110. The row driver 120 can generate a row selection signal to select one or more rows among multiple rows. The row driver 120 can sequentially enable a pixel reset signal for resetting the imaging pixel corresponding to at least one selected row, and a transmission signal for the pixel corresponding to at least one selected row. Thus, a reference signal and an image signal, as analog signals generated by the respective imaging pixels of the selected rows, can be sequentially transmitted to the CDS 130. The reference signal can be an electrical signal provided to the CDS 130 when the sensing node (e.g., a floating diffusion node) of the imaging pixel is reset, and the image signal can be an electrical signal provided to the CDS 130 when the photocharge generated by the imaging pixel accumulates in the sensing node. The reference signal, indicating the unique reset noise of each pixel, and the image signal, indicating the incident light intensity, can be collectively referred to as pixel signals as needed.

[0035] Image sensing device 100 can use correlated double sampling (CDS) to remove unwanted offset values ​​(referred to as fixed pattern noise) of pixels by sampling the pixel signal twice to remove the difference between the two samples. In one example, correlated double sampling (CDS) can remove unwanted offset values ​​of pixels by comparing the pixel output voltage obtained before and after the accumulation of photocharge generated by incident light in the sensing node, so that only the pixel output voltage based on the incident light can be measured. In some embodiments of the disclosed art, CDS 130 can sequentially sample and hold the voltage levels of the reference signal and the image signal provided from pixel array 110 to each of the plurality of column lines. That is, CDS 130 can sample and hold the voltage levels of the reference signal and the image signal corresponding to each column of pixel array 110.

[0036] In some implementations, CDS130 can transmit the reference signals and image signals of each column as correlated double-sampled signals to ADC 140 based on control signals from timing controller 170.

[0037] ADC 140 is used to convert analog CDS signals into digital signals. In some implementations, ADC 140 may be implemented as a ramp comparator ADC. In some embodiments of the disclosed technology, ADC 140 may convert the correlated double-sampled signals generated by CDS 130 for each column into digital signals and output digital signals.

[0038] The ADC 140 may include multiple column counters. Each column of the pixel array 110 is coupled to a column counter, and image data can be generated by using the column counters to convert correlated double-sampled signals received from each column into digital signals. In another embodiment of the disclosed technology, the ADC 140 may include a global counter to convert correlated double-sampled signals corresponding to columns into digital signals using a global code provided from the global counter.

[0039] Output buffer 150 can temporarily hold column-based image data provided from ADC 140 for output image data. In one example, image data provided from ADC 140 to output buffer 150 can be temporarily stored in output buffer 150 based on control signals from timing controller 170. Output buffer 150 can provide an interface to compensate for data rate or transmission rate differences between image sensing device 100 and other devices.

[0040] The column driver 160 can select the column of the output buffer upon receiving a control signal from the timing controller 170, and sequentially output the image data temporarily stored in the selected column of the output buffer 150. In some implementations, upon receiving an address signal from the timing controller 170, the column driver 160 can generate a column selection signal based on the address signal and select the column of the output buffer 150, thereby outputting image data from the selected column of the output buffer 150 as an output signal.

[0041] The timing controller 170 can control the operation of the row driver 120, ADC 140, output buffer 150 and column driver 160.

[0042] The timing controller 170 provides the row driver 120, column driver 160, and output buffer 150 with clock signals required for the operation of the various components of the image sensing device 100, control signals for timing control, and address signals for selecting rows or columns. In embodiments of the disclosed technology, the timing controller 170 may include logic control circuitry, phase-locked loop (PLL) circuitry, timing control circuitry, communication interface circuitry, etc.

[0043] Figure 3 This is a diagram illustrating an example of the arrangement of pixels included in pixel array 110. Figure 3 This is a diagram showing 64 pixels arranged in a matrix array comprising 8 rows and 8 columns. For example, 64 pixels can be used as the smallest unit and can be repeated in both the row and column directions, but are not limited to this.

[0044] The pixel array 110 may include red pixels, blue pixels, and green pixels. Each red pixel may include a red color filter configured to selectively transmit light corresponding to red (i.e., red light) and be capable of sensing red light. Each blue pixel may include a blue color filter configured to selectively transmit light corresponding to blue (i.e., blue light) and be capable of sensing blue light. Each green pixel may include a green color filter configured to selectively transmit light corresponding to green (i.e., green light) and be capable of sensing green light.

[0045] In some implementations, red, blue, and green pixels can be arranged in a quaternary Bayer pattern. A quaternary Bayer pattern refers to a pattern in which the red, blue, and green pixel groups are arranged in a Bayer pattern. In the red pixel group, the red pixels are arranged in a (2×2) matrix. In the blue pixel group, the blue pixels are arranged in a (2×2) matrix. In the green pixel group, the green pixels are arranged in a (2×2) matrix. Therefore, in the quaternary Bayer pattern, the red and blue pixel groups can be arranged diagonally. In the quaternary Bayer pattern, the green pixel group can be positioned to contact one side of each of the red and blue pixel groups, such that all pixels can be arranged in a (4×4) matrix.

[0046] The pixels included in pixel array 110 can be classified according to the different colors of light that each pixel needs to sense. In some implementations, the pixels included in pixel array 110 can also be classified into high-sensitivity pixels and low-sensitivity pixels based on their sensitivity to incident light. Pixels with relatively high sensitivity can be defined as high-sensitivity pixels, and pixels with relatively low sensitivity can be defined as low-sensitivity pixels. (See later...) Figure 4 Provide a detailed description of high-sensitivity pixels and low-sensitivity pixels.

[0047] Pixels in a pixel array, configured as red, blue, and green pixels, can be formed using pixels with relatively high sensitivity or pixels with relatively low sensitivity. For example, a red pixel with relatively high sensitivity can be defined as a high-sensitivity red pixel, and a red pixel with relatively low sensitivity can be defined as a low-sensitivity red pixel. Similarly, a blue pixel with relatively high sensitivity can be defined as a high-sensitivity blue pixel, and a blue pixel with relatively low sensitivity can be defined as a low-sensitivity blue pixel. Furthermore, a green pixel with relatively high sensitivity can be defined as a high-sensitivity green pixel, and a green pixel with relatively low sensitivity can be defined as a low-sensitivity green pixel.

[0048] exist Figure 3In this configuration, 16 red pixels out of the 64 pixels in the pixel array can be high-sensitivity red pixels (RP_H), and 16 blue pixels out of the 64 pixels in the pixel array can be high-sensitivity blue pixels (BP_H). However, the 32 green pixels out of the 64 pixels in the pixel array can include 28 high-sensitivity green pixels (GP_H) and four low-sensitivity green pixels (GP_L).

[0049] Therefore, the total of 64 pixels can include four low-sensitivity pixels and 60 high-sensitivity pixels, and the ratio of low-sensitivity pixels to all pixels (i.e., 64 pixels) can be set to 4 / 64 (=1 / 16), which is less than the ratio of high-sensitivity pixels to all pixels (i.e., 64 pixels).

[0050] The ratio of low-sensitivity pixels to all pixels (i.e., 64 pixels) is not limited to... Figure 3 Examples can be experimentally determined to produce sufficient HDR performance at a level that does not degrade the final image resolution of the image captured by the high-sensitivity pixels.

[0051] although Figure 3 The example shown illustrates that the low-sensitivity pixel is a green pixel, but other implementations are possible and not limited to this. In some implementations, the low-sensitivity pixel can also be a red or blue pixel.

[0052] Figure 4 This shows how the responses of high-sensitivity and low-sensitivity pixels vary depending on the intensity of incident light applied to the corresponding pixel. (Refer to...) Figure 4 The response of a high-sensitivity pixel increases relatively more with increasing incident light intensity, while the response of a low-sensitivity pixel increases relatively less with increasing incident light intensity. Here, pixel response can refer to the image data (i.e., pixel data) of the corresponding pixel. In this case, incident light intensity can refer to the brightness or illuminance of the target object to be sensed by the corresponding pixel. In some implementations, the high-sensitivity pixel may also be referred to as the first pixel, and the low-sensitivity pixel may also be referred to as the second pixel.

[0053] The response can have a signal-to-noise ratio (SNR) limit (represented by the SNR limit) and a saturation level (represented by the Saturation).

[0054] exist Figure 4The diagram illustrates two distinct responses: a signal-to-noise ratio (SNR) threshold level (represented by "SNR Limit") and a saturation level (represented by "Saturation"). The SNR threshold level refers to the threshold at which a predetermined reference SNR is satisfied. Responses below the SNR threshold level are considered invalid responses that do not satisfy the reference SNR, while responses above the SNR threshold level are considered valid responses that satisfy the reference SNR. The reference SNR can be determined experimentally, taking into account the characteristics of the image sensing device 100. However, for ease of description, it is assumed that... Figure 5A The SNR threshold level (SNR limit) is ignored in the examples shown in the following figures.

[0055] The saturation level refers to the maximum level indicating the intensity of incident light. The saturation level can be determined by the pixel's ability to convert the intensity of incident light into photoelectric charge (e.g., the capacitance of a photoelectric conversion element), the ability of the photoelectric charge to convert into an analog signal (e.g., the capacitance of a floating diffuse (FD) region), and the ability of the analog signal to convert into a digital signal (e.g., the input range of an ADC). As the intensity of the incident light increases, the response can increase proportionally to the intensity of the incident light until the response reaches the saturation level. After the response reaches the saturation level, the response may not increase further despite increases in the intensity of the incident light. For example, after the response reaches the saturation level, the response may have the same value as the saturation value and not increase beyond the saturation level.

[0056] The effective response of each pixel can be defined as the response that indicates the intensity of incident light while exhibiting a sufficient reference SNR. The range of incident light intensities corresponding to the effective response of a pixel can be referred to as the dynamic range of the pixel, and includes measurable high intensities of incident light that do not saturate the pixel and measurable low intensities of incident light above the pixel noise level. Therefore, the dynamic range of a pixel can be defined as the range of incident light intensities in which each pixel has an effective response between the aforementioned measurable high and measurable low intensities. In some cases, the dynamic range of a pixel can be represented by the ratio of measurable high intensity to measurable low intensity or the pixel noise intensity.

[0057] High-sensitivity pixels provide a relatively large increase in response to an increase in incident light intensity. Therefore, the response of a high-sensitivity pixel can have a relatively large slope in response to an increase in incident light intensity until the response reaches a saturation level, and after reaching the saturation level, it has a fixed level corresponding to the saturation level regardless of further increases in incident light intensity.

[0058] Low-sensitivity pixels provide a relatively small increase in response to an increase in incident light intensity. Therefore, the response of a low-sensitivity pixel can increase at a relatively small slope in response to an increase in incident light intensity until the response reaches a saturation level, after which it has a fixed level corresponding to the saturation level regardless of further increases in incident light intensity.

[0059] like Figure 4 As shown, the minimum value (or first measurable low light intensity) of the high-sensitivity pixel dynamic range (DR_H) (or first dynamic range) can be smaller than the minimum value (or second measurable low light intensity) of the low-sensitivity pixel dynamic range (DR_L) (or second dynamic range), and the maximum value (or first measurable high light intensity) of the high-sensitivity pixel dynamic range (DR_H) can be smaller than the maximum value (or second measurable high light intensity) of the low-sensitivity pixel dynamic range (DR_L). Therefore, in the low-brightness range where the incident light intensity is relatively low, the high-sensitivity pixel is more suitable for sensing the incident light intensity. In the high-brightness range where the incident light intensity is relatively high, the low-sensitivity pixel is more suitable for sensing the incident light intensity.

[0060] High dynamic range (HDR) can be achieved using both the response of high-sensitivity pixels suited to the low-brightness range and the response of low-sensitivity pixels suited to the high-brightness range. Compared to using only one of the high-sensitivity and low-sensitivity pixels, the aforementioned use of both allows the entire pixel array to have a high dynamic range (HDR) corresponding to a specific range from the minimum of the high-sensitivity pixel dynamic range to the maximum of the low-sensitivity pixel dynamic range. For this purpose, at least a portion of the high-sensitivity pixel dynamic range and at least a portion of the low-sensitivity pixel dynamic range can overlap each other.

[0061] To achieve high-sensitivity and low-sensitivity pixels, the exposure time can be varied. Exposure time refers to the time it takes to generate photocharge in response to the intensity of incident light. Pixels with relatively long exposure times correspond to high-sensitivity pixels, while pixels with relatively short exposure times correspond to low-sensitivity pixels.

[0062] When high-sensitivity pixels and low-sensitivity pixels are achieved by varying the exposure time, the high-sensitivity pixels and low-sensitivity pixels can sense the scene at different points in time, which may cause motion artifacts or motion blur due to the movement of a target object that moves quickly from one place to another.

[0063] In some implementations, high-sensitivity pixels and low-sensitivity pixels are simultaneously arranged in pixel array 110 to sense the scene at the same time. ISP 300 can use both image data from the high-sensitivity pixels and image data from the low-sensitivity pixels to synthesize an HDR image, thereby preventing motion artifacts or motion blur. In this case, the HDR image may refer to an image corresponding to a dynamic range greater than a first dynamic range of the high-sensitivity pixels or a second dynamic range of the low-sensitivity pixels.

[0064] Additionally, the pixel array 110 includes a relatively small number of low-sensitivity pixels, resulting in a relatively low ratio of low-sensitivity pixels. Therefore, the pixel array 110 can include a relatively small number of low-sensitivity pixels. Since the images sensed by high-sensitivity pixels and the images sensed by low-sensitivity pixels have different properties, it is more advantageous in terms of image quality for most images to be sensed by high-sensitivity pixels; thus, the pixel array 110 includes the minimum ratio of low-sensitivity pixels required for HDR functionality.

[0065] Return to reference Figure 1 The row memory 200 may include volatile memory (e.g., DRAM, SRAM, etc.) and / or non-volatile memory (e.g., flash memory). The row memory 200 may have a capacity capable of storing image data corresponding to a predetermined number of rows. In this case, a row may refer to a row of the pixel array 110, and the predetermined number of rows may be less than the total number of rows in the pixel array 110. Therefore, the row memory 200 may be a row memory capable of storing image data corresponding to some rows of the pixel array 110, rather than a frame memory capable of storing image data corresponding to frames captured by the pixel array 110. In some implementations, the row memory 200 may also be replaced with a frame memory as needed.

[0066] Based on the control of ISP 300, row memory 200 can receive image data from image sensing device 100, store the received image data, and send the stored image data to ISP 400.

[0067] The ISP 300 can perform image processing on image data stored in the line memory 200. The ISP 300 can reduce noise in the image data and perform various types of image signal processing (e.g., gamma correction, color filter array interpolation, color matrix, color correction, color enhancement, lens distortion correction, etc.) to improve the image quality of the image data. Additionally, the ISP 300 can compress the image data created by performing image signal processing for image quality improvement, allowing the ISP 300 to create image files using the compressed image data. Alternatively, the ISP 300 can recover image data from an image file. In this case, the compression scheme for such image data can be a reversible or irreversible format. Representative examples of such compression formats include the Joint Picture Experts Group (JPEG) format and the JPEG 2000 format when using still images. Furthermore, when using moving images, multiple frames can be compressed according to the Moving Picture Experts Group (MPEG) standard, enabling the creation of moving image files. For example, image files can be created according to the Exif (Exchangeable Image File Format) standard.

[0068] To generate HDR images, the ISP 300 may include a gain processing unit 310 and an image combining unit 320.

[0069] The gain processing unit 310 can determine the gain to be calculated (multiplied) using pixel data from high-sensitivity pixels and / or pixel data from low-sensitivity pixels. The gain processing unit 310 can determine the gain based on the sensitivity difference between high-sensitivity and low-sensitivity pixels and other conditions, and can provide the determined gain to the image combining unit 320. The gain can be determined experimentally in advance based on sensitivity differences and other conditions, and can be stored in the gain processing unit 310. In some implementations, the gain processing unit 310 can store the experimentally determined gain in a table, so that the gain processing unit 310 can obtain the necessary gain by referring to the contents of the stored table.

[0070] The image combining unit 320 can use pixel data from high-sensitivity pixels and / or pixel data from low-sensitivity pixels to synthesize an HDR image corresponding to high dynamic range.

[0071] In some embodiments of the disclosed technology, the image combining unit 320 may perform interpolation and calculation between the pixel data of high-sensitivity pixels and the pixel data of low-sensitivity pixels based on whether the pixel data of high-sensitivity pixels is saturated, thereby forming an HDR image. In this case, interpolation may refer to the process of generating the pixel data of a corresponding pixel using the pixel data of at least one pixel adjacent to the corresponding pixel.

[0072] For example, when the pixel data of a high-sensitivity pixel is not saturated, the image combining unit 320 can determine the pixel data of the high-sensitivity pixel as HDR image data of the high-sensitivity pixel without changing it. Alternatively, the image combining unit 320 can determine the HDR image data as a value obtained by interpolating the pixel data of the high-sensitivity pixel relative to the low-sensitivity pixel corresponding to the high-sensitivity pixel.

[0073] When the pixel data of a high-sensitivity pixel is saturated, the image combination unit 320 can determine the value obtained by interpolating the pixel data of a low-sensitivity pixel corresponding to the high-sensitivity pixel as the HDR image data of the high-sensitivity pixel, and can determine the pixel data of a low-sensitivity pixel corresponding to the high-sensitivity pixel as the HDR image data without changing it. In this case, during the interpolation of the pixel data of the low-sensitivity pixel, a gain (e.g., 16) based on the sensitivity difference (e.g., 16 times) between the high-sensitivity pixel and the low-sensitivity pixel can be calculated (e.g., multiplied) using the pixel data of the low-sensitivity pixel.

[0074] In another embodiment, when at least one high-brightness area exists in a scene, the image combining unit 320 can generate an HDR image by adjusting the combination weights between the pixel data of high-sensitivity pixels and the pixel data of low-sensitivity pixels at the boundary between the at least one high-brightness area and the remaining area (i.e., low-brightness area). The operation of adjusting the combination weights will be described later. Figures 8 to 12 describe.

[0075] In another embodiment, the image combining unit 320 can generate an HDR image by calculating the sum of a high-sensitivity image corresponding to a set of pixel data of high-sensitivity pixels related to a scene and a low-sensitivity image corresponding to a set of pixel data of low-sensitivity pixels related to a scene. Since the ratio of low-sensitivity pixels in the pixel array 110 is lower than the ratio of high-sensitivity pixels in the pixel array 110, the low-sensitivity image can have a lower resolution than the high-sensitivity image. Therefore, the image combining unit 320 can first perform resolution conversion on the low-sensitivity image and can calculate the sum of the high-sensitivity image and the low-sensitivity image converted to have the same resolution as the high-sensitivity image.

[0076] In one implementation, resolution conversion of the low-sensitivity image can be performed before demosaicing. Demosaicing can be an operation that converts pixel data corresponding to one color (e.g., red, blue, or green) into pixel data corresponding to three colors (e.g., red, blue, and green). When demosaicing is performed on a Bayer pattern image (including pixel data corresponding to one color for each pixel), an RGB image (including pixel data corresponding to three colors for each pixel) can be formed. The reason for performing resolution conversion and demosaicing sequentially is to prevent distortion caused by this demosaicing from being included in the HDR image by performing resolution conversion on the Bayer pattern image corresponding to the original image.

[0077] The ISP 300 can send the image data (i.e., HDR image data) obtained through this image signal processing to the I / O interface 400.

[0078] In another embodiment, the gain processing unit 310 and the image combining unit 320 for generating HDR images may also be included in the image sensing device 100 instead of the ISP 300.

[0079] I / O interface 400 can communicate with host device 20 and can send image data from image signal processing (ISP) to host device 20. In some implementations, I / O interface 400 can be implemented as a Mobile Industry Processor Interface (MIPI), but is not limited thereto.

[0080] The host device 20 may be a processor (e.g., an application processor) for processing ISP image data received from the imaging device 10, a memory (e.g., a non-volatile memory) for storing ISP image data, or a display device (e.g., a liquid crystal display (LCD)) for visually displaying ISP image data.

[0081] Figures 5A to 5D This is a schematic diagram illustrating examples of how high-sensitivity pixels (HPX) and low-sensitivity pixels (LPX) are implemented in different ways based on differences in transmittance, according to some implementation methods of the disclosed technology.

[0082] Figure 5A This is an example diagram showing high-sensitivity pixels (HPX) and low-sensitivity pixels (LPX) that are adjacent to each other.

[0083] Reference Figure 5A The high-sensitivity pixel (HPX) and low-sensitivity pixel (LPX) may partially overlap with the substrate 510, and may include a photoelectric conversion element 520, a filter 530, a microlens 540 and a first light-blocking structure 550.

[0084] For example, substrate 510 can be a P-type or N-type block substrate, or it can be a substrate formed by growing a P-type or N-type epitaxial layer on a P-type block substrate, or it can be a substrate formed by growing a P-type or N-type epitaxial layer on an N-type block substrate.

[0085] The photoelectric conversion element 520 can be formed in the substrate 510. That is, the photoelectric conversion element 520 can generate and accumulate photocharge corresponding to the intensity of the incident light that has passed through the microlens 540 and the filter 530.

[0086] Filter 530 can selectively transmit light having a wavelength band to be transmitted (e.g., red, green, blue, magenta, yellow, cyan, infrared (IR) light, etc.). In this case, the wavelength band may refer to the wavelength band of light to be selectively transmitted by the corresponding filter. For example, each filter 530 may include a colored photosensitive material corresponding to a specific color, or may include alternating thin film layers. Filters included in pixel array 110 may be arranged to correspond to pixels arranged in a matrix array comprising multiple rows and columns, thereby forming a filter array.

[0087] Each microlens 540 can be formed above each filter 530 and can increase the focusing ability of the incident light, thereby increasing the light receiving (Rx) efficiency of the photoelectric conversion element 520.

[0088] A first light-blocking structure 550 may be disposed between a surface of the substrate 510 and the filter 530, such that at least a portion of the incident light that has penetrated the filter 530 in the low-sensitivity pixel (LPX) is blocked by the first light-blocking structure 550 and not transmitted to the photoelectric conversion element 520. The first light-blocking structure 550 may include at least one of a material having high light reflectivity (e.g., silver or aluminum) and a material having high light absorptivity (e.g., tungsten).

[0089] The total area of ​​a low-sensitivity pixel (LPX) can be defined as the sum of the aperture area of ​​the region without the first light-blocking structure 550 and the blocking area of ​​the region with the first light-blocking structure 550. The light transmittance of the low-sensitivity pixel (LPX) can be determined based on the ratio of the blocking area to the aperture area.

[0090] A high-sensitivity pixel (HPX) excluding the first light-blocking structure 550 may have higher light transmittance than a low-sensitivity pixel (LPX) including the first light-blocking structure 550.

[0091] When incident light of the same intensity is incident on a high-sensitivity pixel (HPX) and a low-sensitivity pixel (LPX), the intensity of the light transmitted to the photoelectric conversion element 520 of the low-sensitivity pixel (LPX) may be less than the intensity of the light transmitted to the photoelectric conversion element 520 of the high-sensitivity pixel (HPX).

[0092] Furthermore, the light intensity transmitted to the photoelectric conversion element 520 of the low-sensitivity pixel (LPX) can increase at a relatively low slope in response to an increase in incident light intensity. The light intensity transmitted to the photoelectric conversion element 520 of the high-sensitivity pixel (HPX) can increase at a relatively high slope in response to an increase in incident light intensity.

[0093] Since each of the light intensity transmitted to the photoelectric conversion element 520 of the low-sensitivity pixel (LPX) and the light intensity transmitted to the photoelectric conversion element 520 of the high-sensitivity pixel (HPX) is converted into a pixel signal, the response of the low-sensitivity pixel (LPX) can follow... Figure 4 The response of the low-sensitivity pixel is shown, and the response of the high-sensitivity pixel (HPX) follows the same pattern. Figure 4 The response of the high-sensitivity pixel is shown.

[0094] although Figure 5A The first light-blocking structure 550 shown is positioned at the edge of the low-sensitivity pixel (LPX), but other implementations are also possible. For example, the first light-blocking structure 550 can be positioned at any location of the low-sensitivity pixel (LPX) corresponding to some portion of the low-sensitivity pixel (LPX).

[0095] Based on some implementations of the disclosed technology, the image sensing device 100 can simultaneously implement low-sensitivity pixels and high-sensitivity pixels in only one pixel array 110, so that an HDR image can be formed (or generated) using only one image.

[0096] Figure 5B This is another example of a diagram showing high-sensitivity pixels (HPX) and low-sensitivity pixels (LPX) that are adjacent to each other.

[0097] Reference Figure 5B The high-sensitivity pixel (HPX) and low-sensitivity pixel (LPX) may partially overlap with the substrate 510, and may include a photoelectric conversion element 520, a filter 530, a microlens 540, and a second light-blocking structure 560.

[0098] Figure 5B The substrate 510, photoelectric conversion element 520, filter 530, and microlens 540 shown are... Figure 5A The others are basically the same, so for the sake of convenience, these redundant descriptions will be omitted in this article.

[0099] and Figure 5A The first light-blocking structure 550 shown is different. Figure 5BThe second light-blocking structure 560 shown can be disposed over the entire area of ​​the low-sensitivity pixel (LPX) at a location between one surface of the substrate 510 and the filter 530. Additionally, the second light-blocking structure 560 may have a predetermined thickness to prevent at least a portion of the incident light that has penetrated the filter 530 from being transmitted to the photoelectric conversion element 520. The second light-blocking structure 560 may have a smaller thickness than the first light-blocking structure 550.

[0100] The second light-blocking structure 560 may include at least one of a material with high light reflectivity (e.g., silver or aluminum) and a material with high light absorption (e.g., tungsten).

[0101] High-sensitivity pixels (HPX) that do not include the second light-blocking structure 560 may have higher light transmittance than low-sensitivity pixels (LPX) that include the second light-blocking structure 560.

[0102] Figure 5C This is another example of a diagram showing high-sensitivity pixels (HPX) and low-sensitivity pixels (LPX) that are adjacent to each other.

[0103] Reference Figure 5C The high-sensitivity pixel (HPX) and low-sensitivity pixel (LPX) may partially overlap with the substrate 510, and may include a photoelectric conversion element 520, a filter 530, a microlens 540 and a third light-blocking structure 570.

[0104] Figure 5C The substrate 510, photoelectric conversion element 520, filter 530, and microlens 540 shown are... Figure 5A The others are basically the same, so for the sake of convenience, these redundant descriptions will be omitted in this article.

[0105] Figure 5C The third light-blocking structure 570 shown can be positioned between the filter 530 and the microlens 540 and over the entire area of ​​the low-sensitivity pixel (LPX). For example, the third light-blocking structure 570 can be a neutral density (ND) filter that blocks at least a portion of light across the entire wavelength band, regardless of color.

[0106] High-sensitivity pixels (HPX) that do not include the third light-blocking structure 570 can have higher light transmittance than low-sensitivity pixels (LPX) that include the third light-blocking structure 570.

[0107] Figure 5D This is another example of a diagram showing high-sensitivity pixels (HPX) and low-sensitivity pixels (LPX) that are adjacent to each other.

[0108] Reference Figure 5DHigh-sensitivity pixels (HPX) and low-sensitivity pixels (LPX) may partially overlap with substrate 510, and may include photoelectric conversion element 520, filters 530 and 530', and microlens 540. In this implementation, Figure 5D The low-sensitivity pixel (LPX) shown may not include a separate light-blocking structure.

[0109] Figure 5D The substrate 510, photoelectric conversion element 520, filter 530, and microlens 540 shown are... Figure 5A The others are basically the same, so for the sake of convenience, these redundant descriptions will be omitted in this article.

[0110] The filter 530' included in the low-sensitivity pixel (LPX) may have a lower transmittance than the filter 530 included in the high-sensitivity pixel (HPX). Each of the filters 530 and 530' may include a colored photosensitive material corresponding to a specific color, or may include alternately arranged thin film layers. The photosensitive material included in the filter 530' may have a higher concentration than the photosensitive material included in the filter 530, or the number of thin film layers included in the filter 530' may be higher than the number of thin film layers included in the filter 530.

[0111] Therefore, a high-sensitivity pixel (HPX) including filter 530 can have higher light transmittance than a low-sensitivity pixel (LPX) including filter 530'.

[0112] Although for the sake of convenience, Figures 5A to 5D Various implementations of high-sensitivity pixels (HPX) and low-sensitivity pixels (LPX) are shown, but the scope or spirit of the disclosed technology is not limited thereto. It should be noted that high-sensitivity pixels (HPX) and low-sensitivity pixels (LPX) can also be implemented using any structure that can adjust the transmittance.

[0113] In addition, high-sensitivity pixels (HPX) and low-sensitivity pixels (LPX) can be designed to have a transmittance difference corresponding to a predetermined sensitivity difference (e.g., 16 times).

[0114] Figure 6A and Figure 6B This diagram illustrates examples of how high-sensitivity pixels and low-sensitivity pixels are implemented in different ways based on differences in circuit structure, using some implementation methods of the disclosed technology.

[0115] Figure 6A This is a circuit diagram illustrating an example of the equivalent circuitry for a high-sensitivity pixel (HPX).

[0116] Reference Figure 6AThe high-sensitivity pixel (HPX) may include a first photoelectric conversion element PD1, a first transmission transistor TX1, a first reset transistor RX1, a first floating diffusion region FD1, a first driving transistor DX1, and a first selection transistor SX1. That is, the high-sensitivity pixel (HPX) may have a 4TR (four-transistor) pixel structure.

[0117] The first photoelectric conversion element PD1 can generate and accumulate photocharge corresponding to the intensity of the incident light. For example, the first photoelectric conversion element PD1 can be implemented as a photodiode, a phototransistor, a light gate, a pinned photodiode, or a combination thereof.

[0118] If the first photoelectric conversion element PD1 is implemented as a photodiode, the first photoelectric conversion element PD1 can be a region in a substrate including a first conductive impurity (e.g., a P-type impurity) doped with a second conductive impurity (e.g., an N-type impurity).

[0119] A first transmission transistor TX1 may be connected between a first photoelectric conversion element PD1 and a first floating diffusion region FD1. The first transmission transistor TX1 may be turned on or off in response to a first transmission control signal TG1. If the first transmission transistor TX1 is turned on, the photocharge accumulated in the first photoelectric conversion element PD1 may be transferred to the first floating diffusion region FD1.

[0120] The first reset transistor RX1 may be disposed between the first floating diffusion region FD1 and the power supply voltage (VDD), and the voltage of the first floating diffusion region FD1 may be reset to the power supply voltage (VDD) in response to the first reset control signal RG1.

[0121] The first floating diffusion region FD1 can accumulate photocharge received from the first transmission transistor TX1. For example, the first floating diffusion region FD1 can be a region in a substrate (e.g., a P-type substrate) including a first conductive impurity doped with a second conductive impurity (e.g., an N-type impurity). In this case, the substrate and the doped region can be modeled as a first capacitor (C1) acting as a junction capacitor.

[0122] The first driving transistor DX1 can be connected between the power supply voltage (VDD) and the first selection transistor SX1, amplify the potential change of the first floating diffusion region FD1 that has received the photocharge accumulated in the first photoelectric conversion element PD1, and transmit the amplification result to the first selection transistor SX1.

[0123] The first selection transistor SX1 can be connected between the first driving transistor DX1 and the output signal line, and can be turned on by the first selection control signal SEL1 so that the first selection transistor SX1 can output the electrical signal received from the first driving transistor DX1 as the first pixel signal PS1.

[0124] Figure 6B This is a circuit diagram illustrating an example of the equivalent circuitry for a low-sensitivity pixel (LPX).

[0125] Reference Figure 6B The low-sensitivity pixel (LPX) may include a second photoelectric conversion element PD2, a second reset transistor RX2, a second floating diffusion region FD2, a second driving transistor DX2, and a second selection transistor SX2. That is, the low-sensitivity pixel (LPX) may have a 3TR (three-transistor) pixel structure.

[0126] Figure 6B The basic structure and function of the second photoelectric conversion element PD2, the second reset transistor RX2, the second floating diffusion region FD2, the second driving transistor DX2, and the second selection transistor SX2 shown are similar to those of the second photoelectric conversion element PD2, the second reset transistor RX2, the second floating diffusion region FD2, the second driving transistor DX2, and the second selection transistor SX2. Figure 6A The first photoelectric conversion element PD1, the first reset transistor RX1, the first floating diffusion region FD1, the first drive transistor DX1, and the first selection transistor SX1 shown are similar to those, therefore, for simplicity, these redundant descriptions will be omitted here. For ease of description, the following will focus on those similar to... Figure 6A To describe different characteristics Figure 6B The above-described constituent elements are shown.

[0127] The low-sensitivity pixel (LPX) may not include a separate transmission transistor, and the second photoelectric conversion element PD2 may be directly connected to the source of the second reset transistor RX2 and the gate of the second drive transistor DX2.

[0128] Furthermore, the gate of the second reset transistor RX2 can be connected to the power supply voltage (VDD), and the second reset transistor RX2 can operate in the transistor region. That is, as photocharge corresponding to the incident light intensity is generated and accumulated in the second photoelectric conversion element PD2, the voltage of the second floating diffusion region FD2 can decrease proportionally to the amount of photocharge. Therefore, a current proportional to the voltage drop in the second floating diffusion region FD2 can flow in each of the second reset transistor RX2 and the second photoelectric conversion element PD2, and the second pixel signal PS2 corresponding to the magnitude of the current can be output through the second drive transistor DX2 and the second selection transistor SX2.

[0129] As described above, the operation of the second reset transistor RX2, configured to generate a current corresponding to the amount of photocharge generated by the second photoelectric conversion element PD2, can be defined as a logarithmic operation. Due to this logarithmic operation, even when an incident light intensity that could cause saturation in the first photoelectric conversion element PD1 or the first floating diffusion region FD1 within the high-sensitivity pixel (HPX) is provided, the low-sensitivity pixel (LPX) can generate an effective second pixel signal PS2 corresponding to such incident light intensity. In this case, the second reset transistor RX2 will hereinafter be referred to as a logarithmic transistor.

[0130] According to one implementation, the low-sensitivity pixel (LPX) may not generate a reference signal, so that the correlation double sampling (CDS) operation of the second pixel signal PS2 generated by the low-sensitivity pixel (LPX) can be omitted.

[0131] Strictly speaking, Figure 6A and Figure 6B The high-sensitivity pixels (HPX) and low-sensitivity pixels (LPX) shown are not used to establish a sensitivity difference between the two pixels. For ease of description and better understanding of the disclosed techniques, it may be more preferable to refer to each pixel as a normal pixel and as an HDR pixel for HDR image formation.

[0132] Figure 6A and Figure 6B The illustrated implementation method can be compared with Figures 5A to 5D The implementation methods can be combined without mutual exclusion. For example, Figure 5A The high-sensitivity pixels (HPX) and low-sensitivity pixels (LPX) shown in the cross-sectional view can be respectively made from Figure 6A circuit structure and Figure 6B The circuit structure is represented.

[0133] Figure 7 It is a graph showing an example of the response of some implementations based on the disclosed technology to the illuminance of incident light applied to each of the low-sensitivity green pixel, low-sensitivity red pixel, and low-sensitivity blue pixel.

[0134] Reference Figure 7 Associated with low-sensitivity green pixels (GP_L), low-sensitivity red pixels (RP_L), and low-sensitivity blue pixels (BP_L), the following are shown: the response of the low-sensitivity green pixel (GP_L) to the incident light intensity applied to the low-sensitivity green pixel (GP_L), the response of the low-sensitivity red pixel (RP_L) to the incident light intensity applied to the low-sensitivity red pixel (RP_L), and the response of the low-sensitivity blue pixel (BP_L) to the incident light intensity applied to the low-sensitivity blue pixel (BP_L).

[0135] The low-sensitivity green pixel (GP_L) may include a green color filter, the low-sensitivity red pixel (RP_L) may include a red color filter, and the low-sensitivity blue pixel (BP_L) may include a blue color filter. Due to the characteristics of each color filter, the sensitivity of the low-sensitivity green pixel (GP_L) may be higher (approximately twice) than the sensitivity of the low-sensitivity red pixel (RP_L) or the low-sensitivity blue pixel (BP_L). In this case, it is assumed that the sensitivity of the low-sensitivity red pixel (RP_L) is substantially the same as that of the low-sensitivity blue pixel (BP_L).

[0136] Available from Figure 7 It can be seen that the response of the low-sensitivity green pixel (GP_L) can saturate while increasing at a relatively high slope in response to the increase of incident light intensity, and the responses of the low-sensitivity red pixel (RP_L) and the low-sensitivity blue pixel (BP_L) can saturate while increasing at a relatively low slope in response to the increase of incident light intensity.

[0137] The sensitivity and response differences between low-sensitivity green pixels (GP_L), low-sensitivity red pixels (RP_L), and low-sensitivity blue pixels (BP_L) can cause problems with altered responses in pixels of different colors that are configured to receive the same intensity of incident light. Specifically, noise may appear in HDR images under high-illuminance conditions where low-sensitivity pixels should be used to generate HDR images.

[0138] Therefore, the responses of the low-sensitivity green pixel (GP_L), the low-sensitivity red pixel (RP_L), and the low-sensitivity blue pixel (BP_L) must be identical to each other. To this end, the sensitivity of the low-sensitivity green pixel (GP_L) can be designed to be lower than that of the low-sensitivity red pixel (RP_L) or the low-sensitivity blue pixel (BP_L). For example, when each of the low-sensitivity green pixel (GP_L), low-sensitivity red pixel (RP_L), and low-sensitivity blue pixel (BP_L) has... Figure 5A When the low-sensitivity pixel (LPX) structure is shown, the size of the opening of the low-sensitivity green pixel (GP_L) can be smaller than the size of the opening of the low-sensitivity red pixel (RP_L) or the opening of the low-sensitivity blue pixel (BP_L).

[0139] Figure 8 This is a diagram illustrating an example of a frame generated by an image sensing device 100 based on some implementations of the disclosed technology.

[0140] Reference Figure 8Multiple pixels in the pixel array 110 can simultaneously sense a scene, and the image sensing device 100 can provide the ISP 300 with a set of pixel data for sensing a scene via the row memory 200. In this case, the set of pixel data for sensing a scene can be defined as... Figure 8 The frame shown is (FR).

[0141] The image combining unit 320 of the ISP 300 can determine high-brightness areas based on pixel data of high-sensitivity pixels included in a frame (FR).

[0142] A high-brightness region can refer to an area where the proportion of high-sensitivity pixels with saturated pixel data for each individual pixel group is higher than a predetermined proportion (e.g., 90%). For example, a pixel group may include... Figure 3 The 64 pixels shown are not the only possible implementation; other methods are also possible.

[0143] Assuming that the ratio of high-sensitivity pixels with saturated pixel data in each unit pixel group in each of the first high-brightness regions S1 and the second high-brightness regions S2 is higher than a predetermined ratio, the image combining unit 320 can determine each of the first high-brightness regions S1 and the second high-brightness regions S2 as a high-brightness region.

[0144] In contrast, assuming that the ratio of high-sensitivity pixels with saturated pixel data in each unit pixel group in the remaining areas other than the first high-brightness area S1 and the second high-brightness area S2 is equal to or less than a predetermined ratio, the image combination unit 320 can determine that each remaining area other than the first high-brightness area S1 and the second high-brightness area S2 is a low-brightness area.

[0145] Additionally, it is assumed that the overall brightness gradually increases from the left to the right of the frame (FR). For example, an increase in overall brightness could indicate an increase in the average pixel data of high-sensitivity pixels.

[0146] from Figure 8 It can be seen that when there are one or more high-brightness regions S1 and S2 in a frame (FR), the image combining unit 320 can use the pixel data of low-sensitivity pixels in at least one of the high-brightness regions S1 and S2, and can use the pixel data of high-sensitivity pixels in the remaining regions (i.e., low-brightness regions) other than one or more high-brightness regions S1 and S2 to generate an HDR image.

[0147] Specifically, since most high-sensitivity pixels are saturated in at least one of the high-brightness regions S1 and S2, the image combining unit 320 can apply a first value obtained when calculating the gain (e.g., 16) caused by the sensitivity difference (e.g., a 16-fold sensitivity difference) between high-sensitivity pixels and low-sensitivity pixels, along with a second value obtained by interpolating the first value, to the process of determining the HDR image of at least one of the high-brightness regions S1 and S2. In at least one of the high-brightness regions S1 and S2, the ratio of low-sensitivity pixels is less than the ratio of high-sensitivity pixels, and the image captured by the low-sensitivity pixels has a lower resolution. As a result, most high-sensitivity pixels are saturated (i.e., in a state with little image information), making it potentially more advantageous to use the pixel data of the low-sensitivity pixels to generate the HDR image.

[0148] Furthermore, since most of the high-sensitivity pixels are unsaturated in the remaining areas (i.e., low-brightness areas) outside one or more high-brightness areas S1 and S2, the image combining unit 320 can apply both the pixel data of the high-sensitivity pixels and the values ​​obtained by interpolation of the pixel data to the processing of the HDR image that determines the remaining areas (i.e., low-brightness areas) outside one or more high-brightness areas S1 and S2.

[0149] Figure 9 It is a graph showing the comparison results between pixel data and noise for high-sensitivity pixels and low-sensitivity pixels based on some implementations of the disclosed technology.

[0150] Reference Figure 9 The pixel data of high-sensitivity pixels (HPX pixel data) and low-sensitivity pixels (LPX pixel data) are depicted based on the intensity (or illuminance) of the incident light. Figure 9 The upper side.

[0151] In this scenario, assuming a sensitivity difference of 16 times between high-sensitivity and low-sensitivity pixels, the gain can be multiplied by the pixel data of the low-sensitivity pixel by 16 during the processing to form an HDR image. For example... Figure 9 As depicted on the upper side, the gain control low-sensitivity pixel data "LPX pixel data (gain control)" obtained when the gain is multiplied by the pixel data of the low-sensitivity pixel can be formed into a shape that magnifies the pixel data of the low-sensitivity pixel by 16 times in the Y-axis direction.

[0152] Assuming that the pixel data of the high-sensitivity pixel and the low-sensitivity pixel are saturated by 1024, even when the high-sensitivity pixel is saturated in response to an increase in incident light intensity (indicated by "HPX saturation"), the dynamic range can be extended by gain control of the pixel data of the low-sensitivity pixel to the incident light intensity corresponding to the pixel data with a dynamic range of 16384.

[0153] When generating an HDR image, it is more preferable to use pixel data of high-sensitivity pixels with high SNR (signal-to-noise ratio) under illuminance conditions where the illuminance is less than the incident light intensity that saturates the high-sensitivity pixels. More preferably, under illuminance conditions where the illuminance is higher than the incident light intensity that saturates the high-sensitivity pixels, the gain of pixel data of low-sensitivity pixels is controlled and used instead of the pixel data of saturated high-sensitivity pixels.

[0154] In comparison, Figure 9 The lower side shows the standard deviation of noise for each of the pixel data of the high-sensitivity pixel, the pixel data of the low-sensitivity pixel, and the pixel data of the low-sensitivity pixel controlled by gain, based on the incident light intensity. In this case, the standard deviation of noise (hereinafter referred to as noise standard deviation) may refer to the intensity of shot noise. Each pixel data may include shot noise caused by the random characteristics of photons constituting the incident light. Since shot noise is caused by photons, it may increase in response to an increase in incident light intensity.

[0155] Therefore, as the intensity of incident light increases, the noise standard deviation of pixel data for high-sensitivity pixels and the noise standard deviation of pixel data for low-sensitivity pixels can increase until each pixel is saturated (in the saturated state, photons are not introduced into each pixel, so the noise standard deviation is considered meaningless).

[0156] Furthermore, under the same incident light intensity, the noise standard deviation of pixel data for a high-sensitivity pixel with relatively high sensitivity can be greater than the noise standard deviation of pixel data for a low-sensitivity pixel with relatively low sensitivity. On the other hand, since the gain control of pixel data for a low-sensitivity pixel is obtained by multiplying the pixel data of the low-sensitivity pixel by a predetermined gain (e.g., 16 times), the noise standard deviation can also be increased by the predetermined gain (e.g., 16 times) compared to the pixel data of the low-sensitivity pixel.

[0157] As described above, under illumination conditions where the illuminance is less than the incident light intensity that saturates the high-sensitivity pixels, pixel data from the high-sensitivity pixels can be used to generate HDR images. Under illumination conditions where the illuminance is greater than the incident light intensity that saturates the high-sensitivity pixels, when generating HDR images using pixel data from low-sensitivity pixels with gain control, abrupt differences in shot noise may occur based on the incident light intensity that saturates the high-sensitivity pixels.

[0158] If an HDR image is generated using pixel data from high-sensitivity pixels within a portion of a frame (FR), and if an HDR image is generated using pixel data from low-sensitivity pixels within another adjacent region of the frame (FR), image quality degradation may occur near the boundary between the two regions due to abrupt differences in shot noise.

[0159] Figure 10 This is a diagram illustrating an example of a method for establishing combined weights based on some implementations of the disclosed technology.

[0160] Reference Figure 10 When Figure 8 As the overall brightness gradually increases from the left to the right of the frame (FR), the image combining unit 320 can define a portion of the right end of the frame (FR) as a first transition region TZ1. In the first transition region TZ1, the ratio of high-sensitivity pixels with saturated pixel data for each unit pixel group does not exceed a predetermined ratio, but the minimum ratio of high-sensitivity pixels may be equal to or higher than a minimum ratio (e.g., 70%), and a directionality of increasing overall brightness (e.g., left → right) may occur.

[0161] In the first transition region TZ1, the image combining unit 320 can use a first combining weight to combine the pixel data of low-sensitivity pixels and the pixel data of high-sensitivity pixels to generate an HDR image. The first combining weight can be a value ranging from 0 to 1, and can refer to the weight assigned to the pixel data of the high-sensitivity pixels.

[0162] Specifically, in the first transition region TZ1, the image combining unit 320 can determine the sum of the first value and the second value as the HDR image of the low-sensitivity pixels. In this case, the first value is obtained when the first combining weight is multiplied by the value obtained by interpolation of the pixel data of the high-sensitivity pixels, and the second value is obtained when the value (represented by "1 - first combining weight") is multiplied by the value obtained by arithmetic calculation between the gain and the pixel data of the low-sensitivity pixels.

[0163] According to one embodiment, the first combined weight can be gradually changed in the first transition region TZ1. For example, at the left end of the first transition region TZ1, the first combined weight can be set to 1. The first combined weight can be gradually decreased in the direction from the left end to the right end within the first transition region TZ1, so that at the right end of the first transition region TZ1, the first combined weight can be set to 0.1.

[0164] By establishing a combined weighting method, pixel data of low-sensitivity pixels can be partially reflected in regions where the ratio of saturated high-sensitivity pixels is relatively high, thereby obtaining HDR images with extended dynamic range. Furthermore, the first combined weights can be changed according to the average brightness, thereby obtaining HDR images with optimized dynamic range and SNR.

[0165] Figure 11 This is another example of a method for establishing combined weights based on some implementations of the disclosed technology.

[0166] Reference Figure 11 When Figure 8 When the image includes a first high-brightness region S1 and a second high-brightness region S2, the image combining unit 320 can define the low-brightness region (or the region surrounding the first high-brightness region S1) adjacent to the boundary between the low-brightness region and the first high-brightness region S1 as the second transition region TZ2, and can define the low-brightness region (or the region surrounding the second high-brightness region S2) adjacent to the boundary between the low-brightness region and the second high-brightness region S2 as the third transition region TZ3.

[0167] As described above, when at least one high-brightness region S1 and S2 exists in a frame (FR), the image combining unit 320 can use the pixel data of low-sensitivity pixels in at least one high-brightness region S1 and S2, and can use the pixel data of high-sensitivity pixels in the remaining regions (or low-brightness regions) outside at least one high-brightness region S1 and S2 to generate an HDR image. In this case, there can be a very large difference in shot noise between the pixel data of low-sensitivity pixels and the pixel data of high-sensitivity pixels.

[0168] Therefore, the image combining unit 320 can define the region surrounding the first high-brightness region S1 as the second transition region TZ2. In the second transition region TZ2, the image combining unit 320 can use a second combining weight to combine pixel data from low-sensitivity pixels and pixel data from high-sensitivity pixels to form an HDR image. The second combining weight can be a value ranging from 0 to 1, and can refer to the weight assigned to the pixel data of the high-sensitivity pixels.

[0169] Additionally, the image combining unit 320 can define the region surrounding the second high-brightness region S2 as the third transition region TZ3. Within the third transition region TZ3, the image combining unit 320 can use a third combining weight to combine pixel data from low-sensitivity pixels and pixel data from high-sensitivity pixels to form an HDR image. The third combining weight can be a value ranging from 0 to 1, and can refer to the weight assigned to the pixel data of the high-sensitivity pixels.

[0170] The method of generating HDR images by combining pixel data based on a second or third combination weight and Figure 10 The method shown for generating HDR images by combining pixel data based on a first combination weight is essentially the same, so for ease of description, these redundant descriptions will be omitted here.

[0171] According to one embodiment, the second combined weight or the third combined weight can be gradually changed in the second transition region TZ2 or the third transition region TZ3. For example, at the outer end of the second transition region TZ2 (i.e., at the position furthest from the boundary between the first high-brightness region S1 and the low-brightness region), the second combined weight can be set to 1. The second combined weight can be gradually decreased in the direction from the outer end to the inner end of the second transition region TZ2, so that the second combined weight can become zero at the inner end of the second transition region TZ2 (i.e., at the boundary between the first high-brightness region S1 and the low-brightness region). In addition, the third combined weight can be set to 1 at the outer end of the third transition region TZ3, and can be gradually decreased in the direction from the outer end to the inner end of the third transition region TZ3, so that the third combined weight can become zero at the inner end of the third transition region TZ3.

[0172] According to the method of establishing combined weights, pixel data of low-sensitivity pixels and pixel data of low-sensitivity pixels in the region located around the boundary between the high-brightness region and the remaining region outside the high-brightness region can be combined with each other using specific combined weights, and the second and third combined weights can be changed according to the distance from the high-brightness region, thereby forming an HDR image that reduces image quality degradation caused by noise differences.

[0173] According to one embodiment, the first width W1 of the second transition region TZ2 and the second width W2 of the third transition region TZ3 can be determined by the average value of pixel data of high-sensitivity pixels located adjacent to the first high-brightness region S1 and the second high-brightness region S2. Here, the high-sensitivity pixels adjacent to each of the first high-brightness region S1 and the second high-brightness region S2 can refer to high-sensitivity pixels included within a predetermined distance from each of the first high-brightness region S1 and the second high-brightness region S2. Each of the first width W1 of the second transition region TZ2 and the second width W2 of the third transition region TZ3 can decrease as the average value of the pixel data of the corresponding high-sensitivity pixel decreases, and can increase as the average value of the pixel data of the corresponding high-sensitivity pixel increases. This is because when the brightness difference between each of the first high-brightness region S1 and the second high-brightness region S2 and its adjacent regions is relatively large, the sense of incongruity caused by the difference in shot noise can be relatively small. For example, from Figure 11 It can be seen that the first width W1 of the second transition region TZ2 can be smaller than the second width W2 of the third transition region TZ3.

[0174] Figure 12 This is another example of a method for establishing combined weights based on some implementations of the disclosed technology.

[0175] Reference Figure 12 When Figure 8 When the frame (FR) shown includes a first high-brightness region S1 and a second high-brightness region S2, the image combining unit 320 can determine a portion of the region surrounding the second high-brightness region S2 as a fourth transition region TZ4.

[0176] That is, the image combining unit 320 can define a portion of the region surrounding the second high-brightness region S2 as the fourth transition region TZ4, but not the region surrounding the first high-brightness region S1. In the fourth transition region TZ4, the image combining unit 320 can use a fourth combining weight to combine pixel data from low-sensitivity pixels and pixel data from high-sensitivity pixels to form an HDR image. The fourth combining weight can be a value ranging from 0 to 1, and can refer to the weight assigned to the pixel data of the high-sensitivity pixels. The method of generating an HDR image by combining pixel data based on the fourth combining weight is similar to... Figure 10 The method shown for generating HDR images by combining pixel data based on a first combination weight is essentially the same, so for ease of description, these redundant descriptions will be omitted in this paper.

[0177] According to one implementation, the fourth combined weight can be gradually changed within the fourth transition region TZ4. For example, the fourth combined weight can be set to 1 at the outer end of the fourth transition region TZ4 and can be gradually decreased in the direction from the outer end to the inner end of the fourth transition region TZ4, so that the fourth combined weight becomes zero at the inner end of the fourth transition region TZ4.

[0178] Unlike Figure 11 For the first high-brightness region S1 and the second high-brightness region S2, the image combining unit 320 may only determine a portion of the region surrounding the second high-brightness region S2 as the fourth transition region TZ4.

[0179] That is, the image combining unit 320 can set a transition region only for regions where the average pixel data of each adjacent high-sensitivity pixel in the first high-brightness region S1 and the second high-brightness region S2 is higher than a predetermined threshold. Figure 12 In the example, the average value of the pixel data of some high-sensitivity pixels corresponding to the fourth transition region TZ4 among each of the high-sensitivity pixels adjacent to the first high-brightness region S1 and the second high-brightness region S2 may exceed a predetermined threshold.

[0180] As described above, the transition region is set only in areas meeting specific conditions within the outer region of the brightness region for the following reason. When the brightness difference between each of the first high-brightness region S1 and the second high-brightness region S2 and its adjacent region is relatively large, the sense of incongruity caused by shot noise differences is relatively small. As a result, the method for preventing SNR degradation caused by the combination of pixel data from low-sensitivity pixels and pixel data from high-sensitivity pixels can be considered more effective than the method for mitigating noise differences.

[0181] Figure 13 This is a diagram illustrating an example of the arrangement of high-sensitivity pixels and low-sensitivity pixels based on some implementations of the disclosed technology.

[0182] Reference Figure 13 This diagram illustrates an example of the arrangement of high-sensitivity pixels (HPX) and low-sensitivity pixels (LPX). Specifically, three HPX pixels and one LPX pixel can be arranged in a (2×2) matrix (i.e., an identity matrix). Although the LPX pixel is positioned corresponding to the second row and second column of the (2×2) matrix, other implementations are possible, and it should be noted that the LPX pixel can also be positioned anywhere within the (2×2) matrix. Furthermore, the HPX pixels and LPX pixels can comprise the same type of filter.

[0183] Each of the high-sensitivity pixels (HPX) and low-sensitivity pixels (LPX) can independently include a photoelectric conversion element and a transmission transistor, and the remaining structure can be implemented as a shared pixel structure shared by the four pixels. That is, the high-sensitivity pixels (HPX) and low-sensitivity pixels (LPX) can share only one floating diffusion region. Figure 13 ("one FD" in the context).

[0184] Shared pixels composed of high-sensitivity pixels (HPX) and low-sensitivity pixels (LPX) can operate in either quaternary mode or binning mode. In this case, quaternary mode refers to a mode in which the pixel signals corresponding to the photocharge generated in the high-sensitivity pixel (HPX) and the pixel signals corresponding to the photocharge generated in the low-sensitivity pixel (LPX) are output sequentially at different time points. Binning mode refers to a mode in which the pixel signals corresponding to all photocharges generated in each of the high-sensitivity pixel (HPX) and low-sensitivity pixel (LPX) are output at once. Although binning mode may be disadvantageous in terms of resolution compared to quaternary mode (resolution reduced to 1 / 4), binning mode can directly obtain the pixel data corresponding to all photocharges generated in each of the high-sensitivity pixel (HPX) and low-sensitivity pixel (LPX), thereby generating an HDR image without performing a separate image combining operation.

[0185] When the shared pixel operates in quad mode, the ISP 300 can apply the gain corresponding to the sensitivity difference between the low-sensitivity pixel (LPX) and the high-sensitivity pixel (HPX) to pixel data generated independently based on the photocharge generated in the low-sensitivity pixel (LPX).

[0186] However, when the shared pixel operates in merge mode, the pixel signal of the shared pixel includes not only the component corresponding to the photocharge generated by the low-sensitivity pixel (LPX) but also the component corresponding to the photocharge generated in the high-sensitivity pixel (HPX). As a result, the ISP 300 cannot apply the gain corresponding to the sensitivity difference between the low-sensitivity pixel (LPX) and the high-sensitivity pixel (HPX) to the pixel data of the shared pixel without altering it.

[0187] Figure 14 It is a graph showing the change in pixel data of a shared pixel according to the intensity of incident light. Figure 15 It is a graph showing the comparison results between pixel data of shared pixels and pixel data of gain control pixels.

[0188] Figure 14 The graph shown illustrates how pixel data changes according to incident light intensity for a shared pixel configured to include three high-sensitivity pixels (HPX) and a single low-sensitivity pixel (LPX) and operating in merge mode.

[0189] The pixel data of the shared pixel can increase with the increase of the intensity of the incident light at a first slope, and can have a first value (A) under the first illuminance (ST_HPX) saturation of the high-sensitivity pixel (HPX).

[0190] Subsequently, only the low-sensitivity pixel (LPX) can generate photocharge corresponding to the incident light intensity, and the pixel data of the shared pixel can increase at a second slope less than the first slope as the incident light intensity increases, and can have a second value (S) under a second illuminance (ST_LPX) where the low-sensitivity pixel (LPX) is also saturated.

[0191] Due to the presence of low-sensitivity pixels (LPX), the dynamic range can be extended to the second illuminance (ST_LPX). Compared to an ideal pixel where pixel data increases at a first slope to a third value (B) until reaching the second illuminance (ST_LPX) based on the increase in incident light intensity, the pixel data of a shared pixel can increase to a second value (S) at a second slope (not the first slope) based on the increase in incident light intensity during the time period from the first illuminance (ST_HPX) to the second illuminance (ST_LPX).

[0192] Therefore, in order to make the pixel data representation of the shared pixel correspond to the pixel data of the ideal pixel, the ISP 300 for the time period between the first illuminance (ST_HPX) and the second illuminance (ST_LPX) can perform a calculation (e.g., multiplication) between the pixel data of the shared pixel and the gain that applies the sensitivity difference between the low-sensitivity pixel (LPX) and the high-sensitivity pixel (HPX).

[0193] According to one implementation, the ISP 300 may apply a gain "1" to pixel data of a shared pixel (having a range from 0 to A). For pixel data (X) of a shared pixel (where X is higher than A and less than S), the gain (e.g., 16) resulting from the sensitivity difference between the low-sensitivity pixel (LPX) and the high-sensitivity pixel (HPX) is multiplied by the value obtained by subtracting a second value (S) from the pixel data (X) of the shared pixel, and the sum of the first value (A) and the result of this multiplication is calculated to compute the gain control pixel data. That is, the gain control pixel data of the shared pixel may be represented by A+G*(XA) (where G is the gain (e.g., 16) based on the sensitivity difference between HPX and LPX).

[0194] Figure 15 It is a graph showing the comparison results between pixel data of shared pixels and gain-controlled pixel data based on some implementations of the disclosed technology.

[0195] Based on the example above, the ISP 300 can apply a gain of 1 to pixel data of shared pixels with a range of 0 to A (A: first value), and the gain control pixel data of shared pixels with a range of 0 to A can be increased by a slope of 1.

[0196] Additionally, since the ISP 300 applies a gain of 16 to the pixel data of shared pixels having a range of A to S (where A is the first value and S is the second value), the gain-controlled pixel data can be increased at a slope of 16 relative to the pixel data of shared pixels having a range of A to S (where A is the first value and S is the second value).

[0197] The ISP 300 can use the gain of pixel data suitable for shared pixels operating in merge mode to generate gain-controlled pixel data, thereby obtaining a response close to that of an ideal pixel.

[0198] As is evident from the above description, imaging devices based on some implementations of the disclosed technology are configured such that high-sensitivity pixels and low-sensitivity pixels are arranged together, thus enabling the acquisition of high dynamic range (HDR) images by performing image capture only once (i.e., one shooting action).

[0199] The implementation of the disclosed technology can provide various effects that can be directly or indirectly identified through the aforementioned patent documents.

[0200] Although several exemplary embodiments have been described, it should be understood that modifications and enhancements to the disclosed embodiments and other embodiments may be conceived based on what is described or shown in this patent document.

[0201] Cross-references to related applications

[0202] This patent document claims priority and benefit to Korean Patent Application No. 10-2021-0139219, filed on October 19, 2021, the disclosure of which in its entirety is incorporated herein by reference as part of the disclosure of this patent document.

Claims

1. An imaging device, the imaging device comprising: An image sensing device includes a pixel array of sensing pixels, the pixel array including at least one first pixel for sensing incident light and having a first dynamic range, and at least one second pixel for sensing incident light and having a second dynamic range, the first dynamic range being represented by a first measurable high light intensity that does not saturate the first pixel and a first measurable low light intensity relative to a pixel noise level of the first pixel, the second dynamic range being represented by a second measurable high light intensity that does not saturate the second pixel and a second measurable low light intensity relative to a pixel noise level of the second pixel, the second measurable high light intensity being higher than the first measurable high light intensity of the first pixel, and the second measurable low light intensity being higher than the first measurable low light intensity of the first pixel, wherein the pixel array of the sensing pixels is such that the ratio of the number of first pixels to all sensing pixels is higher than the ratio of the number of second pixels to all sensing pixels; and An image signal processor receives pixel data from the image sensing device and processes the pixel data to generate a high dynamic range (HDR) image corresponding to a dynamic range larger than either the first or the second dynamic range, based on the pixel data of the first pixel and the pixel data of the second pixel in the pixel array. Each of the at least one second pixel includes a red filter corresponding to red, a blue filter corresponding to blue, and a green filter corresponding to green. The second pixel including the green filter has a lower sensitivity than the second pixel including the red filter or the second pixel including the blue filter.

2. The imaging device according to claim 1, wherein, Each of the at least one second pixel includes: A photoelectric conversion element that generates photocharge corresponding to the intensity of incident light; A logarithmic transistor having a gate and a drain connected to a power supply voltage and a source directly connected to the photoelectric conversion element; and A driving transistor generates a voltage signal corresponding to the current flowing into the logarithmic transistor.

3. The imaging device according to claim 1, wherein, The at least one second pixel includes a light-blocking structure that blocks at least a portion of the incident light; and The at least one first pixel does not include the light-blocking structure.

4. The imaging device according to claim 1, wherein, Each of the at least one first pixel and the at least one second pixel includes a filter that selectively transmits light corresponding to a transmission wavelength band. The filter included in the at least one second pixel has a lower transmittance than the filter included in the at least one first pixel.

5. An imaging apparatus, the imaging apparatus comprising: An image sensing device includes a pixel array of sensing pixels, the pixel array including at least one first pixel for sensing incident light and having a first dynamic range, and at least one second pixel for sensing incident light and having a second dynamic range, the first dynamic range being represented by a first measurable high light intensity that does not saturate the first pixel and a first measurable low light intensity relative to a pixel noise level of the first pixel, the second dynamic range being represented by a second measurable high light intensity that does not saturate the second pixel and a second measurable low light intensity relative to a pixel noise level of the second pixel, the second measurable high light intensity being higher than the first measurable high light intensity of the first pixel, and the second measurable low light intensity being higher than the first measurable low light intensity of the first pixel, wherein the pixel array of the sensing pixels is such that the ratio of the number of first pixels to all sensing pixels is higher than the ratio of the number of second pixels to all sensing pixels; and An image signal processor receives pixel data from the image sensing device and processes the pixel data to generate a high dynamic range (HDR) image corresponding to a dynamic range larger than either the first or the second dynamic range, based on the pixel data of the first pixel and the pixel data of the second pixel in the pixel array. in, The image signal processor further determines, for a frame including pixel data of the at least one first pixel and pixel data of the at least one second pixel, high-brightness regions where the ratio of saturated first pixels per unit pixel group is higher than a predetermined ratio and low-brightness regions where the ratio of saturated first pixels per unit pixel group is equal to or less than a predetermined ratio; and The image signal processor also generates the HDR image based on the pixel data of the at least one second pixel in the high-brightness region, and generates the HDR image based on the pixel data of the at least one first pixel in the low-brightness region.

6. The imaging apparatus according to claim 5, wherein, The image signal processor further determines a first transition region in the low-brightness region where the ratio of the saturated first pixel per unit pixel group is equal to or higher than a minimum ratio, and this first transition region has a directionality of increasing brightness; and The image signal processor further generates the HDR image by combining the pixel data of the at least one first pixel and the pixel data of the at least one second pixel using a first combination weight in the first transition region.

7. The imaging apparatus according to claim 5, wherein, The image signal processor also defines the low-brightness region adjacent to the boundary between the low-brightness region and the high-brightness region as a second transition region; and The image signal processor further generates the HDR image by combining the pixel data of the at least one first pixel and the pixel data of the at least one second pixel using a second combination weight in the second transition region.

8. The imaging apparatus according to claim 7, wherein, The second combined weight decreases in the second transition region as it approaches the boundary between the low-brightness region and the high-brightness region.

9. The imaging apparatus according to claim 7, wherein, The second transition region has a predetermined width. The predetermined width is determined by the average value of the pixel data of the first pixel adjacent to the high-brightness region.

10. The imaging apparatus according to claim 7, wherein, The image signal processor further determines that the second transition region is located in the low-brightness region adjacent to the boundary between the low-brightness region and the high-brightness region, and the second transition region is a specific region where the average value of the pixel data of the first pixel adjacent to the high-brightness region is higher than a predetermined threshold.

11. An imaging apparatus, the imaging apparatus comprising: An image sensing device includes a pixel array of sensing pixels, the pixel array including at least one first pixel for sensing incident light and having a first dynamic range, and at least one second pixel for sensing incident light and having a second dynamic range, the first dynamic range being represented by a first measurable high light intensity that does not saturate the first pixel and a first measurable low light intensity relative to a pixel noise level of the first pixel, the second dynamic range being represented by a second measurable high light intensity that does not saturate the second pixel and a second measurable low light intensity relative to a pixel noise level of the second pixel, the second measurable high light intensity being higher than the first measurable high light intensity of the first pixel, and the second measurable low light intensity being higher than the first measurable low light intensity of the first pixel, wherein the pixel array of the sensing pixels is such that the ratio of the number of first pixels to all sensing pixels is higher than the ratio of the number of second pixels to all sensing pixels; and An image signal processor receives pixel data from the image sensing device and processes the pixel data to generate a high dynamic range (HDR) image corresponding to a dynamic range larger than either the first or the second dynamic range, based on the pixel data of the first pixel and the pixel data of the second pixel in the pixel array. in, The image signal processor also generates the HDR image by applying a gain corresponding to the sensitivity difference between the first pixel and the second pixel to the pixel data of the at least one second pixel.

12. An imaging apparatus, the imaging apparatus comprising: An image sensing device includes a pixel array of sensing pixels, the pixel array including at least one first pixel for sensing incident light and having a first dynamic range, and at least one second pixel for sensing incident light and having a second dynamic range, the first dynamic range being represented by a first measurable high light intensity that does not saturate the first pixel and a first measurable low light intensity relative to a pixel noise level of the first pixel, the second dynamic range being represented by a second measurable high light intensity that does not saturate the second pixel and a second measurable low light intensity relative to a pixel noise level of the second pixel, the second measurable high light intensity being higher than the first measurable high light intensity of the first pixel, and the second measurable low light intensity being higher than the first measurable low light intensity of the first pixel, wherein the pixel array of the sensing pixels is such that the ratio of the number of first pixels to all sensing pixels is higher than the ratio of the number of second pixels to all sensing pixels; and An image signal processor receives pixel data from the image sensing device and processes the pixel data to generate a high dynamic range (HDR) image corresponding to a dynamic range larger than either the first or the second dynamic range, based on the pixel data of the first pixel and the pixel data of the second pixel in the pixel array. in, The at least one first pixel and the at least one second pixel constitute a shared pixel that shares a single floating diffusion region; and In the shared pixel generation mode that combines the pixel signals corresponding to the photocharge generated in the at least one first pixel and the at least one second pixel, the image signal processor also applies a gain corresponding to the sensitivity difference between the first pixel and the second pixel to the value obtained when the pixel data corresponding to the illumination saturated by the at least one first pixel is subtracted from the pixel data of the shared pixel, thereby generating the HDR image.

13. An imaging apparatus, the imaging apparatus comprising: An image sensing device comprising at least one first pixel and at least one second pixel with a sensitivity lower than that of the at least one first pixel; as well as An image signal processor that generates a high dynamic range (HDR) image corresponding to a dynamic range greater than a first dynamic range of the at least one first pixel or a second dynamic range of the at least one second pixel, based on pixel data of the at least one first pixel and pixel data of the at least one second pixel. Wherein, the ratio of the at least one first pixel to all pixels included in the image sensing device is higher than the ratio of the at least one second pixel to all pixels included in the image sensing device. The first dynamic range is represented by a first measurable high light intensity that does not saturate the first pixel and a first measurable low light intensity relative to the pixel noise level of the first pixel. The second dynamic range is represented by a second measurable highlight intensity that does not saturate the second pixel and a second measurable low light intensity relative to the pixel noise level of the second pixel, wherein the second measurable highlight intensity is higher than the first measurable highlight intensity of the first pixel, and the second measurable low light intensity is higher than the first measurable low light intensity of the first pixel. Each of the at least one second pixel includes a red filter corresponding to red, a blue filter corresponding to blue, and a green filter corresponding to green. The second pixel including the green filter has a lower sensitivity than the second pixel including the red filter or the second pixel including the blue filter.

14. The imaging apparatus according to claim 13, wherein, The at least one first pixel includes a photoelectric conversion element to which light transmission increases in intensity in response to an increase in the intensity of incident light, the intensity increasing at a higher rate compared to the case of the at least one second pixel.

15. The imaging apparatus according to claim 13, wherein, The image signal processor also generates the HDR image by adjusting one or more weights of the pixel data given to the at least one first pixel and the pixel data of the at least one second pixel.

16. The imaging apparatus according to claim 15, wherein, The image signal processor also adjusts the one or more weights for one or more transition regions of various parts of a frame based on the brightness level.

17. The imaging apparatus according to claim 13, wherein, The at least one second pixel includes a light-blocking structure that blocks at least a portion of the incident light and is disposed above the substrate.

18. The imaging apparatus according to claim 13, wherein, Each of the at least one first pixel and the at least one second pixel includes a filter that selectively transmits light corresponding to a transmission wavelength band, and wherein the filter included in the at least one second pixel has a lower transmittance than the filter included in the at least one first pixel.

Images