Imaging device

Abstract

An imaging device may include: an image sensing device including a plurality of pixels, each pixel being configured to detect incident light from a scene to generate a pixel signal corresponding to the intensity of the incident light and to generate image data corresponding to the pixel signal, wherein the image sensing device operates in response to a control signal to perform an imaging operation; a brightness acquisition unit configured to acquire image data corresponding to a first pixel associated with a target area of ​​an image of a scene captured by the image sensing device among the plurality of pixels; a controllable item acquisition unit configured to acquire one or more sensitivity items indicating the sensitivity of each pixel to light as controllable items; and a set value calculation unit configured to generate a control signal for the image sensing device by calculating a set value for the controllable item based on the image data of the target area and the controllable item.

Description

Technical Field

[0001] The techniques and implementations disclosed in this patent document generally relate to an image sensing device for generating a high dynamic range (HDR) image. Background Art

[0002] Image sensing devices are devices that capture optical images by converting light into electrical signals using photosensitive semiconductor materials that react to light. With the development of the automotive, medical, computer, and communication industries, the demand for high-performance image sensing devices is increasing in various fields such as smartphones, digital cameras, game consoles, IoT (Internet of Things), robots, security cameras, and medical micro cameras.

[0003] Image sensing devices can be roughly divided into CCD (charge coupled device) image sensing devices and CMOS (complementary metal oxide semiconductor) image sensing devices. Compared with CMOS image sensing devices, CCD image sensing devices provide better image quality, but they tend to consume more power and are larger. Compared with CCD image sensing devices, CMOS image sensing devices are smaller in size and consume less power. In addition, CMOS sensors are manufactured using CMOS manufacturing technology, so the photosensitive element and other signal processing circuits can be integrated into a single chip, allowing miniaturized image sensing devices to be produced at a lower cost. For these reasons, CMOS image sensing devices are being developed for many applications including mobile devices. Summary of the invention

[0004] Various embodiments of the disclosed technology are directed to an imaging device that uses an optimal method to generate a high dynamic range (HDR) image.

[0005] According to an embodiment of the disclosed technology, an imaging device may include: an image sensing device, which includes a plurality of pixels, each pixel being configured to detect incident light from a scene to generate a pixel signal corresponding to the intensity of the incident light and to generate image data corresponding to the pixel signal, wherein the image sensing device operates in response to a control signal to perform an imaging operation; a brightness acquisition unit, which is configured to acquire image data corresponding to a first pixel associated with a target area of ​​an image of a scene captured by the image sensing device among the plurality of pixels; a controllable item acquisition unit, which is configured to acquire one or more sensitivity items indicating the sensitivity of each pixel to light as controllable items; and a set value calculation unit, which is configured to generate a control signal for the image sensing device by calculating a set value of the controllable item based on the image data of the target area and the controllable item.

[0006] According to another embodiment of the disclosed technology, an imaging device may include: an image sensing device, which includes a plurality of pixels, and the plurality of pixels are configured to detect incident light to generate pixel signals corresponding to the intensity of the incident light and generate image data corresponding to the pixel signals; a high dynamic range (HDR) controller, which is configured to generate a control signal by calculating a set value for at least one of an exposure amount, an exposure time, a conversion gain, and an analog gain as controllable items based on the image data; and an image signal processor, which is connected to receive the control signal from the high dynamic range (HDR) controller and is configured to synthesize image data with different sensitivities generated by adjusting the sensitivity of each of the plurality of pixels in response to the received control signal and generate a high dynamic range (HDR) image based on the image data.

[0007] It is to be understood that both the foregoing general description and the following detailed description of the disclosed technology are exemplary and explanatory and are intended to provide further explanation of the disclosure as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a block diagram illustrating an example of a photographing device based on some implementations of the disclosed technology.

[0009] Figure 2 is a graph showing an example of responses that vary according to the brightness of high-sensitivity pixels and low-sensitivity pixels achieved based on some implementations of the disclosed technology.

[0010] Figure 3 Some implementations based on the disclosed technology are shown in Figure 1 A block diagram of an example of an image sensing device and a high dynamic range (HDR) controller is shown.

[0011] Figure 4 Some implementations based on the disclosed technology are shown in Figure 3 A circuit diagram of an example of pixels included in the pixel array shown.

[0012] Figure 5 is a diagram showing an example of adjusting the sensitivity of a pixel based on a difference in light transmittance.

[0013] Figure 6 is a timing diagram illustrating an example method of adjusting the sensitivity of individual pixels based on differences in exposure time.

[0014] Figure 7 is a timing diagram illustrating an example method of adjusting the sensitivity of individual pixels based on differences in conversion gain.

[0015] Figure 8is a graph illustrating an example method of adjusting the sensitivity of individual pixels based on a difference in analog gain.

[0016] Fig. 9 is a flowchart illustrating an example method of forming an HDR image by a camera based on some implementations of the disclosed technology.

[0017] Fig.10 Some implementations based on the disclosed technology are shown in Fig. 9 A flow chart of an example operation S30 is shown. DETAILED DESCRIPTION

[0018] This patent document provides implementations and examples of shooting or imaging devices for generating high dynamic range (HDR) images, which can be used to substantially solve one or more technical or engineering problems and alleviate limitations or disadvantages encountered in some other imaging devices. Some implementations of the disclosed technology involve shooting or imaging devices that use the best method selected from various methods to generate high dynamic range (HDR) images. In the context of this patent document, the word "optimal" used in conjunction with the method of generating HDR images is used to indicate a method that provides better performance for the imaging device. In this sense, the word "optimal" may or may not convey the best possible performance that the imaging device can achieve. The disclosed technology can be implemented in some embodiments to determine the brightness and controllable items associated with the target object to be captured and control the controllable items by reflecting the hardware characteristics (e.g., response characteristics) into the process of controlling the sensitivity of the imaging device, thereby generating a high dynamic range (HDR) image with a maximum dynamic range while minimizing noise.

[0019] Reference will now be made in detail to embodiments of the disclosed technology, examples of which are illustrated in the accompanying drawings. Whenever possible, the same reference numerals will be used throughout the accompanying drawings to represent the same or similar parts. Although the present disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof are illustrated by way of example in the accompanying drawings. However, the present disclosure should not be construed as being limited to the embodiments set forth herein.

[0020] Hereinafter, various embodiments will be described with reference to the accompanying drawings. However, it should be understood that the disclosed technology is not limited to a specific embodiment, but includes various modifications, equivalents and / or substitutions of the embodiments. The embodiments of the disclosed technology can provide various effects that can be directly or indirectly recognized by the disclosed technology.

[0021] Figure 1 is a block diagram illustrating an example of a photographing or imaging device 1 based on some implementations of the disclosed technology. Figure 2is a graph showing an example of a response that varies according to the brightness of high-sensitivity pixels and low-sensitivity pixels based on some implementations of the disclosed technology. In this patent document, the word "sensitivity" may be used to indicate the sensitivity of an image sensor to light. Therefore, high-sensitivity pixels are more sensitive to light than low-sensitivity pixels.

[0022] Reference Figure 1 The imaging device 1 may include any mechanical or electronic device that can capture still or moving pictures, such as a digital still camera for capturing still images or a digital video camera for capturing moving pictures. Examples of the imaging device may include a digital single-lens reflex (DSLR) camera, a mirrorless camera, or a smart phone, etc. The imaging device 1 may include a lens and an image pickup element to capture (or capture) light of a target object and create an image of the target object.

[0023] The imaging device 1 may include a lens 10 , an iris 20 , a lens driver 30 , an iris driver 40 , an image sensing device 100 , and an image signal processor 200 .

[0024] The lens 10 may include an optical lens or lens assembly aligned with respect to an optical axis. The lens 10 may be disposed in front of the image sensing device 100 so that light can be focused to a point on the image sensing device 100. The position of the lens 10 may be adjusted by the lens driver 30. For example, the lens 10 may be moved along the optical axis by the lens driver 30.

[0025] The aperture 20 may be disposed in front of the image sensing device 100. The aperture driver 40 may control the amount of light reaching the image sensing device 100 by adjusting the degree to which the aperture 20 is opened or closed. Figure 1 The aperture 20 is shown by way of example only as being disposed between the lens 10 and the image sensing device 100 to receive light having passed through the lens 10. In other implementations, the aperture 20 may be disposed between lenses included in the lens 10, or may be disposed at the front end of the lens 10.

[0026] The light that has passed through the lens 10 and the aperture 20 reaches a light receiving surface of the image sensing device 100 , thereby forming an image of a target object to be captured on the image sensing device 100 .

[0027] The lens driver 30 may adjust the position of the lens 10 in response to a control signal (CS) received from the image signal processor 200. The lens driver 30 may perform various operations such as auto focusing, zooming in and out, and focusing by adjusting the position of the lens 10.

[0028] The aperture driver 40 may adjust the exposure level by controlling the opening / closing of the aperture 20 in response to a control signal (CS) received from the image signal processor 200. In this way, the aperture driver 40 may adjust the amount of light reaching the image sensing device 100.

[0029] The image sensing device 100 may be a complementary metal oxide semiconductor image sensor (CIS) for converting incident light into an electrical signal. The image sensing device 100 may adjust the exposure time, conversion gain, analog gain, etc. through the image signal processor 200. The image sensing device 100 may convert the incident light into an electrical signal based on pixels, thereby generating image data (IDATA). The image sensing device 100 may include a plurality of pixels having different sensitivities to light. In some implementations, sensitivity may refer to an increase in the value of the image data (IDATA) (or an increase in response) in response to an increase in the intensity of the incident light. That is, as the sensitivity increases, the amount of increase in the value of the image data (IDATA) corresponding to the intensity of the incident light also increases. As the sensitivity decreases, the amount of increase in the value of the image data (IDATA) corresponding to the intensity of the incident light also decreases. The sensitivity may be determined based on transmittance, conversion gain, exposure time, analog gain, etc. Reference will be made later. Figure 3 The detailed structure and operation of the image sensing device 100 are described.

[0030] The image signal processor 200 may process the image data (IDATA) received from the image sensing device 100, and may control the constituent elements of the imaging device 1 according to the processing result of the image data or the external input signal. The image signal processor 200 may reduce the noise in the image data (IDATA), and may perform various types of image signal processing (e.g., gamma correction, color filter array interpolation, color matrix, color correction, color enhancement, etc.) for image quality improvement in the image data (IDATA). In addition, the image signal processor 200 may compress the image data (IDATA) created by the image signal processing and create an image file using the compressed image data. Alternatively, the image signal processor 200 may restore the image data (IDATA) from the image file. In this case, the format for compressing such image data (IDATA) may be a reversible format or an irreversible format. Examples of these compression formats include the Joint Photographic Experts Group (JPEG) format and the JPEG2000 format for still images. A moving picture may be compressed by compressing a plurality of frames according to the Moving Picture Experts Group (MPEG) standard. For example, an image file may be created according to the Exchangeable Image File Format (Exif) standard.

[0031] In addition, the image signal processor 200 may generate an HDR image by synthesizing at least two images generated using image sensing pixels with different sensitivities. The configuration for generating an HDR image may be defined as an image synthesis unit different from the HDR controller 300. For example, the image sensing device 100 may output a low sensitivity image generated by low sensitivity pixels having relatively low sensitivity to light and a high sensitivity image generated by high sensitivity pixels having relatively high sensitivity to light. The image signal processor 200 may combine the low sensitivity image and the high sensitivity image to generate an HDR image. Although the examples discussed above use low sensitivity images and high sensitivity images to generate HDR images, the disclosed technology may be implemented in some embodiments using N images with N different sensitivities, where N is an integer greater than or equal to 2. In one example, the image sensing processor 200 may generate an HDR image using image data (IDATA) with N different sensitivities.

[0032] The HDR image generated by the image signal processor 200 may be stored in an internal memory or an external memory of the imaging device 1 in response to a user request or in an autonomous manner to display the stored HDR image on a display device.

[0033] In addition, the image signal processor 200 may perform unsharp removal processing, blur removal processing, edge emphasis processing, image analysis processing, image recognition processing, image effect processing, and the like.

[0034] In addition, the image signal processor 200 can perform display image signal processing for display. For example, the image signal processor 200 can perform brightness level adjustment, color correction, contrast adjustment, contour emphasis adjustment, screen segmentation processing, character image generation and image synthesis processing.

[0035] The image signal processor 200 may control the lens driver 30 , the aperture driver 40 , and the image sensing device 100 according to (1) control information automatically generated by image data (IDATA) input in real time, or (2) control information manually input by a user.

[0036] In some specific implementations, the image signal processor 200 may include the HDR controller 300. In other implementations, the HDR controller 300 may also be implemented independently of the image signal processor 200. For example, the HDR controller 300 may be included in the image sensing device 100.

[0037] The HDR controller 300 may control at least one of the aperture driver 40 and the image sensing device 100 so that the pixels of the image sensing device 100 may have an optimal dynamic range.

[0038] Figure 2 The response as a function of the intensity of light incident on the high-sensitivity pixel and the low-sensitivity pixel is shown. As the intensity of the incident light increases, the response of the high-sensitivity pixel shows a relatively large increase, and as the intensity of the incident light increases, the response of the low-sensitivity pixel shows a relatively small increase. The response of the high-sensitivity pixel and the response of the low-sensitivity pixel vary according to the brightness or intensity of the incident light applied to the corresponding pixel. Here, the response may refer to the image data (IDATA) of the corresponding pixel or the value of the image data (IDATA).

[0039] The response may have a signal-to-noise ratio (SNR) limit (denoted by SNRLimit) and a saturation level (denoted by Saturation).

[0040] exist Figure 2 In the embodiment of the present invention, the signal-to-noise ratio (SNR) threshold level refers to a threshold value that can satisfy a predetermined reference SNR. A response less than the SNR threshold level can be regarded as an invalid response that does not satisfy the reference SNR, and a response greater than the SNR threshold level can be regarded as a valid response that satisfies the reference SNR. The reference SNR can be determined experimentally in consideration of the characteristics of the image sensing device 100.

[0041] The saturation level refers to the maximum level indicating the intensity of the incident light. The saturation level may be determined based on: the ability of the pixel to convert the intensity of the incident light into a photocharge (e.g., the capacitance of the photoelectric conversion element); the ability to convert the photocharge into an analog signal (e.g., the capacitance of the floating diffusion region (FD)); and the ability to convert the analog signal into a digital signal (e.g., the input range of the ADC). As the intensity of the incident light increases, the response may increase in proportion to the intensity of the incident light until the response reaches the saturation level. After the response reaches the saturation level, the response does not increase despite the increase in the intensity of the incident light. For example, after the response reaches the saturation level, the response remains at the same value as the saturation value and does not increase above the saturation level.

[0042] The effective response of each pixel may refer to a response that may indicate the intensity of the incident light while satisfying the reference SNR. The range of the intensity of the incident light corresponding to the effective response of the pixel may be referred to as the dynamic range of the pixel. That is, the dynamic range of the pixel may refer to the intensity range of the incident light for which each pixel has an effective response.

[0043] The response of a high-sensitivity pixel in response to an increase in the intensity of incident light is relatively large. Figure 2 The response curve graph of the high-sensitivity pixel in has a relatively large slope until the response reaches the saturation level, and after the response reaches the saturation level, has a fixed level corresponding to the saturation level regardless of an increase in the intensity of incident light.

[0044] The response of a low-sensitivity pixel to an increase in the intensity of incident light is relatively small. Figure 2 The response curve graph of the low-sensitivity pixel in has a relatively small slope until the response reaches the saturation level, and after the response reaches the saturation level, has a fixed level corresponding to the saturation level regardless of an increase in the intensity of incident light.

[0045] like Figure 2 As shown, the minimum value (DR_H) of the high-sensitivity pixel dynamic range may be smaller than the minimum value (DR_L) of the low-sensitivity pixel dynamic range, and the maximum value (DR_H) of the high-sensitivity pixel dynamic range may be smaller than the maximum value (DR_L) of the low-sensitivity pixel dynamic range. Therefore, in a low-brightness range where the intensity of the incident light is relatively small, the high-sensitivity pixel may be more suitable for detecting the intensity of the incident light. In a high-brightness range where the intensity of the incident light is relatively large, the low-sensitivity pixel may be more suitable for detecting the intensity of the incident light.

[0046] A high dynamic range (HDR) can be achieved using both the response of a high-sensitivity pixel suitable for a low brightness range and the response of a low-sensitivity pixel suitable for a high brightness range. In other words, using both high-sensitivity pixels and low-sensitivity pixels can allow the entire pixel array to have a high dynamic range (HDR) corresponding to a specific range from a minimum value of the high-sensitivity pixel dynamic range to a maximum value of the low-sensitivity pixel dynamic range, compared to using only one of the high-sensitivity pixels and the low-sensitivity pixels. To this end, at least a portion of the high-sensitivity pixel dynamic range and at least a portion of the low-sensitivity pixel dynamic range can overlap each other.

[0047] In some implementations, a high dynamic range (HDR) image corresponding to a high dynamic range (HDR) may be synthesized using high sensitivity pixels (HPX) and low sensitivity pixels (LPX) by using: a method of synthesizing an HDR image by calculating (e.g., summing) an HPX response and an LPX response; and / or a method of forming an image based on the HPX response at a low brightness level and forming an image based on the LPX response at a high brightness level.

[0048] like Figure 2 As shown, when the sensitivity of each pixel (i.e., the slope of the response) is adjusted, the dynamic range of the corresponding pixel can be adjusted. The sensitivity of the pixel can be determined based on one or more sensitivity items. Here, the sensitivity items include at least one of the following: exposure; transmittance; exposure time; conversion gain; and analog gain, which indicate pixel sensitivity. Therefore, the HDR controller 300 can adjust the dynamic range of each pixel by adjusting the controllable items among the sensitivity items discussed above. The controllable items may include sensitivity items that can be controlled by the HDR controller 300.

[0049] As the dynamic range becomes wider, there is an advantage in that an effective response corresponding to incident light having a wide brightness range can be obtained. However, the widening of the dynamic range may lead to excessive changes in the exposure time and the time point at which each pixel captures the incident light, causing each pixel to capture the incident light at a different time point. As a result, motion artifacts of fast-moving objects (e.g., target objects moving at high speed) may increase. If the dynamic range is unnecessarily extended to a low brightness range while capturing a very bright scene, the response of high-sensitivity pixels may be saturated, thereby deteriorating the overall image quality.

[0050] Therefore, the HDR controller 300 may control at least one of the aperture driver 40 and the image sensing device 100 based on the characteristics of the scene and the characteristics of the controllable item so that the pixels of the image sensing device 100 may have an optimal dynamic range. In the context of this patent document, the word "optimal" used in conjunction with the dynamic range is used to indicate a dynamic range that provides better performance for the imaging device.

[0051] Figure 3 Some implementations based on the disclosed technology are shown in Figure 1 A block diagram of an example of an image sensing device and a high dynamic range (HDR) controller 300 is shown.

[0052] Reference Figure 3 , the image sensing device 100 may include a pixel array 110, a row driver 120, a correlated double sampler (CDS) 130, an analog-to-digital converter (ADC) 140, an output buffer 150, a column driver 160, and a timing controller 170. This discussion is by way of example only. Figure 1 Although the components of image sensing device 100 are shown, this patent document encompasses numerous other alterations, substitutions, variations, modifications, and modifications.

[0053] The pixel array 110 may include a plurality of unit imaging pixels arranged in rows and columns. In one example, the plurality of unit imaging pixels may be arranged in a two-dimensional pixel array including rows and columns. In another example, the plurality of unit imaging pixels may be arranged in a three-dimensional pixel array. The plurality of unit pixels may convert optical signals into electrical signals based on unit pixels or pixel groups, wherein the unit pixels in the pixel group share at least a specific internal circuit. The pixel array 110 may receive a drive signal including a row selection signal, a pixel reset signal, and a transfer signal from the row driver 120. Upon receiving the drive signal, the corresponding imaging pixel in the pixel array 110 may be enabled to perform operations corresponding to the row selection signal, the pixel reset signal, and the transfer signal.

[0054] The row driver 120 may enable the pixel array 110 to perform a specific operation on the imaging pixels in the corresponding row based on the command and control signals provided by the controller circuit such as the timing controller 170. In some implementations, the row driver 120 may select one or more imaging pixels arranged in one or more rows of the pixel array 110. The row driver 120 may generate a row selection signal to select one or more rows among the multiple rows. The row driver 120 may sequentially enable a pixel reset signal for resetting the imaging pixels corresponding to at least one selected row and a transmission signal for the pixels corresponding to the at least one selected row. Therefore, as analog signals generated by the respective imaging pixels of the selected row, the reference signal and the image signal may be sequentially transmitted to the CDS 130. The reference signal may be an electrical signal provided to the CDS 130 when the sensing node (e.g., the floating diffusion node) of the imaging pixel is reset, and the image signal may be an electrical signal provided to the CDS 130 when the photocharge generated by the imaging pixel is accumulated in the sensing node. The reference signal indicating the unique reset noise of each pixel and the image signal indicating the intensity of the incident light may be collectively referred to as a pixel signal as needed.

[0055] CMOS image sensors may use correlated double sampling (CDS) to remove undesired offset values ​​(called fixed pattern noise) of pixels by sampling the pixel signal twice to remove the difference between the two samples. In one example, correlated double sampling (CDS) may remove undesired offset values ​​of pixels by comparing the pixel output voltages obtained before and after the photocharges generated by incident light are accumulated in the sensing node so that only the pixel output voltage based on the incident light may be measured. In some embodiments of the disclosed technology, CDS 130 may sequentially sample and maintain the voltage levels of the reference signal and the image signal provided to each of the plurality of column lines from the pixel array 110. That is, CDS 130 may sample and maintain the voltage levels of the reference signal and the image signal corresponding to each column of the pixel array 110.

[0056] In some implementations, the CDS 130 may transmit the reference signal and the image signal of each column as a correlated double sampling signal to the ADC 140 based on a control signal from the timing controller 170 .

[0057] ADC 140 is used to convert the analog CDS signal into a digital signal. In some implementations, ADC 140 may convert the correlated double sampling signal generated by CDS 130 for each column into a digital signal and output the digital signal.

[0058] The ADC 140 may include a plurality of column counters. Each column of the pixel array 110 is connected to the column counter, and image data may be generated by converting a correlated double sampling signal received from each column into a digital signal using the column counter. In another embodiment of the disclosed technology, the ADC 140 may include a global counter to convert a correlated double sampling signal corresponding to a column into a digital signal using a global code provided from the global counter.

[0059] The output buffer 150 may temporarily hold the column-based image data provided from the ADC 140 to output the image data. In one example, the image data provided from the ADC 140 to the output buffer 150 may be temporarily stored in the output buffer 150 based on a control signal of the timing controller 170. The output buffer 150 may provide an interface to compensate for a data rate difference or a transmission rate difference between the image sensing device 100 and other devices.

[0060] The column driver 160 may select a column of the output buffer upon receiving a control signal from the timing controller 170, and sequentially output the image data temporarily stored in the selected column of the output buffer 150. In some implementations, upon receiving an address signal from the timing controller 170, the column driver 160 may generate a column selection signal based on the address signal and select a column of the output buffer 150, thereby outputting the image data from the selected column of the output buffer 150 as an output signal.

[0061] The timing controller 170 may control operations of the row driver 120 , the ADC 140 , the output buffer 150 , and the column driver 160 .

[0062] The timing controller 170 may provide clock signals required for the operation of the respective components of the image sensing device 100, control signals for timing control, and address signals for selecting rows or columns to the row driver 120, the CDS 130, the ADC 140, the output buffer 150, and the column driver 160. In an embodiment of the disclosed technology, the timing controller 170 may include a logic control circuit, a phase-locked loop (PLL) circuit, a timing control circuit, a communication interface circuit, and the like.

[0063] The timing controller 170 may control the sensitivity of each pixel included in the pixel array 110. The sensitivity of each pixel may be determined by transmittance, exposure time, conversion gain, and analog gain. Here, transmittance may refer to the ratio of the light intensity reaching the device (i.e., the photoelectric conversion element described later) in the pixel that converts light into photocharges relative to the light intensity incident on the pixel. Exposure time may refer to the time required to convert the light incident on the pixel into photocharges. Conversion gain may refer to the ratio of the pixel signal (i.e., voltage) obtained by converting the photocharges to the amount of photocharges generated by the pixel. Analog gain may refer to the ratio of the digital value (i.e., image data) obtained by converting the pixel signal to the level of the pixel signal output from the pixel. The higher the transmittance, the higher the pixel sensitivity. The shorter the exposure time, the higher the pixel sensitivity. The higher the conversion gain, the higher the pixel sensitivity. In addition, the higher the analog gain, the higher the pixel sensitivity. On the contrary, the lower the transmittance, the lower the pixel sensitivity. The longer the exposure time, the lower the pixel sensitivity. The lower the conversion gain, the lower the pixel sensitivity. In addition, the lower the analog gain, the lower the pixel sensitivity.

[0064] The transmittance may have a fixed value predetermined for each pixel. The exposure time, conversion gain, and analog gain may be controllable items. In order to control the exposure time or conversion gain of each pixel, the timing controller 170 may control the row driver for supplying a control signal to the pixel array 110. In order to control the analog gain of each pixel, the timing controller 170 may control the ADC 40 configured to perform analog-to-digital conversion (ADC).

[0065] In some embodiments, the disclosed technology can be implemented to control the exposure time, conversion gain and analog gain and control the sensitivity of each pixel through the timing controller 170, as will be referred to below. Figure 6 discussed.

[0066] The HDR controller 300 may include a brightness acquisition unit 310 , a controllable item acquisition unit 320 , and a setting value calculation unit 330 .

[0067] The brightness acquisition unit 310 may acquire brightness information associated with a target area in the captured scene image based on the image data (IDATA) generated by the image sensing device 100. In this case, the captured scene may be represented by the image data (IDATA) corresponding to the frame captured by the image sensing device 100. That is, the scene may be an image captured by the entire pixel array 110 during a predetermined period corresponding to the frame, and pixels belonging to a target area corresponding to at least a portion of the scene image may be defined as target pixels. The target area may refer to an area designated by a user (e.g., a specific subject) as an area included in the scene image, may refer to an area in which image data (IDATA) with high brightness is concentrated (i.e., the brightest area) in the scene image, or may refer to an area corresponding to the entire scene image. In addition, brightness may refer to the value of the image data (IDATA). In some other implementations, the brightness acquisition unit 310 may also use a separate device (e.g., a photometric sensor, other camera devices, etc.) to acquire the brightness of the target area.

[0068] The controllable item acquisition unit 320 may acquire controllable items among exposure, exposure time, conversion gain, and analog gain. Here, the exposure may be a value determined according to the degree of opening of the aperture 20 under the control of the aperture driver 40. Each of the exposure, exposure time, conversion gain, and analog gain may be a predetermined fixed value or a controllable value. For example, in night shooting mode, the system (e.g., an application processor) may fix or set the exposure to a maximum exposure. In another example, in a high-speed shooting mode or a video mode where the time allocated to each frame is very short, the system may limit the exposure time to less than or equal to a predetermined time, or may fix the exposure time to a constant time. In order to reduce the power consumption required for the control and image processing of the image sensing device 100, the system may limit the control of at least one of the conversion gain and the analog gain.

[0069] The setting value calculation unit 330 may calculate the setting value of each controllable item based on the target area brightness received from the brightness acquisition unit 310 and the controllable items received from the controllable item acquisition unit 320, and may generate a control signal (CS) indicating the calculated setting value. Here, the setting value may refer to information for controlling the controllable items (e.g., exposure amount, exposure time, conversion gain, and analog gain).

[0070] Will refer to later Fig.10 The detailed operation of the setting value calculation unit 330 is described.

[0071] Figure 4 Some implementations based on the disclosed technology are shown in Figure 3 2 is a circuit diagram of an example of pixels included in the pixel array 110 shown.

[0072] Reference Figure 4 , the pixel (PX) may be any one of a plurality of pixels included in the pixel array 110. Although for convenience of description, Figure 4 Only one pixel (PX) is shown, but it should be noted that other pixels may also have a structure similar to or the same as that of the pixel (PX) and perform operations similar to or the same as that of the pixel (PX).

[0073] The pixel (PX) may include a photoelectric conversion element (PD), a transfer transistor (TX), a reset transistor (RX), a floating diffusion region (FD), a conversion gain (CG) transistor (CX), a first capacitor C1 and a second capacitor C2, a source follower transistor (SF), and a selection transistor (SX). Although by way of example Figure 4 The pixel (PX) is shown to include only one photoelectric conversion element (PD), but it should be noted that the pixel (PX) may also be a shared pixel including a plurality of photoelectric conversion elements. In this case, a plurality of transfer transistors may be provided to correspond to the photoelectric conversion elements, respectively.

[0074] Each photoelectric conversion element (PD) may generate and accumulate photocharges corresponding to the intensity of incident light. For example, each photoelectric conversion element (PD) may be implemented as a photodiode, a phototransistor, a photogate, a pinned photodiode, or a combination thereof.

[0075] If the photoelectric conversion element (PD) is implemented as a photodiode, the photoelectric conversion element (PD) may be a region in which a second conductive impurity (eg, N-type impurity) is doped in a substrate including a first conductive impurity (eg, P-type impurity).

[0076] The transfer transistor (TX) may be connected between the photoelectric conversion element (PD) and the floating diffusion region (FD). The transfer transistor (TX) may be turned on or off in response to a transfer control signal (TG). If the transfer transistor (TX) is turned on, the photocharge accumulated in the corresponding photoelectric conversion element (PD) may be transferred to the floating diffusion region (FD).

[0077] The reset transistor (RX) may be disposed between the floating diffusion region (FD) and the power supply voltage (VDD), and the voltage of the floating diffusion region (FD) may be reset to the power supply voltage (VDD) in response to a reset control signal (RG).

[0078] The floating diffusion region (FD) may accumulate photocharges received from the transfer transistor (TX). The floating diffusion region (FD) may be coupled to a first capacitor (C1) connected to a ground terminal. For example, the floating diffusion region (FD) may be a region doped with a second conductive impurity (e.g., an N-type impurity) in a substrate (e.g., a P-type substrate) including a first conductive impurity. In this case, the substrate and the impurity-doped region may be modeled as a first capacitor (C1) acting as a junction capacitor.

[0079] The CG transistor (CX) may be coupled between the floating diffusion region (FD) and the second capacitor (C2), and may selectively connect the second capacitor (C2) to the floating diffusion region (FD) in response to a CG control signal (CG). The second capacitor (C2) may include at least one of a metal insulator metal (MIM) capacitor, a metal insulator polysilicon (MIP) capacitor, a metal oxide semiconductor (MOS) capacitor, and a junction capacitor. When the CG transistor (CX) is turned off, the floating diffusion region (FD) may have an electrostatic capacitance corresponding to the capacitance of the first capacitor (C1). When the CG transistor (CX) is turned on, the floating diffusion region (FD) may have an electrostatic capacitance corresponding to the sum of the capacitance of the first capacitor (C1) and the capacitance of the second capacitor (C2). That is, the CG transistor (CX) may control the capacitance of the floating diffusion region (FD).

[0080] Although for the sake of convenience, Figure 4 Only one CG transistor (CX) is shown, but it should be noted that multiple CG transistors may also be used. In this case, the capacitance of the floating diffusion region (FD) may vary.

[0081] The source follower transistor (SF) can be connected between the power supply voltage (VDD) and the selection transistor (SX), can amplify the change in the potential or voltage of the floating diffusion region (FD) that receives the photocharge accumulated in the photoelectric conversion element (PD), and can transmit the amplified result to the selection transistor (SX).

[0082] The selection transistor (SX) may be connected between the source follower transistor (SF) and the output signal line and may be turned on by a selection control signal (SEL) so that the selection transistor (SX) may output the electrical signal received from the source follower transistor (SF) as a pixel signal (PS).

[0083] Figure 5 is a conceptual diagram illustrating an example of a method of adjusting the sensitivity of each pixel based on a difference in light transmittance.

[0084] Reference Figure 5, the high transmittance pixel (HPX) has a relatively high transmittance, and the low transmittance pixel (LPX) has a relatively low transmittance. In other words, three high transmittance pixels (HPX) and only one low transmittance pixel (LPX) may be arranged in a (2×2) matrix (e.g., a unit matrix). Each of the high transmittance pixel (HPX) and the low transmittance pixel (LPX) may have the same Figure 4 In some implementations, each of the high-transmittance pixel (HPX) and the low-transmittance pixel (LPX) may include a photoelectric conversion element and a transfer transistor, and the remaining constituent elements other than the photoelectric conversion element and the transfer transistor may be implemented as a shared pixel structure shared by four pixels.

[0085] In some implementations, the high light transmittance pixels (HPX) and the low light transmittance pixels (LPX) may be pixels that sense light of the same color (e.g., red, blue, or green). In this case, the pixel array may form a four-Bayer pattern structure based on a (2×2) matrix.

[0086] Reference Figure 5 , a high-transmittance pixel (HPX) and a low-transmittance pixel (LPX) taken along line AA′ may include a substrate 510, at least one photoelectric conversion element 520, at least one filter 530, at least one microlens 540, and at least one light blocking structure 550.

[0087] For example, substrate 510 may be a P-type or N-type bulk substrate, may be a substrate formed by growing a P-type or N-type epitaxial layer on a P-type bulk substrate, or may be a substrate formed by growing a P-type or N-type epitaxial layer on an N-type bulk substrate.

[0088] The photoelectric conversion element 520 may be formed in the substrate 510 and may correspond to Figure 4 That is, the photoelectric conversion element 520 may generate and accumulate photocharges corresponding to the intensity of incident light that has passed through the microlens 540 and the optical filter 530 .

[0089] The optical filter 530 may selectively transmit light having a wavelength band to be transmitted (e.g., red light, green light, blue light, magenta light, yellow light, cyan light, infrared (IR) light). In this case, the wavelength band may refer to a wavelength band of light to be selectively transmitted by the corresponding optical filter. For example, each optical filter 530 may include a colored photosensitive material corresponding to a specific color, or may include thin film layers arranged alternately. The optical filters included in the pixel array 110 may be arranged to correspond to pixels arranged in a matrix array including a plurality of rows and a plurality of columns, thereby forming an optical filter array.

[0090] Each microlens 540 may be formed above each filter 530 , and may increase light-condensing capability of incident light, thereby resulting in an increase in light-receiving (Rx) efficiency of the photoelectric conversion element 520 .

[0091] The light blocking structure 550 may be disposed between one surface of the substrate 510 and the optical filter 530 so that at least a portion of incident light having passed through the optical filter 530 in the low light transmittance pixel (LPX) is blocked by the light blocking structure and is not transmitted to the photoelectric conversion element 520. The light blocking structure 550 may include at least one of a material having high light reflectivity (e.g., silver or aluminum) and a material having high light absorptivity (e.g., tungsten).

[0092] The total area of ​​the low light transmittance pixel (LPX) may be defined as the sum of the blocking area of ​​the region where the light blocking structure 550 is not disposed and the open area of ​​the region where the light blocking structure 550 is disposed. The transmittance of the low light transmittance pixel (LPX) may be determined according to the ratio of the blocking area to the open area.

[0093] The high light transmittance pixel (HPX) not including the light blocking structure 550 may have a higher light transmittance than the low light transmittance pixel (LPX) including the light blocking structure 550 .

[0094] That is, when incident light having the same intensity is incident on a high-transmittance pixel (HPX) and a low-transmittance pixel (LPX), the intensity of light transmitted to the photoelectric conversion element 520 of the low-transmittance pixel (LPX) may be less than the intensity of light transmitted to the photoelectric conversion element 520 of the high-transmittance pixel (HPX).

[0095] In addition, the intensity of light transmitted to the photoelectric conversion element 520 of the low-transmittance pixel (LPX) may increase with a relatively low slope in response to an increase in the intensity of the incident light. The intensity of light transmitted to the photoelectric conversion element 520 of the high-transmittance pixel (HPX) may increase with a relatively high slope in response to an increase in the intensity of the incident light.

[0096] Since each of the intensity of light transmitted to the photoelectric conversion element 520 of the low-light-transmittance pixel (LPX) and the intensity of light transmitted to the photoelectric conversion element 520 of the high-light-transmittance pixel (HPX) is converted into a pixel signal, the response of the low-light-transmittance pixel (LPX) can be Figure 2 The response of the low-sensitivity pixel shown is similar to that of the high-transmittance pixel (HPX). Figure 2 The response of the high sensitivity pixel shown is similar.

[0097] although Figure 5The light blocking structure 550 shown is disposed at the edge of the low light transmittance pixel (LPX), but other implementations are also possible. For example, the light blocking structure 550 may be disposed at any position of the low light transmittance pixel (LPX), and in the case where the light blocking structure 550 is partially blocked, the light blocking structure 550 may also be disposed in the entire area of ​​the low light transmittance pixel (LPX).

[0098] The image sensing device 100 based on some implementations of the disclosed technology may generate an HDR image using only one image by implementing low-sensitivity pixels and high-sensitivity pixels in only one pixel array 110 .

[0099] Figure 6 is a timing chart showing an example of a method of adjusting the sensitivity of each pixel based on the difference in exposure time.

[0100] Reference Figure 6 , for the convenience of description, it is assumed that at least one long exposure pixel having a relatively long exposure time and at least one short exposure pixel having a relatively short exposure time are arranged together in a row of the pixel array 110. The pixel array 110 may be enabled on a row basis by the row driver 120 so that the long exposure pixel and the short exposure pixel may receive the same reset signal (RG) and the same row selection signal (SEL). However, in order to allow the long exposure pixel and the short exposure pixel to have different exposure times, the long exposure pixel and the short exposure pixel may receive different transmission signals such as a first transmission signal (TG_L) and a second transmission signal (TG_S). Although not shown in the figure, the long exposure pixel and the short exposure pixel may have the same CG signal (CG) or different CG signals (CG).

[0101] Each of the reset signal (RG), the row selection signal (SEL), the first transmission signal (TG_L), and the second transmission signal (TG_S) may have a logic low level (L) and a logic high level (H). A transistor configured to receive a signal having a logic low level (L) may be turned off, and a transistor configured to receive a signal having a logic high level (H) may be turned on.

[0102] The operation period (i.e., the time period for operation) of each of the long exposure pixel and the short exposure pixel may include a reset period (RS), an exposure period, and a readout period (RD). In other implementations, the operation period may also include a readout period (RD) for generating a reference signal after the reset period (RS).

[0103] The reset period (RS) may be a period of time during which the photocharges remaining unused in the corresponding pixel are removed and the floating diffusion region (FD) is reset to the power supply voltage (VDD). In the reset period (RS), each of the reset signal (RG), the row selection signal (SEL), the first transfer signal (TG_L), and the second transfer signal (TG_S) may have a logic high level (H).

[0104] The exposure period EX1 may be a period of time during which the photoelectric conversion element (PD) of the long exposure pixel generates and accumulates photocharges corresponding to the intensity of the incident light, and then the accumulated photocharges are transferred to the floating diffusion region (FD). The exposure period EX2 may be a period of time during which the photoelectric conversion element (PD) of the short exposure pixel generates and accumulates photocharges corresponding to the intensity of the incident light, and then the accumulated photocharges are transferred to the floating diffusion region (FD).

[0105] After the reset period (RS), the photoelectric conversion element (PD) of the long exposure pixel may generate and accumulate photocharges corresponding to the intensity of the incident light. Before the first transfer signal (TG_L) changes from a logic high level (H) to a logic low level (L), the transfer transistor of the long exposure pixel may transfer the photocharges from the photoelectric conversion element (PD) of the long exposure pixel to the floating diffusion region (FD) of the long exposure pixel. In other words, the long exposure pixel may accumulate the photocharges generated during the first exposure period (EX1) in the floating diffusion region (FD).

[0106] After the reset period (RS), the photoelectric conversion element (PD) of the short exposure pixel may generate and accumulate photocharges corresponding to the intensity of the incident light. Before the second transfer signal (TG_S) changes from a logic high level (H) to a logic low level (L), the transfer transistor of the short exposure pixel may transfer the photocharges from the photoelectric conversion element (PD) of the short exposure pixel to the floating diffusion region (FD) of the short exposure pixel. In other words, the short exposure pixel may accumulate the photocharges generated during the second exposure period (EX2) in the floating diffusion region (FD). Each of the exposure periods EX1 and EX2 may refer to a time period in which photocharges are generated and accumulated in each pixel, and the transfer signal applied to each pixel may determine the length of the exposure period.

[0107] The readout period (RD) may refer to a period in which each of the long exposure pixel and the short exposure pixel generates an electric signal corresponding to photocharges accumulated in the floating diffusion area (FD) and then outputs the electric signal as a pixel signal (PS).

[0108] The time point at which the first transmission signal (TG_L) changes from a logic high level (H) to a logic low level (L) may be later than the time point at which the second transmission signal (TG_S) changes from a logic high level (H) to a logic low level (L). Therefore, the first exposure period (EX1) may be longer than the second exposure period (EX2).

[0109] In other words, when incident light having the same intensity is incident on the long exposure pixel and the short exposure pixel, the amount of photocharge accumulated in the floating diffusion region (FD) of the short exposure pixel may be smaller than that of the long exposure pixel.

[0110] In addition, the amount of photocharge accumulated in the floating diffusion region (FD) of the short-exposure pixel may increase with a relatively low slope in response to an increase in the intensity of the incident light. The amount of photocharge accumulated in the floating diffusion region (FD) of the long-exposure pixel may increase with a relatively high slope in response to an increase in the intensity of the incident light.

[0111] Since each of the amount of photocharge accumulated in the floating diffusion region (FD) of the short exposure pixel and the amount of photocharge accumulated in the floating diffusion region (FD) of the long exposure pixel is converted into a pixel signal, the response of the short exposure pixel can follow Figure 2 The response of the low-sensitivity pixel is shown, and the response of the long-exposure pixel can be followed Figure 2 The response of the high-sensitivity pixel is shown.

[0112] The image sensing device 100 according to another embodiment of the disclosed technology may generate an HDR image using only one image by implementing low-sensitivity pixels and high-sensitivity pixels in only one pixel array 110 .

[0113] Figure 7 is a timing chart showing an example of a method of adjusting the sensitivity of each pixel based on the difference in conversion gain.

[0114] Reference Figure 7 For the convenience of description and better understanding of the disclosed technology, it is assumed that at least one high conversion gain (CG) pixel with a relatively high conversion gain and at least one low CG pixel with a relatively low conversion gain are arranged together in a row of the pixel array 110. The pixel array 110 can be enabled on a row basis by the row driver 120 so that the high CG pixel and the low CG pixel can receive the same reset signal (RG) and the same row selection signal (SEL). For the convenience of description, it is assumed that Figure 7 The high CG pixels and low CG pixels shown receive the same transmission signal (TG), but the high CG pixels and low CG pixels may also be Figure 6In order to allow the high CG pixel and the low CG pixel to have different conversion gains, the high CG pixel may receive a first CG signal (CG_H) and the low CG pixel may receive a second CG signal (CG_L).

[0115] The operation period (i.e., time period for operation) of each of the high CG pixel and the low CG pixel may include a reset period (RS), an exposure period, and a readout period (RD). The operation in each of the reset period (RS), the exposure period, and the readout period (RD) is similar to that in the low CG pixel. Figure 6 Those of are basically the same, so for the convenience of description, their redundant descriptions will be omitted in this article.

[0116] During the reset period (RS), the exposure period, and the readout period (RD), the high CG pixel may receive the first CG signal (CG_H) maintained at a logic low level (L), and the low CG pixel may receive the second CG signal (CG_L) maintained at a logic high level (H). Therefore, the capacitance of the floating diffusion region (FD) of the high CG pixel may correspond to the capacitance of the first capacitor (C1), and the capacitance of the floating diffusion region (FD) of the low CG pixel may correspond to the sum of the capacitance of the first capacitor (C1) and the capacitance of the second capacitor (C2).

[0117] In the readout period (RD), the photocharge accumulated in the floating diffusion region (FD) may generate a voltage change of the floating diffusion region (FD), and this voltage change of the floating diffusion region (FD) may be converted into an electrical signal by the source follower transistor (SF). In this case, the degree of voltage change in the floating diffusion region (FD) may be determined by the capacitance of the floating diffusion region (FD). In association with the same amount of photocharge, as the capacitance of the floating diffusion region (FD) decreases, the voltage change of the floating diffusion region (FD) may increase, and as the capacitance of the floating diffusion region (FD) increases, the voltage change of the floating diffusion region (FD) may decrease.

[0118] That is, when the same amount of photocharge is accumulated in the floating diffusion region (FD) of the high CG pixel and the floating diffusion region (FD) of the low CG pixel, the size of the pixel signal generated by the high CG pixel may be greater than that of the pixel signal generated by the low CG pixel.

[0119] In addition, the size of the pixel signal of the low CG pixel may increase with a relatively low slope in response to an increase in the amount of photocharge, and the size of the pixel signal of the high CG pixel may increase with a relatively high slope in response to an increase in the amount of photocharge.

[0120] In addition, each of the size of the pixel signal of the low CG pixel and the size of the pixel signal of the high CG pixel may be converted into image data (IDATA) so that the response of the low CG pixel may follow Figure 2 The response of low-sensitivity pixels is shown, and the response of high-CG pixels can be followed Figure 2 The response of the high-sensitivity pixel is shown.

[0121] The image sensing device 100 according to another embodiment of the disclosed technology can simultaneously implement low-sensitivity pixels and high-sensitivity pixels in only one pixel array 110 , and thus can form (or generate) an HDR image using only one image.

[0122] Figure 8 is a graph showing an example of a method of adjusting the sensitivity of each pixel based on a difference in analog gain.

[0123] Reference Figure 8 , ADC 140 may be implemented as a ramp comparison ADC. In some implementations, the ramp comparison ADC may include a comparator for comparing a ramp signal that decreases over time with an analog pixel signal and a counter for performing counting until the ramp signal matches the analog pixel signal. In addition, ADC 140 may be independently provided for each column line to which pixels belonging to the same column of pixel array 110 are connected. Each ADC 140 may perform analog-to-digital conversion (ADC) using the same or different ramp signals.

[0124] exist Figure 8 In the graph shown, the X-axis may represent time and the Y-axis may represent voltage. Figure 8 1 and 2 are depicted in FIG. Each of the first ramp signal RAMP1 and the second ramp signal RAMP2 may be maintained at a constant voltage until reaching a first time point (t1), and then may be linearly decreased after the elapse of the first time point (t1).

[0125] The slope of the first ramp signal (RAMP1) may be smaller than the slope of the second ramp signal (RAMP2). The slope of the first ramp signal (RAMP1) and the slope of the second ramp signal (RAMP2) may be adjusted by controlling the resistance value of a variable resistor included in a ramp circuit configured to generate these ramp signals under the control of the timing controller 170, but is not limited thereto.

[0126] A time period from a first time point (t1) (at which each of the first ramp signal RAMP1 and the second ramp signal RAMP2 starts to linearly decrease) to a second time point (t2) after a predetermined time from the first time point (t1) may be defined as a countable range. The countable range may refer to a maximum time period during which the counter of the ADC 140 may perform counting. The countable range may be determined in response to a maximum number of counts per unit time of the counter, and may represent an output range of the ADC 140.

[0127] The input range of the ADC 140 may refer to a voltage range of a pixel signal that may be effectively converted into image data (IDATA) belonging to a predetermined output range (eg, a digital number (DN) of 0-1023) of the ADC 140 .

[0128] When the first ramp signal (RAMP1) is input to the comparator, the voltage range of the pixel signal that can be effectively converted into pixel data within a countable time or range determined by the output range of the ADC 140 may correspond to the first input range (IR1). Figure 8 In the example of , when the first ramp signal (RAMP1) and the pixel signal (PS) are input to the comparator, the counter starts counting from the first time point (t1), and the counting is continuously performed until the pixel signal (PS) reaches a third time point (t3) at which the value is equal to or higher than the first ramp signal (RAMP1), so that the cumulative count value can be output as image data. The pixel signal (PS) may be a signal belonging to the first input range (IR1) and may be effectively converted into image data. In this case, the operation of effectively converting the pixel signal (PS) into image data may mean that the pixel signal (PS) is converted into image data indicating the voltage of the pixel signal (PS). If the pixel signal (PS) has a voltage less than the first ramp signal (RAMP1) at the second time point (t2), the pixel signal (PS) cannot be effectively converted into pixel data.

[0129] When the second ramp signal (RAMP2) is input to the comparator, the voltage range of the pixel signal that can be effectively converted into pixel data within the countable time determined by the output range of the ADC 140 may correspond to the second input range (IR2). Figure 8 In the example, when the second ramp signal (RAMP2) and the pixel signal (PS) are input to the comparator, the counter starts counting from a first time point (t1) and continues counting until reaching a fourth time point (t4) when the pixel signal (PS) reaches a value equal to or higher than the second ramp signal (RAMP2), so that the accumulated count value can be output as image data.

[0130] One pixel or less connected to the ADC 140 performing ADC using the first ramp signal ( RAMP1 ) will be referred to as a high AG pixel, and one pixel or less connected to the ADC 140 performing ADC using the second ramp signal ( RAMP2 ) will be referred to as a low AG pixel.

[0131] In other words, for a pixel signal (PS) having the same voltage, image data output from the ADC 140 connected to a high AG pixel may be higher than image data output from the ADC 140 connected to a low AG pixel.

[0132] In addition, the image data output from the ADC 140 connected to the low AG pixel may increase with a relatively low slope in response to an increase in the pixel signal (for example, the increase in the pixel signal may indicate an increase in the absolute value of the pixel signal), and the image data output from the ADC 140 connected to the high AG pixel may increase with a relatively high slope in response to an increase in the pixel signal.

[0133] Therefore, the response of low AG pixels can follow Figure 2 The response of low-sensitivity pixels is shown, and the response of high-AG pixels can be followed Figure 2 The response of the high-sensitivity pixel is shown.

[0134] The image sensing device 100 according to another embodiment of the disclosed technology may generate an HDR image using only one image by implementing low-sensitivity pixels and high-sensitivity pixels in only one pixel array 110 .

[0135] Figures 5 to 8 The various embodiments described in the foregoing for adjusting the sensitivity of each pixel in response to a difference in transmittance, a difference in exposure time, a difference in conversion gain, and a difference in analog gain can be combined with each other as needed. For example, a CG transistor (CG) of a low-transmittance pixel (LPX) having a relatively low transmittance can be turned on to reduce the slope of the response, or a CG transistor (CG) of a low-transmittance pixel (LPX) can be turned off to increase the slope of the response. In addition, the sensitivity of each pixel can be adjusted using at least two selected from transmittance, exposure time, conversion gain, and analog gain.

[0136] Although for the sake of convenience, Figures 5 to 8 Each of the transmittance, exposure time, conversion gain, and analog gain described in is described as having only two types of pixels (eg, high transmittance pixel HPX and low transmittance pixel LPX), but it should be noted that Figures 5 to 8 Each of the transmittance, exposure time, conversion gain, and analog gain described in the embodiment may have three or more types (eg, high transmittance pixel (HPX), low transmittance pixel (LPX), and medium transmittance pixel).

[0137] Fig. 9 is a flow chart illustrating an example of a method of forming an HDR image by an imaging device based on some implementations of the disclosed technology. Fig.10 Some implementations based on the disclosed technology are shown in Fig. 9 A flow chart of an example of operation S30 is shown.

[0138] Reference Fig. 9, the brightness acquisition unit 310 may obtain the brightness of the target area in the scene image based on the image data (IDATA) generated by the image sensing device 100 ( S10 ).

[0139] The controllable item acquisition unit 320 may obtain at least one controllable item among the sensitivity items (e.g., exposure, transmittance, exposure time, conversion gain, and analog gain) (S20). Here, although the transmittance item belongs to the sensitivity item, the transmittance of each pixel is inevitably fixed in hardware, so that the controllable item acquisition unit 320 does not regard the transmittance item as a controllable item.

[0140] In response to the brightness of the target area received from the brightness acquisition unit 310 and the controllable items received from the controllable item acquisition unit 320, the setting value calculation unit 330 may calculate setting values ​​of the respective controllable items and may generate a control signal (CS) indicating the calculated setting values ​​(S30).

[0141] The setting value calculation unit 330 may calculate the setting values ​​of the respective controllable items so that the respective pixels of the image sensing device 100 have an optimal dynamic range. Here, the optimal dynamic range may refer to a dynamic range that can minimize noise while being as wide as possible.

[0142] In order to maximize the dynamic range, the amount of photocharge generated by the photoelectric conversion element (PD) should be increased to the maximum amount of photocharge unless saturated in the photoelectric conversion element (PD), and image data should be created with maximum gain (e.g., conversion gain and analog gain) unless saturated in ADC 140.

[0143] In this case, saturation within the photoelectric conversion element (PD) may indicate that the amount of photocharge has increased to exceed a full well capacity (FWC) indicating the amount of photocharge that can be maximally generated and accumulated by the photoelectric conversion element (PD). In addition, saturation within the ADC 140 may indicate that image data corresponding to the upper limit of the output range of the ADC 140 is generated.

[0144] The method for increasing the amount of photocharge generated by the photoelectric conversion element (PD) can be implemented as a method of increasing the opening degree of the aperture 20 through the aperture driver 40 so that the exposure amount can be increased, or can be implemented as a method of allowing the row driver 120 to increase the time period in which the transmission signal has a logic high level so that the exposure time can be increased.

[0145] The method for increasing the gain required to generate image data may be implemented as a method of turning on the CG transistor through the row driver 120 so that the conversion gain can be increased, or may be implemented as a method of reducing the slope of the ramp signal through the timing controller 170 so that the analog gain can be increased.

[0146] The set value calculation unit 330 may determine whether each pixel included in the target area is saturated based on the image data of the target area received from the brightness acquisition unit 310. The saturation of each pixel may conceptually include saturation in the photoelectric conversion element (PD) and saturation in the ADC 140. In some implementations, it is assumed that when the image data of each pixel corresponds to the upper limit of the output range of the ADC 140, this means that the corresponding pixel is saturated.

[0147] In addition, the setting value calculation unit 330 can determine whether each pixel is saturable by controlling the controllable items. To this end, the setting value calculation unit 330 can pre-store response characteristics indicating the response of the corresponding pixel to be changed in response to controlling each controllable item (e.g., transmittance, exposure time, conversion gain, and analog gain) to a specific setting value. These response characteristics can be determined experimentally by measuring the response of each pixel while changing the setting value of each controllable item. For example, based on the current image data of the high-transmittance pixel of the target area and the response characteristics of the high-transmittance pixel of the target area, the setting value calculation unit 330 can determine whether the high-transmittance pixel is saturable when the setting value of each controllable item (e.g., transmittance, exposure time, conversion gain, and / or analog gain) is controlled to maximize the sensitivity of the high-transmittance pixel. If the high-transmittance pixel cannot be saturated, it means that the current brightness is very low. As a result, although the exposure time is very short like the short-exposure pixel, the effect of dynamic range expansion cannot be obtained, thereby deteriorating the signal-to-noise ratio (SNR). Therefore, the setting value calculation unit 330 may exclude the exposure time from the controllable items and may forcibly set the exposure time to the longest exposure time.

[0148] In another embodiment of the disclosed technology, when the average value (i.e., average brightness) or the maximum value (i.e., maximum brightness) of the current image data of the high transmittance pixel (HPX) is lower than a predetermined value (for example, for a night shooting mode or a motion image shooting mode, etc.), the setting value calculation unit 330 may determine that there is an environment with very low brightness, may exclude the exposure time from the controllable items, and may set the exposure time to the longest time that serves as a unique value.

[0149] The method of adjusting the sensitivity of each pixel by controlling the transmittance can spatially adjust the amount of light incident on the pixel so that the low transmittance pixel (LPX) will not be saturated even at high brightness where the high transmittance pixel (HPX) is saturated, resulting in an effective response. As a result, the above-mentioned method of adjusting the sensitivity of each pixel by controlling the transmittance can significantly expand the dynamic range of each pixel by increasing the basic full well capacity (FWC). However, the transmittance may be pre-fixed in the process of manufacturing the image sensing device 100 and the transmittance cannot be dynamically controlled, so that the transmittance can be considered as an uncontrollable item.

[0150] The method of adjusting the sensitivity of each pixel by controlling the exposure time can temporarily adjust the amount of light incident on the pixel so that the low-exposure pixel is not saturated even at a high brightness where the long-exposure pixel is saturated, resulting in an effective response. As a result, the above-mentioned method of adjusting the sensitivity of each pixel by controlling the exposure time can significantly expand the dynamic range of each pixel by increasing the basic full well capacity (FWC). However, when the exposure time is over-adjusted, the time point of capturing the light incident on each pixel changes significantly. As a result, the motion artifacts of the fast-moving object to be captured increase, resulting in a decrease in image quality.

[0151] The method of adjusting the sensitivity of each pixel by controlling the conversion gain can control the conversion gain of converting the photocharge generated by the pixel into a pixel signal (ie, voltage), and even at low brightness where the amount of generated photocharge is considered insufficient, the amplified voltage can be obtained in response to a constant gain, thereby preventing the occurrence of noise. However, since an effective response cannot be obtained even when the conversion gain is controlled for a pixel in which the photoelectric conversion element (PD) is saturated, there is a limitation in extending the dynamic range of the pixel.

[0152] The method of adjusting the sensitivity of each pixel by controlling the analog gain can control the gain of converting the pixel signal into image data (e.g., digital value), and obtain amplified image data in response to a constant gain even at low brightness where the amount of generated photocharge is considered insufficient, thereby preventing the occurrence of noise. However, since an effective response cannot be obtained even when the analog gain is controlled for a pixel in which the photoelectric conversion element (PD) is saturated, there is a limitation in extending the dynamic range of the pixel. In addition, when the analog gain is increased, such as Figure 8 As shown, an input range corresponding to a voltage range of a pixel signal that can be effectively converted into image data by the ADC 140 is inevitably reduced, so that a side effect that a dynamic range is reduced according to brightness may occur.

[0153] The imaging device 1 can expand the dynamic range of the image data based on the characteristics of the controllable items that can be used to adjust the sensitivity of each pixel. In an embodiment, the imaging device 1 can adjust the sensitivity of each pixel in the order of transmittance, sensitivity control using conversion gain, sensitivity control using exposure time, and sensitivity control using analog gain, so that the imaging device 1 can expand the dynamic range through such sensitivity control. The order or priority of the sensitivity control discussed above can be determined based on the dynamic range expansion and side effects. In another embodiment, the above-mentioned sensitivity control priority can also be changed as needed. In addition, the imaging device 1 can calculate the set value of each controllable item so that the light intensity incident on each pixel can be maximized when controlling the sensitivity of each pixel.

[0154] In some implementations, the set values ​​of the controllable items may be calculated by the set value calculation unit 330, as will be described below with reference to Fig.10 For convenience of description, it is assumed that the conversion gain and the analog gain can always be controlled, and each of the exposure amount and the exposure time is controllable or uncontrollable. That is, the controllable item acquisition unit 320 can acquire any controllable item among the exposure amount and the exposure time.

[0155] It is also assumed that the arrangement and structure of the low-transmittance pixels and the high-transmittance pixels are the same as Figure 5 That is, for the convenience of description and better understanding of the disclosed technology, it is assumed that a single unit matrix includes one low-transmittance pixel and three high-transmittance pixels.

[0156] The setting value calculation unit 330 may determine whether the exposure amount is a controllable item ( S300 ).

[0157] If the exposure amount is an uncontrollable item (No, S300 ), the setting value calculation unit 330 may determine whether the exposure time is a controllable item (Yes, S310 ).

[0158] If the exposure time is a controllable item (yes, S310), the setting value calculation unit 330 may perform the first setting control (S320). Here, the first setting control (S320) may refer to a method of controlling the controllable items when the exposure amount is an uncontrollable item and each of the exposure time, the conversion gain, and the analog gain is a controllable item.

[0159] Specifically, the setting value calculation unit 330 can determine whether the requested dynamic range can be obtained from the target area by controlling the conversion gain of the high transmittance pixel and the conversion gain of the low transmittance pixel. In this case, the requested dynamic range may refer to a dynamic range suitable for photographing the target area.

[0160] In some implementations, when the ratio of high-transmittance pixels in the target area each having a predetermined value or less of response is less than a first ratio through the control of the conversion gain of each of the high-transmittance pixels and the low-transmittance pixels, and when the ratio of low-transmittance pixels in the target area each being saturated is equal to or less than a second ratio, the set value calculation unit 330 may determine whether the requested dynamic range can be obtained from the target area. On the contrary, regardless of the control of the conversion gain of each of the high-transmittance pixels and the low-transmittance pixels, when the ratio of high-transmittance pixels in the target area each having a predetermined value or less of response is equal to or higher than the first ratio, and when the ratio of low-transmittance pixels in the target area each being saturated is higher than the second ratio, the set value calculation unit 330 may determine that the requested dynamic range cannot be obtained from the target area. Here, the first ratio may refer to the ratio of the number of high-transmittance pixels in the target area each having a predetermined value or less to the total number of high-transmittance pixels. The second ratio may refer to the ratio of the number of low-transmittance pixels in the target area each being saturated to the total number of low-transmittance pixels in the target area. The first ratio and the second ratio may be predetermined by experiment in response to the requested dynamic range. For example, the predetermined value may refer to Figure 2 In addition, the first condition is that the ratio of high-transmittance pixels each having a predetermined value in the target area is less than the first ratio, and the second condition is that the ratio of low-transmittance pixels each saturated in the target area is equal to or less than the second ratio.

[0161] In another embodiment, the setting value calculation unit 330 may receive information on whether the requested dynamic range can be obtained from another structure of the image signal processor 200 (eg, a structure for synthesizing an HDR image).

[0162] In the first setting control, if it is assumed that the requested dynamic range can be obtained from the target area by controlling the conversion gain of each of the high-transmittance pixel and the low-transmittance pixel, the setting value calculation unit 330 can set or determine the maximum exposure time and the optimal analog gain for each of the high-transmittance pixel and the low-transmittance pixel. Here, when the ratio of the high-transmittance pixels each having a predetermined value in the target area is less than the first ratio, and when the ratio of the low-transmittance pixels each saturated in the target area is equal to or less than the second ratio, the maximum exposure time can be set to the maximum exposure time. In addition, the lowest analog gain can be the minimum analog gain that can be set or determined by the ADC 140. As the analog gain decreases, the input range of the ADC 140 can increase so that the dynamic range can be prevented from being accidentally limited.

[0163] If the requested dynamic range cannot be obtained from the target area by controlling the conversion gain of each of the high-transmittance pixel and the low-transmittance pixel, the setting value calculation unit 330 may expand the dynamic range by controlling the exposure time. Figure 6 It is seen that the length of the first exposure period (EX1) of the long exposure pixel will be referred to as the first exposure time below, and the length of the second exposure period (EX2) of the short exposure pixel will be referred to as the second exposure time below.

[0164] The setting value calculation unit 330 may determine the first exposure time so that the ratio of high-transmittance pixels each having a predetermined value or less in the target area is less than the first ratio. In addition, the setting value calculation unit 330 may determine the second exposure time so that the ratio of low-transmittance pixels each saturated in the target area is equal to or less than the second ratio. However, the time difference between the first exposure time and the second exposure time may be minimized to reduce motion artifacts. In order to maximize the dynamic range by maximizing the difference in sensitivity between low-transmittance pixels and high-transmittance pixels using the first exposure time and the second exposure time with the smallest difference therebetween, the second exposure time may be configured in the low-transmittance pixels, and the first exposure time may be configured in the high-transmittance pixels.

[0165] Available from Figure 7 It can be seen that the conversion gain of the high CG pixel will be referred to as the first conversion gain below, and the conversion gain of the low CG pixel will be referred to as the second conversion gain below. Figure 8 As depicted in FIG. 1 , the analog gain of the high AG pixel will be referred to as the first analog gain hereinafter, and the analog gain of the low AG pixel will be referred to as the second analog gain hereinafter. For convenience of description, it is assumed that the second analog gain is equal to the above-mentioned lowest analog gain.

[0166] The setting value calculation unit 330 may set or determine a second conversion gain, a second exposure time, and a second analog gain for the low transmittance pixel. In this case, the reason for configuring the second conversion gain and the second exposure time for the low transmittance pixel is to expand the dynamic range of each low transmittance pixel. The reason for configuring the second analog gain for the low transmittance pixel is to prevent limiting the dynamic range.

[0167] For high transmittance pixels, the setting value calculation unit 330 may set a plurality of conversion gains including a first conversion gain, a plurality of exposure times including a first exposure time, and a second analog gain. That is, the plurality of conversion gains may include a first conversion gain, the plurality of exposure times may include a first exposure time, so that the dynamic range of the high transmittance pixel can be extended, and the reason for configuring the second analog gain for the high transmittance pixel is to prevent limiting the dynamic range. In addition, the plurality of conversion gains and the plurality of exposure times for the high transmittance pixel are combined with each other and the combination result is set or configured for the high transmittance pixel so that the high transmittance pixel can have various dynamic ranges. When high transmittance pixels with various dynamic ranges are synthesized as described above, only image data with excellent SNR is synthesized to generate an HDR image, resulting in improved image quality. As an example of a unit matrix, the setting value calculation unit 330 may set a first conversion gain and a first exposure time for only one high transmittance pixel, may set a second conversion gain and a second exposure time for another high transmittance pixel, and may set a second conversion gain and a first exposure time for the remaining high transmittance pixels other than the above two high transmittance pixels.

[0168] If the exposure time is an uncontrollable item (No, S310 ), the setting value calculation unit 330 may determine whether the ratio of saturated low-transmittance pixels in the target area exceeds a second ratio ( S330 ).

[0169] If the ratio of saturated low-transmittance pixels in the target area exceeds the second ratio (yes, S330), the setting value calculation unit 330 may perform the second setting control (S340). In this case, the second setting control may refer to a method of controlling the controllable items under the condition that the exposure amount and exposure time are uncontrollable items, the conversion gain and the analog gain are controllable items, and the ratio of saturated low-transmittance pixels in the target area exceeds the second ratio. The above conditions may refer to a relatively high brightness condition in which low-transmittance pixels corresponding to a predetermined ratio or higher are saturated.

[0170] The set value calculation unit 330 may set a first conversion gain or a second conversion gain for low-transmittance pixels and high-transmittance pixels in a predetermined pattern. The set value calculation unit 330 may set or configure a first analog gain for each pixel having a first conversion gain. The set value calculation unit 330 may set or configure a second analog gain for each pixel having a second conversion gain. As a result, the conversion gain and the analog gain may be used to achieve the greatest possible sensitivity difference. However, when a pixel having a first conversion gain is set to have a first analog gain, there is a possibility that the dynamic range is limited instead. In another embodiment, the set value calculation unit 330 may set a second analog gain for a pixel having a first conversion gain, or may selectively set a first analog gain or a second analog gain for a pixel having a first conversion gain as needed.

[0171] If the ratio of saturated low-transmittance pixels in the target area is equal to or less than the second ratio (No, S330), the set value calculation unit 330 may determine whether the ratio of high-transmittance pixels in the target area, each having a response less than a predetermined value, is less than the first ratio (S350).

[0172] If the ratio of high-transmittance pixels each having a response of a predetermined value or less in the target area is less than the first ratio (yes, S350), the setting value calculation unit 330 may perform a third setting control (S360). In this case, the third setting control may refer to a method of controlling controllable items under the condition that the exposure amount and the exposure time are uncontrollable items, the conversion gain and the analog gain are controllable items, the ratio of saturated low-transmittance pixels in the target area is equal to or less than the second ratio, and the ratio of high-transmittance pixels each having a predetermined value or less in the target area is less than the first ratio. The above conditions may refer to appropriate brightness conditions in which the ratio of saturated low-transmittance pixels in the target area is equal to or less than the second ratio and the ratio of high-transmittance pixels each having a response of a predetermined value or less is less than the first ratio.

[0173] When setting the first conversion gain for the low-transmittance pixels, the set value calculation unit 330 may determine whether the ratio of the saturated low-transmittance pixels in the target area can be maintained at the second ratio or less. If the saturated low-transmittance pixels can be maintained at the second ratio or less, the set value calculation unit 330 may set the first conversion gain for the low-transmittance pixels. If the ratio of the saturated low-transmittance pixels exceeds the second ratio, the set value calculation unit 330 may set the second conversion gain for the low-transmittance pixels.

[0174] On the other hand, the set value calculation unit 330 may set the first conversion gain or the second conversion gain for the high-transmittance pixel in a predetermined pattern. In this case, the predetermined pattern may set the same conversion gain as the conversion gain of the low-transmittance pixel for the high-transmittance pixel belonging to the same row as the low-transmittance pixel, and may set another conversion gain different from the conversion gain of the low-transmittance pixel for the high-transmittance pixel belonging to another row different from the low-transmittance pixel, but is not limited thereto.

[0175] Under the condition that the conversion gain is set for each low-transmittance pixel and the conversion gain is set for each high-transmittance pixel as described above, the set value calculation unit 330 can set the maximum analog gain for each of the low-transmittance pixel and the high-transmittance pixel within the range that the ratio of the saturated low-transmittance pixels in the target area is equal to or less than the second ratio and the ratio of the high-transmittance pixels each having a response of a predetermined value or less is less than the first ratio. As a result, a dynamic range as large as possible can be obtained and the occurrence of noise can also be prevented.

[0176] If the ratio of high-transmittance pixels each having a response of a predetermined value or less in the target area is equal to or higher than the first ratio (No, S350), the setting value calculation unit 330 may perform a fourth setting control (S370). In this case, the fourth setting control may refer to a method of controlling controllable items under the conditions that the exposure amount and the exposure time are uncontrollable items, the conversion gain and the analog gain are controllable items, the ratio of saturated low-transmittance pixels in the target area is equal to or lower than the second ratio, and the ratio of high-transmittance pixels each having a predetermined value or less in the target area is equal to or higher than the first ratio. The above conditions may refer to a relatively low brightness condition in which the ratio of high-transmittance pixels each having a response of a predetermined value or less is equal to or higher than the first ratio.

[0177] The setting value calculation unit 330 may set a first conversion gain for the low-transmittance pixel.

[0178] On the other hand, the set value calculation unit 330 may set the first conversion gain or the second conversion gain for the high-transmittance pixel in a predetermined pattern. In this case, when each high-transmittance pixel is set to the first conversion gain and the ratio of the high-transmittance pixels each having a response of a predetermined value or less is equal to or higher than the first ratio, the set value calculation unit 330 may set the first conversion gain for the high-transmittance pixel instead of the predetermined pattern. This is because regardless of the second conversion gain set for the high-transmittance pixel, the dynamic range cannot be expected to be extended.

[0179] In a state where the conversion gain is set for the low-transmittance pixel and the conversion gain is set for the high-transmittance pixel as described above, the setting value calculation unit 330 can set the analog gain as high as possible for the low-transmittance pixel and the high-transmittance pixel while maintaining a specific condition that the ratio of the saturated low-transmittance pixels in the target area is equal to or less than the second ratio. As a result, the largest possible dynamic range can be obtained and the occurrence of noise can also be prevented.

[0180] If the exposure is a controllable item (Yes, S300), the setting value calculation unit 330 may set the exposure so that the aperture 20 may be maximally opened within a range in which the ratio of saturated low-transmittance pixels in the target area is equal to or less than the second ratio (S380).

[0181] In a state where the exposure amount is fixed to a specific setting value, the setting value calculation unit 330 may determine whether the exposure time is a controllable item ( S390 ).

[0182] If the exposure time is a controllable item (Yes, S390), the setting value calculation unit 330 may perform the first setting control (S320). The reason for performing the first setting control is that the exposure time is considered to be a controllable item under the condition that the exposure amount is fixed and uncontrollable.

[0183] If the exposure time is an uncontrollable item (No, S390), the setting value calculation unit 330 may perform the third setting control (S360). The reason for performing the third setting control is as follows. In a state where the exposure amount is fixed and uncontrollable and the exposure time is a controllable item, the aperture 20 may be set to be maximally opened within a range where the ratio of the low-transmittance pixels saturated in the target area is equal to or less than the second ratio, so that there is a high probability that the ratio of the high-transmittance pixels each having a response of a predetermined value or less in the target area is less than the first ratio. As a result, the setting value calculation unit 330 may perform the third setting control (S360).

[0184] Return to reference Fig. 9 , the setting value calculation unit 330 may send a control signal (CS) indicating the setting value of each controllable item to the aperture driver 40 and the image sensing device 100. The aperture driver 40 may control the opening degree of the aperture 20 so that the aperture driver 40 has an exposure amount corresponding to the control signal (CS). In addition, the timing controller 170 of the image sensing device 100 may control the row driver 120 and the ADC 140 so that each pixel has an exposure time, a conversion gain, and an analog gain corresponding to the control signal (CS).

[0185] In a state where the control of the control signal (CS) is completed, the image sensing device 100 may capture an image of the scene to form image data (IDATA), and may send the image data (IDATA) to the image signal processor 200. The image signal processor 200 may synthesize at least two images having different sensitivities, and thus may form an HDR image using the result of the image synthesis (S40). The sensitivity type of the image data (IDATA) may be determined in various ways by a combination of transmittance, exposure time, conversion gain, and analog gain. For example, the pixels included in the pixel array 110 may have any one of five sensitivities, and each of the two pixels having different sensitivities may correspond to a low-sensitivity pixel or a high-sensitivity pixel according to the magnitude relationship of the sensitivity between the two pixels.

[0186] An imaging device 1 implemented based on some embodiments of the disclosed technology can determine the brightness of a subject to be captured and one or more controllable items to control the controllable items by reflecting hardware characteristics (e.g., response characteristics) when controlling sensitivity, thereby generating an HDR image with a maximum dynamic range while minimizing noise.

[0187] In various implementations, an imaging device based on some implementations of the disclosed technology may determine the brightness and controllable items of a target object to be captured (or photographed) and control the controllable items by reflecting hardware characteristics (e.g., response characteristics) into the process of controlling the sensitivity of the imaging device, thereby generating a high dynamic range (HDR) image with a maximum dynamic range while minimizing noise.

[0188] Although a number of exemplary embodiments have been described, it should be understood that modifications and enhancements to the disclosed embodiments and other embodiments are contemplated based on what is described and / or illustrated in this patent document.

[0189] CROSS-REFERENCE TO RELATED APPLICATIONS

[0190] This patent document claims priority to and the benefit of Korean Patent Application No. 10-2021-0096596, filed on Jul. 22, 2021, the disclosure of which is incorporated herein by reference in its entirety as a part of the disclosure of this patent document.

Claims

1. An imaging device, comprising: an image sensing device including a plurality of pixels, each pixel detecting incident light from a scene to generate a pixel signal corresponding to an intensity of the incident light and generating image data corresponding to the pixel signal, wherein the image sensing device operates in response to a control signal to perform an imaging operation; a brightness acquisition unit that acquires brightness information corresponding to a first pixel associated with a target area of ​​the image of the scene captured by the image sensing device among the plurality of pixels; a controllable item acquisition unit that acquires one or more sensitivity items indicating the sensitivity of each pixel to light as controllable items; and a set value calculation unit that generates the control signal for the image sensing device by calculating a set value of the controllable item based on the brightness information of the target area and the controllable item, wherein the one or more sensitivity items include at least one of exposure, transmittance, exposure time, conversion gain, and analog gain, and The setting value calculation unit determines whether each of the first pixels is saturated by controlling the controllable items.

2. The imaging device according to claim 1, further comprising: An image synthesis unit synthesizes image data with different sensitivities generated by adjusting the sensitivity of each of the plurality of pixels in response to the control signal; and generates a high dynamic range (HDR) image of the scene captured by the image sensing device.

3. The imaging device according to claim 1, further comprising: an aperture driver that adjusts the opening or closing degree of an aperture disposed in front of the image sensing device, The exposure amount is controlled by adjusting the opening or closing degree of the aperture.

4. The imaging device according to claim 1, further comprising: a row driver that generates a transfer signal for transferring photocharges generated in each of the plurality of pixels in response to the intensity of the incident light to a floating diffusion region, The exposure time is controlled by adjusting the time period in which the transmission signal has a predetermined level.

5. The imaging device according to claim 1, further comprising: a row driver that generates a conversion gain signal for controlling a capacitance of a floating diffusion region that accumulates photocharges, The conversion gain is controlled by adjusting the voltage level of the conversion gain signal.

6. The imaging device according to claim 1, further comprising: an analog-to-digital converter that generates the image data in response to comparing the pixel signal of each of the plurality of pixels with a ramp signal, The analog gain is controlled by adjusting the slope of the ramp signal.

7. The imaging device according to claim 1, wherein: The plurality of pixels include at least one high-transmittance pixel and at least one low-transmittance pixel, wherein the transmittance of the high-transmittance pixel is higher than the transmittance of the low-transmittance pixel, The low-transmittance pixel includes a light blocking structure for blocking at least a portion of the incident light.

8. The imaging device according to claim 7, wherein: The set value calculation unit sets the exposure time to a fixed value when the high-transmittance pixels included in the first pixels are not saturated.

9. An imaging device, comprising: an image sensing device including a plurality of pixels, each pixel detecting incident light from a scene to generate a pixel signal corresponding to an intensity of the incident light and generating image data corresponding to the pixel signal, wherein the image sensing device operates in response to a control signal to perform an imaging operation; a brightness acquisition unit that acquires brightness information corresponding to a first pixel associated with a target area of ​​the image of the scene captured by the image sensing device among the plurality of pixels; a controllable item acquisition unit that acquires one or more sensitivity items indicating the sensitivity of each pixel to light as controllable items; and a set value calculation unit that generates the control signal for the image sensing device by calculating a set value of the controllable item based on the brightness information of the target area and the controllable item, wherein the one or more sensitivity items include at least one of exposure, transmittance, exposure time, conversion gain, and analog gain, The plurality of pixels include at least one high-transmittance pixel and at least one low-transmittance pixel, wherein the transmittance of the high-transmittance pixel is higher than the transmittance of the low-transmittance pixel, wherein the low-transmittance pixel comprises a light blocking structure for blocking at least a portion of the incident light, and When an average value of image data of the high-transmittance pixels included in the first pixels is less than a predetermined value, the setting value calculation unit sets the exposure time to a fixed value.

10. An imaging device, comprising: an image sensing device including a plurality of pixels, each pixel detecting incident light from a scene to generate a pixel signal corresponding to an intensity of the incident light and generating image data corresponding to the pixel signal, wherein the image sensing device operates in response to a control signal to perform an imaging operation; a brightness acquisition unit that acquires brightness information corresponding to a first pixel associated with a target area of ​​the image of the scene captured by the image sensing device among the plurality of pixels; a controllable item acquisition unit that acquires one or more sensitivity items indicating the sensitivity of each pixel to light as controllable items; and a set value calculation unit that generates the control signal for the image sensing device by calculating a set value of the controllable item based on the brightness information of the target area and the controllable item, wherein the one or more sensitivity items include at least one of exposure, transmittance, exposure time, conversion gain, and analog gain, The plurality of pixels include at least one high-transmittance pixel and at least one low-transmittance pixel, wherein the transmittance of the high-transmittance pixel is higher than the transmittance of the low-transmittance pixel, wherein the low-transmittance pixel comprises a light blocking structure for blocking at least a portion of the incident light, The controllable items include the exposure time, the conversion gain and the analog gain, and the exposure amount is an uncontrollable item. The setting value calculation unit sets a plurality of conversion gains including a first conversion gain for the high-transmittance pixel, sets a plurality of exposure times including a first exposure time for the high-transmittance pixel, and sets a first analog gain for the high-transmittance pixel, and The setting value calculation unit sets a second conversion gain, a second exposure time and a second analog gain for the low-transmittance pixel. in, The first conversion gain is greater than the second conversion gain; The first exposure time is longer than the second exposure time; and The second analog gain is smaller than the first analog gain.

11. The imaging device according to claim 10, wherein: The set value calculation unit: determining the first exposure time so that a ratio of the high-transmittance pixels in the target area each having a response of a predetermined value or less is smaller than a first ratio; and The second exposure time is determined so that a ratio of the saturated low-transmittance pixels in the target area is less than or equal to a second ratio.

12. The imaging device according to claim 11, wherein: In a case where the at least one controllable item includes the conversion gain and the analog gain, and the exposure amount and the exposure time are uncontrollable items, when the ratio of the saturated low-transmittance pixels in the target area is higher than the second ratio, The setting value calculation unit sets the first conversion gain or the second conversion gain for the low-transmittance pixel and the high-transmittance pixel in a predetermined pattern, The set value calculation unit sets the first analog gain for the pixel set to the first conversion gain, and sets the second analog gain for the pixel set to the second conversion gain.

13. The imaging device according to claim 11, wherein: In a case where the at least one controllable item includes the conversion gain and the analog gain, and the exposure amount and the exposure time are uncontrollable items, when the ratio of the low-transmittance pixels saturated in the target area is equal to or less than the second ratio, and the ratio of the high-transmittance pixels in the target area each having a response of a predetermined value or less is less than the first ratio, The set value calculation unit determines whether the ratio of the low-transmittance pixels saturated in the target area is maintained at the second ratio or less in a state where the first conversion gain is set for the low-transmittance pixels, sets the first conversion gain or the second conversion gain for the low-transmittance pixels according to a result of the determination, and sets the first conversion gain or the second conversion gain for the high-transmittance pixels in a predetermined pattern; and The set value calculation unit sets the maximum analog gain for the low transmittance pixels and the high transmittance pixels within a predetermined range that maintains the conditions that the ratio of the low transmittance pixels indicating saturation in the target area is equal to or less than the second ratio and the ratio of the high transmittance pixels in the target area each having a response of the predetermined value or less is less than the first ratio.

14. The imaging device according to claim 11, wherein: In a case where the at least one controllable item includes the conversion gain and the analog gain, and the exposure amount and the exposure time are uncontrollable items, when the ratio of the low-transmittance pixels saturated in the target area is equal to or less than the second ratio, and the ratio of the high-transmittance pixels in the target area each having a response of a predetermined value or less is equal to or higher than the first ratio, The set value calculation unit sets the first conversion gain for the low-transmittance pixel, and sets the first conversion gain or the second conversion gain for the high-transmittance pixel in a predetermined pattern; and The set value calculation unit sets maximum analog gains for the low-transmittance pixels and the high-transmittance pixels within a predetermined range that maintains a condition that the ratio of the low-transmittance pixels indicating saturation in the target area is equal to or smaller than the second ratio.

15. The imaging device according to claim 11, wherein The controllable item includes the exposure amount, wherein the setting value calculation unit sets the exposure amount so that the aperture is maximally opened within a predetermined range in which a ratio of the low-transmittance pixels saturated in the target area is equal to or less than the second ratio.

16. An imaging device, comprising: an image sensing device including a plurality of pixels, each pixel detecting incident light from a scene to generate a pixel signal corresponding to an intensity of the incident light and generating image data corresponding to the pixel signal, wherein the image sensing device operates in response to a control signal to perform an imaging operation; a brightness acquisition unit that acquires brightness information corresponding to a first pixel associated with a target area of ​​the image of the scene captured by the image sensing device among the plurality of pixels; a controllable item acquisition unit that acquires one or more sensitivity items indicating the sensitivity of each pixel to light as controllable items; and a set value calculation unit that generates the control signal for the image sensing device by calculating a set value of the controllable item based on the brightness information of the target area and the controllable item, wherein the one or more sensitivity items include at least one of exposure, transmittance, exposure time, conversion gain, and analog gain, and The setting value calculation unit calculates the setting value so as to adjust the sensitivity of each of the pixels in the order of the transmittance, the conversion gain, the exposure time, and the analog gain.

17. An imaging device, comprising: an image sensing device including a plurality of pixels that detect incident light to generate pixel signals corresponding to the intensity of the incident light and generate image data corresponding to the pixel signals; a high dynamic range (HDR) controller that generates a control signal by calculating a set value for at least one of an exposure amount, an exposure time, a conversion gain, and an analog gain as controllable items based on the image data; as well as an image signal processor coupled to receive the control signal from the high dynamic range HDR controller, and the image signal processor synthesizes image data with different sensitivities generated by adjusting the sensitivity of each of the plurality of pixels in response to the received control signal and generates a high dynamic range HDR image based on the image data, The high dynamic range (HDR) controller determines whether pixels associated with a target area of ​​an image of a scene captured by the image sensing device among the plurality of pixels are saturated by controlling the controllable item.

18. The imaging device according to claim 17, further comprising: a row driver that generates a transfer signal for transferring photocharges generated in each of the plurality of pixels in response to the intensity of the incident light to a floating diffusion region, The exposure time is controlled by adjusting the time period in which the transmission signal has a predetermined level.

Images