Imaging apparatus and operating method of imaging apparatus

Abstract

An imaging apparatus and an operating method of the imaging apparatus are provided. The imaging apparatus includes: an image sensor including a pixel array including a plurality of pixels in a matrix form, the image sensor configured to output first image data acquired in a rolling shutter mode and second image data acquired in a global shutter mode; and at least one processor configured to process the first image data and the second image data, and output third image data based on the processing.

Description

[0001] This application claims priority to Korean Patent Application No. 10-2024-0147951, filed on October 25, 2024, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference. Technical Field

[0002] Embodiments of this disclosure relate to an imaging device and a method of operating the imaging device. Specifically, embodiments of this disclosure relate to an imaging device and a method of operating the imaging device for generating an image with improved quality using image data acquired in each of a global shutter mode and a rolling shutter mode. Background Technology

[0003] An image sensor is a device that converts light signals into electrical signals. The processor of an imaging device can perform image signal processing on the image data output from the image sensor to improve the quality of the image data.

[0004] Imaging devices are being developed to improve image quality in various environments. For example, in addition to conventional imaging devices equipped with image sensors operating in rolling shutter mode, there are also imaging devices equipped with image sensors operating in global shutter mode. The rolling shutter effect can exist in the image data output by image sensors operating in rolling shutter mode. Summary of the Invention

[0005] One or more embodiments provide an imaging device and a method of operating the imaging device, the imaging device having an image sensor capable of acquiring images in each of a global shutter mode and a rolling shutter mode, and generating an image with improved quality by using image data acquired in each of the global shutter mode and the rolling shutter mode.

[0006] According to one or more embodiments, an imaging device is provided, comprising: an image sensor including a pixel array comprising a plurality of pixels in a matrix form, the image sensor being configured to output first image data acquired in a rolling shutter mode and second image data acquired in a global shutter mode; and at least one processor configured to process the first image data and the second image data, and output third image data based on the processing.

[0007] According to another aspect of one or more embodiments, an imaging device is provided, comprising: an image sensor including a pixel array comprising a plurality of pixels in a matrix form, the image sensor being configured to generate first image data in a rolling shutter mode and second image data in a global shutter mode based on a shooting command received from at least one processor, and to output the first image data and the second image data; a memory device configured to store the first image data, the second image data, and at least one command executed by the at least one processor, the memory device being electrically connected to the at least one processor; and the at least one processor being configured to execute a shooting command sent to the image sensor by user input for acquiring an image or video based on a command, and to generate third image data by changing the pixel position of the pixel data of the first image data based on the second image data or by changing the pixel value of the pixel data of the second image data based on the first image data.

[0008] According to another aspect of one or more embodiments, an operating method for an imaging device is provided, comprising: receiving user input commanding to acquire an image or video; sending a first command for acquiring the image or video to an image sensor based on the user input via at least one processor; generating first image data in a rolling shutter mode via the image sensor based on the first command; outputting the first image data via the image sensor; generating second image data in a global shutter mode via the image sensor based on the first command; outputting the second image data via the image sensor; and generating third image data based on the first image data and the second image data via the at least one processor. Attached Figure Description

[0009] The embodiments will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings.

[0010] Figure 1 This is a diagram illustrating an imaging apparatus according to one or more embodiments.

[0011] Figure 2 This is a diagram illustrating an image sensor according to one or more embodiments.

[0012] Figure 3A and Figure 3B It is a timing diagram describing the operation of an image sensor in global shutter mode and rolling shutter mode according to one or more embodiments.

[0013] Figure 4 It is a block diagram of a group of pixels according to one or more embodiments.

[0014] Figure 5 It is a circuit diagram of a pixel group according to one or more embodiments.

[0015] Figure 6A and Figure 6B This is a diagram describing a sub-pixel according to one or more embodiments.

[0016] Figure 7A , Figure 7B , Figure 7C , Figure 7D , Figure 7E and Figure 7F This is a diagram illustrating the operation of an imaging device according to one or more embodiments, based on a shooting mode.

[0017] Figure 8A , Figure 8B and Figure 8C This is a diagram illustrating the image data used for synthesis when an imaging device according to one or more embodiments performs image data synthesis.

[0018] Figure 9A , Figure 9B , Figure 9C , Figure 9D and Figure 9E It is a description based on Figure 8A A diagram illustrating the operation of an imaging device to synthesize global shutter image data and rolling shutter image data.

[0019] Figure 10 It is a description based on Figure 8B A diagram illustrating the operation of an imaging device to synthesize global shutter image data and rolling shutter image data.

[0020] Figure 11A and Figure 11B It is a description based on Figure 8C An illustration of the operation of the imaging device in an embodiment to synthesize global shutter image data and rolling shutter image data.

[0021] Figure 12 This is a block diagram describing an image sensor according to one or more embodiments.

[0022] Figure 13 This is a block diagram describing an image sensor according to one or more embodiments.

[0023] Figure 14 It is a block diagram describing an imaging apparatus according to one or more embodiments.

[0024] Figure 15 It is a flowchart describing a method of operating an imaging device according to one or more embodiments. Detailed Implementation

[0025] One or more embodiments will be described below.

[0026] It will be understood that although the terms first, second, third, fourth, etc., may be used herein to describe various elements, components, regions, layers, and / or portions (collectively, “elements”), these elements should not be limited by these terms. These terms are used only to distinguish one element from another. Therefore, without departing from the teachings of this disclosure, a first element described in this specification may be referred to as a second element in the claims section, and vice versa.

[0027] It should be understood that when an element or layer is referred to as being "above," "on," "on," "below," "under," "below," "connected to," or "bonded to" another element or layer, the element or layer may be directly above, above, on, below, or below the other element or layer, or intermediate elements or layers may exist. In contrast, when an element is referred to as being "directly above," "directly above," "directly below," "directly under," "directly connected to," or "directly bonded to" another element or layer, no intermediate elements or layers exist.

[0028] As used herein, the expression "at least one of" following the list of elements modifies the entire list of elements, not individual elements within the list. For example, the expression "at least one of a, b, and c" should be understood to include only a, only b, only c, both a and b, both a and c, both b and c, or all of a, b, and c.

[0029] Figure 1 This is an exemplary block diagram of an image sensor 10 according to one or more embodiments.

[0030] Imaging device 10 according to one or more embodiments can process image data acquired by image sensor 100 in each of global shutter mode and rolling shutter mode to generate an image with reduced rolling shutter effect.

[0031] In this specification, global shutter image data can be image data output by the image sensor 100 operating in global shutter mode. Roll shutter image data can be image data output by the image sensor 100 operating in roll shutter mode.

[0032] Imaging device 10 may include image sensor 100 and processor 200 (e.g., at least one processor 200).

[0033] Image sensor 100 according to one or more embodiments may operate in one of a global shutter mode and a rolling shutter mode. In one or more embodiments, image sensor 100 may operate alternately in global shutter mode and rolling shutter mode based on a command CMD from processor 200.

[0034] For example, image sensor 100 may operate in rolling shutter mode and output at least one frame of rolling shutter image data IDT1 based on a first command from processor 200 to acquire a still image, and then operate again in global shutter mode to output at least one frame of global shutter image data IDT2. As another example, in reverse order, image sensor 100 may first output at least one frame of global shutter image data IDT2, and then switch to output at least one frame of rolling shutter image data IDT1.

[0035] As another example, image sensor 100 may operate in rolling shutter mode and output at least one frame of rolling shutter image data IDT1 based on a second command to acquire video from processor 200, and then operate again in global shutter mode to output at least one frame of global shutter image data IDT2.

[0036] Image sensor 100 can directly or indirectly transmit rolling shutter image data IDT1 and global shutter image data IDT2 to processor 200. For example, image sensor 100 can directly transmit image data to processor 200 via a communication line connected to processor 200, or it can transmit image data to a separate interface device. Image data transmitted to a separate interface device can be stored in a memory device, and processor 200 can load and process the image data from the memory device.

[0037] Image sensor 100 can generate rolling shutter image data IDT1 and global shutter image data IDT2 based on the same pixels. For example, at least one pixel can operate in rolling shutter mode to output a pixel signal for generating rolling shutter image data IDT1, and operate in global shutter mode to output a pixel signal for generating global shutter image data IDT2.

[0038] In one or more embodiments, the rolling shutter image data IDT1 may include metadata indicating that the rolling shutter image data IDT1 is acquired in rolling shutter mode. The global shutter image data IDT2 may include metadata indicating that the global shutter image data IDT2 is acquired in global shutter mode. Therefore, the processor 200 can verify whether the received image data is rolling shutter image data IDT1 or global shutter image data IDT2.

[0039] The processor 200 can process the rolling shutter image data IDT1 and the global shutter image data IDT2 output by the image sensor 100, and generate image data IDT3 as the result of the processing. In one or more embodiments, the processor 200 can synthesize the rolling shutter image data IDT1 and the global shutter image data IDT2, and generate image data IDT3 as the result of the synthesis.

[0040] According to one or more embodiments, synthesizing multiple image data can mean using information from one image data and information from other image data to generate at least one pixel data of the synthesized image data. The image data may include multiple pixel data. To generate the pixel data, the processor 200 may use at least one of the multiple image data to determine the pixel value of the pixel data in the synthesized image data. As another example, the processor 200 may use at least one of the multiple image data to determine the pixel position of the pixel data in the synthesized image data. The pixel position may be the position of the pixel data in an image frame and / or the order of the pixel data in the image data.

[0041] Imaging apparatus 10 according to one or more embodiments can synthesize rolling shutter image data IDT1 and global shutter image data IDT2 output from the same image sensor 100. Compared with rolling shutter image data IDT1, synthesized image data IDT3 may have reduced rolling shutter effect. Furthermore, compared with global shutter image data IDT2, synthesized image data IDT3 may have reduced noise. Therefore, imaging apparatus 10 can output image data with improved quality.

[0042] Figure 2 This is an exemplary block diagram of an image sensor 100 according to one or more embodiments.

[0043] The image sensor 100 according to one or more embodiments can operate in global shutter mode or rolling shutter mode based on the shooting mode control signal (or mode control signal) MC.

[0044] In one or more embodiments, each pixel group PXG of the image sensor 100 can output a pixel signal in both a global shutter mode and a rolling shutter mode. Each pixel group PXG may include a rolling shutter circuit for outputting the pixel signal to an output line in the rolling shutter mode and a global shutter circuit for outputting the pixel signal to an output line in the global shutter mode.

[0045] In one or more embodiments, each pixel group PXG may include a plurality of pixels, and at least some of the plurality of pixels may share pixel circuitry.

[0046] In one or more embodiments, the output lines through which pixel signals are output in rolling shutter mode and the output lines through which pixel signals are output in global shutter mode may be different among the output lines connected to each pixel group PXG of the image sensor 100.

[0047] Reference Figure 2 The image sensor 100 will be described in detail.

[0048] The image sensor 100 can generate image data as visual information of an object acquired through a lens, and the image signal processor can be implemented to process the image data generated by the image sensor 100 and output it to a display device or store it on a storage device.

[0049] Image sensor 100 may include pixel array 110, row driver 120, timing controller 130, ramp signal generator (or ramp generator) 140, readout circuitry 150 and output buffer 160.

[0050] Pixel array 110 may include multiple pixel groups PXG. Pixel array 110 may receive multiple pixel drive signals CSn (such as select signals controlling select transistors, reset signals controlling reset transistors, and transmit transistor control signals controlling transmit transistors) from row driver 120 via row line RLn. Each of the multiple pixels PX in pixel array 110 may be operated according to the control of the received pixel drive signal CSn.

[0051] Multiple pixel groups (PXGs) can be arranged, for example, in a matrix. Each pixel group (PXG) can be electrically connected to at least one row line and at least one column line among multiple row lines (RWn) and multiple column lines (CLm). In one or more embodiments, each pixel group (PXG) may include multiple transistors controlled by row driver 120.

[0052] In one or more embodiments, each pixel group PXG may include a plurality of pixels, and at least some of the plurality of pixels may share pixel circuitry.

[0053] A pixel group (PXG) may include at least one photoelectric conversion element that converts an incident light signal into an electrical signal.

[0054] The photoelectric conversion element can be a photodiode (PD). The photoelectric conversion element can be one or a combination of a photodiode (PD), a photocapacitor, a photogate, a pinned photodiode (PPD), a partially pinned photodiode, an organic photodiode (OPD), and a quantum dot (QD). Embodiments of this disclosure will be described under the assumption that the photoelectric conversion element is a PD; however, other photoelectric conversion elements described above can be used, and the photoelectric conversion element is not limited to a PD.

[0055] The row driver 120 can drive a pixel group PXG arranged in one row of the pixel array 110 or a pixel group PXG arranged in multiple rows, according to the control of the timing controller 130.

[0056] The pixel signal PXS of pixel group PXG can be sent to readout circuit 150 through multiple column lines CLm.

[0057] The pixel signal PXS may include a reset voltage signal and a pixel voltage signal. The pixel voltage signal may be the voltage of the floating diffusion region, which reflects the charge generated from the PD included in each of the multiple pixels. The reset voltage signal may be the voltage of the floating diffusion region used as a reference voltage for performing correlated double sampling (CDS) with the pixel voltage signal.

[0058] The timing controller 130 can control the pixel array 110, the row driver 120, the ramp signal generator 140, and the readout circuit 150. The timing controller 130 can provide the timing control signal TC to the row driver 120.

[0059] The timing control signal TC can be set differently according to one or more embodiments based on the shooting mode control signal MC. For example, the shooting mode control signal MC is a signal based on a shooting mode selected by the user, and the shooting mode may include still image mode, video mode, etc. The shooting mode control signal MC may be a signal that controls the image sensor 100 to operate in global shutter mode or rolling shutter mode.

[0060] In one or more embodiments, the timing controller 130 may directly or indirectly obtain from Figure 1 The processor 200 receives the shooting mode control signal MC.

[0061] The line driver 120 can operate each of multiple pixel groups PXG in global shutter mode or rolling shutter mode based on the timing control signal TC.

[0062] In one or more embodiments, when the line driver 120 drives each of a plurality of pixel groups PXG in rolling shutter mode, the line driver 120 is operable to provide a plurality of conversion gain modes for the pixel groups PXG.

[0063] The timing controller 130 can control the ramp signal generator 140 via the ramp control signal CS_RP, and control the readout circuit 150 via the ADC control signal CS_ADC. The ramp control signal CS_RP may include a ramp enable signal, a mode signal, etc.

[0064] The ramp signal generator 140 can generate a ramp signal RAMP in response to the ramp control signal CS_RP. The ramp signal generator 140 can generate a ramp signal RAMP with a preset slope. The ramp signal generator 140 can provide the generated ramp signal RAMP to the readout circuit 150.

[0065] The readout circuit 150 can convert the reset voltage signal and pixel voltage signal in the pixel signal PXS into pixel data PXD as a digital signal based on the ramp signal RAMP, and output the converted pixel data PXD. For example, the readout circuit 150 can use a correlated double sampling method to convert each of the reset voltage signal and pixel voltage signal into a digital signal based on the ramp signal RAMP, and output the difference between the reset voltage signal and the pixel voltage signal as pixel data PXD, where pixel data PXD is a digital signal.

[0066] The readout circuit 150 may include a comparator and a counting circuit. The pixel signal PXS and the ramp signal RAMP may be provided to the comparator. The counting circuit may count a clock signal corresponding to the level of the reset voltage signal and the level of the pixel voltage signal. The counting circuit may generate the difference between the level of the reset voltage signal and the level of the pixel voltage signal as pixel data PXD, whereby pixel data PXD is a digital signal.

[0067] The output buffer 160 may include multiple column memory blocks corresponding to each column to store pixel data PXD. The output buffer 160 may include a sense amplifier SA for amplifying the pixel data PXD stored in the column memory blocks. The sense amplifier SA may output the amplified pixel data PXD as image data IDT.

[0068] Figure 3A It is shown Figure 1 A diagram illustrating the operation of the global shutter mode of the image sensor 100. Figure 3B It is shown Figure 1 Timing diagram of the operation of the rolling shutter mode of the image sensor 100.

[0069] Reference Figure 1 and Figure 3AWhen operating in global shutter mode, image sensor 100 can control each pixel group PXG such that the photocharge integration time P1 of the PD lies within the same time interval of all pixels in pixel array 110. The photocharge integration time can represent the time during which the PD essentially integrates the photocharge. Image sensor 100 can perform signal dumping (e.g., exposure) operations within the photocharge integration time P1.

[0070] In one or more embodiments, a frame corresponding to an image signal is read out. Figure 2 The time required for all pixel groups PXG of pixel array 110 can be the frame readout time P2.

[0071] In one or more embodiments, in global shutter mode, image sensor 100 can perform a rolling readout operation. For example, image sensor 100 can control each pixel group PXG such that the time interval for performing readout after integration time is different for each row R1, R2, ..., Rn of the pixel array. As another example, the time interval for performing readout can be different for rows of a specific group. In this case, readout for pixel groups PXG located in rows of the same group can be performed within the same time interval.

[0072] Reference Figure 1 and Figure 3B When the image sensor 100 operates in rolling shutter mode, it can control each pixel group PXG such that the time interval for the photocharge integration time P3 of the PD differs for each row R1, R2, ..., Rn of the pixel array. In one or more embodiments, the time interval for performing photocharge integration may differ for each row R1, R2, ..., Rn. As another example, the time interval for performing photocharge integration may differ for a specific group of rows. In this case, the photocharge integration of pixel groups PXG located in the same group of rows can be performed within the same time interval.

[0073] In one or more embodiments, in rolling shutter mode, the image sensor 100 can perform a rolling readout operation. For example, the image sensor 100 can control each pixel group PXG such that the time interval P4 for performing readout after the integration time is different for each row R1, R2, ..., Rn of the pixel array. As another example, the time interval for performing readout can be different for rows of a specific group. In this case, readout for pixel groups PXG located in rows of the same group can be performed within the same time interval.

[0074] Figure 4 This is a block diagram of a pixel group (PXG) according to one or more embodiments. (See also...) Figure 4 The described pixel group PXG can correspond to Figure 1 The pixel group PXG in the image. (Refer to...) Figure 4 A pixel group PXG is described according to one or more embodiments.

[0075] Reference Figure 4 According to one or more embodiments, a pixel group PXG may include a plurality of pixels PX1, PX2, PX3 and PX4, a first rolling shutter circuit 113, a second rolling shutter circuit 114, a global selection circuit 115 and a global shutter circuit 116. Figure 4 The pixel group PXG is shown to include four pixels PX1, PX2, PX3 and PX4, but the invention is not limited thereto, and the pixel group PXG may include different numbers of pixels.

[0076] In one or more embodiments, at least some of the plurality of pixels PX1, PX2, PX3 and PX4 may share at least some of the pixel circuitry.

[0077] For example, refer to Figure 4 Multiple pixels PX1, PX2, PX3, and PX4 can share a global selection circuit 115 and a global shutter circuit 116. The first pixel PX1 and the third pixel PX3 can share a first rolling shutter circuit 113. The second pixel PX2 and the fourth pixel PX4 can share a second rolling shutter circuit 114.

[0078] In one or more embodiments, each of the plurality of pixels PX1, PX2, PX3 and PX4 may include at least one PD.

[0079] In one or more embodiments, at least some of the pixels PX1, PX2, PX3, and PX4 can operate simultaneously in global shutter mode or rolling shutter mode.

[0080] For example, in rolling shutter mode, the first pixel PX1 and the third pixel PX3 can be simultaneously controlled as a sub-pixel group 111. The first pixel PX1 and the third pixel PX3 can simultaneously send integrated photocharge to the first rolling shutter circuit 113 via the first line LN1. Furthermore, the second pixel PX2 and the fourth pixel PX4 can be simultaneously controlled as another sub-pixel group 112. The second pixel PX2 and the fourth pixel PX4 can simultaneously send integrated photocharge to the second rolling shutter circuit 114 via the second line LN2 via the same control signal. The first line LN1 and the second line LN2 can be metallic wiring.

[0081] For example, in rolling shutter mode, each of pixels PX1, PX2, PX3, and PX4 can be controlled independently. The first pixel PX1 and the second pixel PX2 can be simultaneously controlled by different rolling shutter circuits 113 and 114, respectively. The first pixel PX1 and the second pixel PX2 can send integrated photocharge to the first rolling shutter circuit 113 and the second rolling shutter circuit 114 via the first line LN1 and the second line LN2 during the same first time period. Similarly, the third pixel PX3 and the fourth pixel PX4 can be simultaneously controlled by different rolling shutter circuits 113 and 114, respectively. The third pixel PX3 and the fourth pixel PX4 can send integrated photocharge to the first rolling shutter circuit 113 and the second rolling shutter circuit 114 via the first line LN1 and the second line LN2 during the same second time period.

[0082] For example, in the combined mode of the rolling shutter mode, all of pixels PX1, PX2, PX3, and PX4 can simultaneously send the integrated photocharge to the first rolling shutter circuit 113 and the second rolling shutter circuit 114 via the first line LN1 and the second line LN2. The first rolling shutter circuit 113 and the second rolling shutter circuit 114 can output pixel signals through different output lines VOUT1 and VOUT2, respectively, but... Figure 2 The readout circuit 150 can integrate the pixel signals output through different output lines VOUT1 and VOUT2 to output image data.

[0083] For example, in global shutter mode, all of pixels PX1, PX2, PX3, and PX4 can be controlled simultaneously. Pixels PX1, PX2, PX3, and PX4 can simultaneously send integrated photocharge to either the first rolling shutter circuit 113 or the second rolling shutter circuit 114 via the same control signal. Pixel signals based on the photocharge generated in pixels PX1, PX2, PX3, and PX4 can all be sent to the global shutter circuit 116 simultaneously.

[0084] According to reference Figure 2 The pixel group PXG described in one or more embodiments can transmit optical signals to the same rolling shutter circuits 113 and 114 via the same lines LN1 and LN2 by dividing multiple pixels PX1, PX2, PX3, and PX4 into an even number of pixels, and output the signals to multiple output lines VOUT1 and VOUT2. Therefore, the pixel group PXG can simultaneously read out pixel signals of multiple pixels in rolling shutter mode. Furthermore, since the transmission transistors of pixels controlled identically during the same time period can use the same transmission transistor control signal, the number of metal wirings used to transmit the transmission transistor control signal can be reduced. As a result, the process can be further simplified, and the parasitic capacitance caused by the metal wiring can be reduced. In addition, Figure 1The image sensor 100 can drive multiple pixels PX1, PX2, PX3, and PX4 in various operating methods in rolling shutter mode. Therefore, the image sensor 100 can perform various functions of rolling shutter mode.

[0085] Furthermore, since the global shutter circuit 116 is separated from the rolling shutter circuits 113 and 114 via the global selection circuit 115, the global shutter circuit 116 can be more easily operated in global mode.

[0086] Figure 5 It is a circuit diagram of a pixel group according to one or more embodiments. Figure 5 The circuit can correspond to Figure 4 The pixel group PXG. (Refer to...) Figure 5 This describes the circuit construction according to one or more embodiments of a pixel group (PXG). See also... Figure 4 Detailed descriptions of repeated or similar parts will be omitted.

[0087] Reference Figure 5 According to one or more embodiments, the pixel group PXGa may include a plurality of pixels PX1, PX2, PX3 and PX4, a first rolling shutter circuit 113a, a second rolling shutter circuit 114a, a global selection circuit 115a and a global shutter circuit 116a. Figure 5 The pixel group PXGa is shown to include four pixels PX1, PX2, PX3 and PX4, but the invention is not limited thereto, and the pixel group PXGa may include different numbers of pixels.

[0088] Reference Figure 5 The first rolling shutter circuit 113a is described. The second rolling shutter circuit 114a can operate in a similar manner to the first rolling shutter circuit 113a by means of the same operation as the corresponding transistor in the first rolling shutter circuit 113a.

[0089] First pixel PX1 and third pixel PX3 can be connected to first floating diffusion region FD1 via the same first line. First floating diffusion region FD1 can be connected to pixel voltage electrode Vpix via first reset transistor RX1.

[0090] In one or more embodiments, at least one conversion gain control transistor DCG1 and DCG2 may be connected in series between the first reset transistor RX1 and the first floating diffusion region FD1. Each of the conversion gain control transistors DCG1 and DCG2 may be turned on or off by conversion gain control signals CS1 and CS2. When the conversion gain control transistors DCG1 and DCG2 are turned on, the capacitance of the first floating diffusion region FD1 may increase and the conversion gain may decrease. Therefore, the first pixel PX1 and the third pixel PX3 may operate in a low conversion gain (LCG) mode. Conversely, when the conversion gain control transistors DCG1 and DCG2 are turned off, the capacitance of the first floating diffusion region FD1 may increase and the conversion gain may increase. Therefore, the first pixel PX1 and the third pixel PX3 may operate in a high conversion gain (HCG) mode.

[0091] The pixel voltage or reset voltage of the first floating diffusion region FD1 can be converted into a pixel signal for the rolling shutter mode by the first source follower transistor SF1, and output to the first output line VOUT1 through the first node N1 and the first selection transistor SEL1. In the rolling shutter mode, the global selection transistors GSX1 and GSX2 of the global selection circuit 115a can be turned off by the global selection signal GSS.

[0092] The global selection circuit 115a can connect the third node N3 of the global shutter circuit 116a to each of the first node N1 and the second node N2. The first node N1 is the node between the first source follower transistor SF1 and the first selection transistor SEL1 of the first rolling shutter circuit 113a, and the second node N2 is the node between the second source follower transistor SF2 and the second selection transistor SEL2 of the second rolling shutter circuit 114a.

[0093] In global shutter mode, the first global selection transistor GSX1 is turned on, and the pixel signals of the first pixel PX1 and the third pixel PX3 can be stored in one of the first capacitor C1, the second capacitor C2, and the third capacitor C3 via the first node N1 and the third node N3. Similarly, the second global selection transistor GSX2 is turned on, and the pixel signals of the second pixel PX2 and the fourth pixel PX4 can be stored in one of the first capacitor C1, the second capacitor C2, and the third capacitor C3 via the second node N2 and the third node N3. The first global selection transistor GSX1 and the second global selection transistor GSX2 can be turned on or off simultaneously via the global selection signal GSS. Therefore, the pixel signals of pixels PX1, PX2, PX3, and PX4 can be stored in the same capacitor among the first capacitor C1, the second capacitor C2, and the third capacitor C3.

[0094] One terminal of each of the first capacitor C1, the second capacitor C2, and the third capacitor C3 can be connected to the first sampling transistor SMP1, the second sampling transistor SMP2, and the third sampling transistor SMP3, respectively. The other terminal of each of the first capacitor C1, the second capacitor C2, and the third capacitor C3 can be connected to the pixel voltage electrode VPIX. The first sampling transistor SMP1, the second sampling transistor SMP2, and the third sampling transistor SMP3 can be controlled by the first sampling control signal SS1, the second sampling control signal SS2, and the third sampling control signal SS3, respectively.

[0095] In one or more embodiments, the first capacitor C1 can store the reset voltage signals of the first floating diffusion region FD1 and the second floating diffusion region FD2 under the control of the first sampling transistor SMP1. The second capacitor C2 can store the autofocus signals of pixels PX1, PX2, PX3, and PX4 under the control of the second sampling transistor SMP2. For example, each of pixels PX1, PX2, PX3, and PX4 may include a first photodiode on one side and a second photodiode on the other side. The autofocus signal may be a pixel voltage signal based on the photocharge of the first or second photodiode of pixels PX1, PX2, PX3, and PX4. The third capacitor C3 can store the pixel voltage signals based on all the photodiodes of pixels PX1, PX2, PX3, and PX4 under the control of the third sampling transistor SMP3.

[0096] In one or more embodiments, each of the pixel signals stored in the first capacitor C1, the second capacitor C2, and the third capacitor C3 may be output via the third output line VOUT3 at a different readout time.

[0097] Figure 6A and Figure 6B This is an illustration depicting sub-pixels SPX1, SPX2, SPX3, and SPX4 according to one or more embodiments. (Refer to...) Figure 4 and Figure 5 Each of the described pixels PX1, PX2, PX3, and PX4 can be based on Figure 6A or Figure 6B The sub-pixels SPX1, SPX2, SPX3, and SPX4 are configured. Figure 6A and Figure 6B Pixels PX1, PX2, PX3, and PX4 can correspond to Figure 4 and Figure 5 Pixels PX1, PX2, PX3, and PX4. Pixels PX1, PX2, PX3, and PX4 can have similar structures. Therefore, reference will be made to... Figure 6A and Figure 6B The first pixel PX1 is described exemplarily.

[0098] In one or more embodiments, reference is made to Figure 6A and Figure 6B The first pixel PX1 may include multiple sub-pixels SPX1, SPX2, SPX3, and SPX4. Each of the sub-pixels SPX1, SPX2, SPX3, and SPX4 may include a photodiode. Sub-pixels SPX1, SPX2, SPX3, and SPX4 may each include transfer transistors TX11, TX12, TX13, and TX14. Each of the sub-pixels SPX1, SPX2, SPX3, and SPX4 may transfer the photocharge of the photodiode to the floating diffusion region via the transfer transistors TX11, TX12, TX13, and TX14 and lines LN1 and LN2.

[0099] In one or more embodiments, reference is made to Figure 6A Multiple sub-pixels SPX1, SPX2, SPX3, and SPX4 located within the same pixel can share a microlens. For example, multiple sub-pixels SPX1, SPX2, SPX3, and SPX4 of the first pixel PX1 can share a first microlens ML1. Therefore, each of pixels PX1, PX2, PX3, and PX4 can include a separate microlens.

[0100] In one or more embodiments, reference is made to Figure 6B Pixels PX1, PX2, PX3, and PX4 can share a single microlens ML. Therefore, a pixel group may include a single microlens ML.

[0101] In one or more embodiments, a color filter that transmits light of the same spectrum may be provided. Figure 6A and Figure 6B In pixels PX1, PX2, PX3, and PX4.

[0102] Figures 7A to 7F This is a diagram illustrating the operation of the imaging device 10 according to one or more embodiments in a shooting mode. (Refer to...) Figures 7A to 7F The described imaging device 10 may correspond to Figure 1 Imaging device 10. According to Figures 7A to 7F The image sensor 100 and processor 200 of the imaging device 10 in one or more embodiments may correspond to Figure 1 and Figure 2 Image sensor 100 and processor 200.

[0103] In reference Figures 7A to 7FIn one or more of the described embodiments, the first image data IDT1 may be image data output by the image sensor 100 in rolling shutter mode, and the second image data IDT2 may be image data output by the image sensor 100 in global shutter mode. The third image data IDT3 may be image data synthesized by the processor 200 from the first image data IDT1 and the second image data IDT2.

[0104] Reference Figure 7A In one or more embodiments, processor 200 may send a preview command PREV_CMD to image sensor 100. Image sensor 100 may output first image data IDT1 acquired in rolling shutter mode in response to the preview command PREV_CMD received from processor 200. Processor 200 may control a display device to display the first image data IDT1 on the display device.

[0105] For example, the image sensor 100 can operate in preview mode only in rolling shutter mode and output first image data IDT1. The imaging device 10 can display the first image data IDT1 output by the image sensor 100 in rolling shutter mode on a display device, without performing the synthesis of the first image data IDT1 and the second image data IDT2 in preview mode.

[0106] Reference Figures 7B to 7D In one or more embodiments, the processor 200 may send a capture command SHOT_CMD and a mode control command to the image sensor 100. The capture command SHOT_CMD may be a command to acquire a still image or video. The mode control command may be one of a plurality of mode control commands. For example, refer to... Figures 7B to 7D Multiple mode control commands may include the first mode control command IMG_MOD1, the second mode control command IMG_MOD2, and the third mode control command IMG_MOD3.

[0107] In one or more embodiments, the multiple mode control commands may be commands that command the image data output by the image sensor 100, either the first image data IDT1 or the second image data IDT2. The multiple mode control commands may also be commands that command the operating mode of the image sensor 100.

[0108] For example, refer to Figure 7B The first mode control command IMG_MOD1 can be a command from processor 200 to image sensor 100 to output first image data IDT1. The first mode control command IMG_MOD1 can also be a command from processor 200 to image sensor 100 to operate in rolling shutter mode.

[0109] For example, refer to Figure 7C The second mode control command IMG_MOD2 can be a command from processor 200 instructing image sensor 100 to output both first image data IDT1 and second image data IDT2. The second mode control command IMG_MOD2 can also be a command from processor 200 instructing image sensor 100 to operate alternately in rolling shutter mode and global shutter mode.

[0110] Continue to refer to Figure 7C The image sensor 100 can operate alternately in rolling shutter mode and global shutter mode in response to the second mode control command IMG_MOD2, and alternately output at least one frame of first image data IDT1 and at least one frame of second image data IDT2. For example, when the shooting command SHOT_CMD is a command to acquire video, the image sensor 100 can repeat the output of one frame of first image data IDT1 and one frame of second image data IDT2. As another example, the image sensor 100 can repeat the output of one frame of first image data IDT1 and two frames of second image data IDT2. As another example, the image sensor 100 can repeat the output of two frames of first image data IDT1 and one frame of second image data IDT2.

[0111] For example, refer to Figure 7D The third mode control command IMG_MOD3 can be a command from processor 200 instructing image sensor 100 to output second image data IDT2. The third mode control command IMG_MOD3 can also be a command from processor 200 instructing image sensor 100 to operate in global shutter mode. In one or more embodiments, processor 200 can send one of the first mode control command IMG_MOD1, the second mode control command IMG_MOD2, and the third mode control command IMG_MOD3 to image sensor 100 based on the illumination of the shooting environment.

[0112] For example, when the ambient light level is less than a preset first reference light level, the processor 200 can send a first mode control command IMG_MOD1 to the image sensor 100. When the ambient light level is greater than the preset first reference light level but less than a preset second reference light level, the processor 200 can send a second mode control command IMG_MOD2 to the image sensor 100. When the ambient light level is greater than the preset second reference light level, the processor 200 can send a third mode control command IMG_MOD3 to the image sensor 100.

[0113] Reference Figures 7B to 7DIn one or more embodiments, the processor 200 may respond to the first mode control command IMG_MOD1 and the third mode control command IMG_MOD3 without performing image data synthesis. The processor 200 may respond to the second mode control command IMG_MOD2 to synthesize first image data IDT1 and second image data IDT2, and output third image data IDT3 as the result of the synthesis.

[0114] Reference Figure 7E In one or more embodiments, the processor 200 may send a shooting command SHOT_CMD to the image sensor 100 without sending a mode control command. The image sensor 100 may operate alternately in rolling shutter mode and global shutter mode in response to the shooting command SHOT_CMD, and output first image data IDT1 and second image data IDT2.

[0115] For example, refer to Figure 7E The image sensor 100 can always output first image data IDT1 and second image data IDT2 in response to the shooting command SHOT_CMD received from the processor 200. The processor 200 can determine whether to synthesize the first image data IDT1 and the second image data IDT2 based on the shooting environment or preset conditions.

[0116] For example, based on the illumination of the shooting environment, the processor can determine to synthesize first image data IDT1 and second image data IDT2, and determine the first image data IDT1 as the final image data, or determine the second image data IDT2 as the final image data.

[0117] As another example, the processor 200 may perform scene analysis on the first image data IDT1 or the second image data IDT2, and determine the first image data IDT1 or the second image data IDT2 as the final image data based on the type of the identified scene, or determine to synthesize the first image data IDT1 and the second image data IDT2.

[0118] In one or more embodiments, reference is made to Figure 7F The processor 200 can send a shooting command SHOT_CMD without sending a mode control command to the image sensor 100. In this case, the image sensor 100 can determine whether to output at least one of the first image data IDT1 and the second image data IDT2 based on the shooting environment or preset conditions.

[0119] For example, the image sensor 100 may include an illuminance sensor, or determine whether to output at least one of the first image data IDT1 and the second image data IDT2 based on illuminance information received from the processor 200.

[0120] For example, image sensor 100 may operate in rolling shutter mode to generate first image data IDT1, perform scene analysis on the first image data IDT1, and determine, based on the type of the identified scene, whether to additionally output second image data IDT2 by operating in global shutter mode. In this case, image sensor 100 may also include logic circuitry for performing scene analysis based on a neural network. Furthermore, image sensor 100 may optionally include memory circuitry and / or digital signal processing logic circuitry for performing the operations of the logic circuitry. In one or more embodiments, when image sensor 100 performs scene analysis on the first image data IDT1, image sensor 100 may output the results of the scene analysis separately from the image data.

[0121] Therefore, return to the reference. Figure 7F Image sensor 100 can operate in rolling shutter mode in response to the shooting command SHOT_CMD and output first image data IDT1. As another example, image sensor 100 can operate alternately in rolling shutter mode and global shutter mode in response to the shooting command SHOT_CMD and output first image data IDT1 and second image data IDT2. As yet another example, image sensor 100 can operate in global shutter mode in response to the shooting command SHOT_CMD and output second image data IDT2.

[0122] Figures 8A to 8C This is a diagram illustrating the image data used for synthesis when an imaging device according to one or more embodiments performs image data synthesis.

[0123] exist Figures 8A to 8C In one or more embodiments, the image sensor 100 may operate alternately in rolling shutter mode and global shutter mode in response to the mode control command IMG_MOD_SYN. Figures 8A to 8C Image data output by image sensor 100 according to one or more embodiments is shown. The output image data may be sent directly or indirectly to processor 200, and processor 200 may use the image data output by image sensor 100 for synthesis.

[0124] Reference Figures 8A to 8C In one or more embodiments, processor 200 may send a capture command SHOT_CMD and a mode control command IMG_MOD_SYN to image sensor 100. The description will refer to the assumption that the capture command SHOT_CMD is a command for acquiring still images. Figures 8A to 8C One or more embodiments are described. However, even when the shooting command SHOT_CMD is a command for acquiring video, the image sensor 100 can be repeatedly executed. Figures 8A to 8CIt is operated in the manner described in the embodiments.

[0125] The mode control command IMG_MOD_SYN can be a command from processor 200 instructing image sensor 100 to output both first image data IDT1 and second image data IDT2. The mode control command IMG_MOD_SYN can also be a command from processor 200 instructing image sensor 100 to operate alternately in rolling shutter mode and global shutter mode.

[0126] In one or more embodiments, the mode control command IMG_MOD_SYN may correspond to Figure 7C The second mode control command is IMG_MOD2, but it is not limited to this. That is, the processor 200 can send the mode control command IMG_MOD_SYN to the image sensor 100 based on the illumination of the shooting environment. As another example, the processor can send the mode control command IMG_MOD_SYN to the image sensor 100 based on other preset conditions.

[0127] Reference Figure 8A In one or more embodiments, the image sensor 100 may operate alternately in rolling shutter mode and global shutter mode in response to the mode control command IMG_MOD_SYN, and alternately output one frame of first image data IDT1 (DAT1) and one frame of second image data IDT2 (DAT2). For example, the image sensor 100 may operate in rolling shutter mode to output one frame of first image data IDT1 (DAT1), and then operate again in global shutter mode to output one frame of second image data IDT2 (DAT2). The order of outputting first image data IDT1 and second image data IDT2 is not particularly limited. The processor 200 may synthesize one frame of first image data IDT1 (DAT1) and one frame of second image data IDT2 (DAT2) and output third image data.

[0128] Reference Figure 8BIn one or more embodiments, the image sensor 100 may operate alternately in rolling shutter mode and global shutter mode in response to the mode control command IMG_MOD_SYN, and output one frame of first image data IDT1 (DAT2) and two frames of second image data IDT2 (DAT1 and DAT3). For example, the image sensor 100 may operate in global shutter mode and output one frame of second image data IDT2 (DAT1). Subsequently, the image sensor 100 may operate in rolling shutter mode and output one frame of first image data IDT1 (DAT2). The image sensor 100 may again operate in global shutter mode and output one frame of second image data IDT2 (DAT3). The processor 200 may synthesize one frame of first image data IDT1 (DAT2) and two frames of second image data IDT2 (DAT1 and DAT3) and output third image data.

[0129] Reference Figure 8C In one or more embodiments, the image sensor 100 may operate alternately in rolling shutter mode and global shutter mode in response to the mode control command IMG_MOD_SYN, and output two frames of first image data IDT1 (DAT1 and DAT3) and one frame of second image data IDT2 (DAT2). For example, the image sensor 100 may operate in rolling shutter mode and output one frame of first image data IDT1 (DAT1). Subsequently, the image sensor 100 may operate in global shutter mode and output one frame of second image data IDT2 (DAT2). The image sensor 100 may again operate in rolling shutter mode and output one frame of first image data IDT1 (DAT3). The processor 200 may synthesize the two frames of first image data IDT1 (DAT1 and DAT3) and one frame of second image data IDT2 (DAT2), and output third image data.

[0130] The number of image data used by processor 200 to synthesize image data is not limited to Figures 8A to 8C One or more embodiments. For example, processor 200 can... Figures 8A to 8C The first image data IDT1 of the embodiment has more frames and / or more frames than the first image data IDT1. Figures 8A to 8C In this embodiment, frames of the second image data IDT2 with more frames are used for synthesis.

[0131] Figures 9A to 9E It is a description based on Figure 8A Implementation examples Figure 1 A diagram illustrating the operation of the imaging device 10 in synthesizing global shutter image data and rolling shutter image data. (Refer to...) Figure 8A as well as Figures 9A to 9EA method for combining a frame of global shutter image data and a frame of rolling shutter image data by an imaging device 10. Figures 9A to 9E The method can be derived from Figure 1 The imaging device 10 is executed by the processor 200.

[0132] Reference Figure 9A In one or more embodiments, Figure 8A The processor 200 can process one frame of first image data IDT1 (DAT1) and one frame of second image data IDT2 (DAT2) to output third image data IDT3 (DAT3). The processing can be represented as a synthesis performed by the processor 200.

[0133] Image data may include multiple pixel data. Processor 200 may display the image data on a display device or perform image data processing based on the pixel position and pixel value of each of the multiple pixel data.

[0134] In one or more embodiments, the processor 200 can generate third image data IDT3 based on the pixel values ​​of at least one first pixel data of first image data IDT1 and the pixel positions of at least one second pixel data of second image data IDT2. The processor 200 can generate third image data IDT3 using the pixel values ​​of first image data IDT1 acquired in rolling shutter mode and the pixel positions of second image data IDT2 acquired in global shutter mode. Therefore, an image with reduced rolling shutter effect and noise can be generated.

[0135] For example, Figure 9B The diagram shows a first object OB1 composed of multiple pixel blocks of first image data IDT1 (DAT1), a second object OB2 composed of multiple pixel blocks of second image data IDT2 (DAT2), and a third object OB3 composed of multiple pixel blocks of third image data IDT3 (DAT3). Each pixel block may include at least one pixel data.

[0136] Reference Figure 9B ,Depend on Figure 8A The pixel value of each pixel block of the third object OB3 in the third image data IDT3 (DAT3) output by the processor 200 can correspond to the pixel value of each pixel block of the first object OB1 in the first image data IDT1 (DAT1). Furthermore, the pixel position of each pixel block of the third object OB3 in the third image data IDT3 (DAT3) can correspond to the pixel position of each pixel block of the second object OB2 in the second image data IDT2 (DAT2).

[0137] For example, the pixel data of the first pixel block GRB1 constituting the third object OB3 can be generated based on the pixel values ​​of the pixel data of the first pixel block RB1 constituting the first object OB1 and the pixel positions of the pixel data of the first pixel block GB1 constituting the second object OB2. The pixel data of the second pixel block GRB2 constituting the third object OB3 can be generated based on the pixel values ​​of the pixel data of the second pixel block RB2 constituting the first object OB1 and the pixel positions of the pixel data of the second pixel block GB2 constituting the second object OB2.

[0138] In this disclosure, generating a pixel block can be achieved by generating pixel data that constitutes the pixel block. Determining the pixel position of a pixel block can be achieved by determining the pixel position of the pixel data that constitutes the pixel block, and determining the pixel value of a pixel block can be achieved by determining the pixel value of the pixel data that constitutes the pixel block.

[0139] Reference Figure 9C , Figure 8A The processor 200 can perform block matching between a frame of the first image data IDT1 (DAT1) and a frame of the second image data IDT2 (DAT2), and determine correspondences based on the block matching. For example, the processor 200 can determine the motion vector field MF based on the block matching. The processor 200 can perform block matching and determine the motion vector field MF using various conventional methods.

[0140] In one or more embodiments, processor 200 may perform block matching and determine each pixel block of second image data IDT2 (DAT2) corresponding to each pixel block of first image data IDT1 (DAT1). Processor 200 may determine a motion vector field MF based on changes in the position of the corresponding pixel blocks in the first image data IDT1 (DAT1) and the second image data IDT2 (DAT2). The motion vector field MF may include multiple motion vectors MV.

[0141] For example, the first pixel block RB1 of the first image data IDT1 (DAT1), which is rolling shutter image data, can be matched with the first pixel block GB1 of the second image data IDT2 (DAT2), which is global shutter image data. The first pixel block RB1 of the first image data IDT1 (DAT1) and the first pixel block GB1 of the second image data IDT2 (DAT2) can have a relationship with a first motion vector MV1. Similarly, the second pixel block RB2 of the first image data IDT1 (DAT1) and the second pixel block GB2 of the second image data IDT2 (DAT2) can have a relationship with a second motion vector MV2.

[0142] Reference Figure 9D In one or more embodiments, Figure 8AThe processor 200 can synthesize first image data IDT1 (DAT1) based on second image data IDT2 (DAT2) and generate third image data IDT3 (DAT3).

[0143] For example, the processor 200 can determine the pixel positions of the first pixel block GRB1 and the second pixel block GRB2 of the third image data IDT3 (DAT3) based on the pixel positions of the first pixel block GB1 and the second pixel block GB2 of the second image data IDT2 (DAT2).

[0144] For example, the processor 200 can determine the pixel values ​​of the first pixel block GRB1 and the second pixel block GRB2 of the third image data IDT3 (DAT3) based on the pixel values ​​of the first pixel block RB1 and the second pixel block RB2 of the first image data IDT1 (DAT1). As described above, the first pixel block RB1 and the second pixel block RB2 of the first image data IDT1 (DAT1) can have a relationship with the first pixel block GB1 and the second pixel block GB2 of the second image data IDT2 (DAT2) having a first motion vector MV1 and a second motion vector MV2, respectively.

[0145] According to one or more embodiments, when the processor 200 determines the pixel values ​​of a pixel block of third image data IDT3 (DAT3), the processor 200 may use various image processing techniques (such as interpolation).

[0146] Reference Figure 9E In one or more embodiments, Figure 8A The processor 200 can synthesize a second image data IDT2 (DAT2) based on the first image data IDT1 (DAT1), and generate a third image data IDT3 (DAT3).

[0147] For example, processor 200 can generate at least one transformation matrix based on a motion vector field MF. Processor 200 can apply the transformation matrix to the pixel position of each pixel block in the first image data IDT1 (DAT1) to determine the pixel position of each pixel block in the third image data IDT3 (DAT3). For example, processor 200 can apply a transformation matrix based on a first motion vector MV1 and a transformation matrix based on a second motion vector MV2 to the pixel positions of the first pixel block RB1 and the second pixel block RB2 of the first image data IDT1 (DAT1), respectively, and determine the pixel positions of the first pixel block RGB1 and the second pixel block RGB2 of the third image data IDT3 (DAT3), respectively. In one embodiment, the third image data IDT3 (DAT3) can be generated based on the pixel positions of the first pixel block RGB1 and the second pixel block RGB2 of the third image data IDT3 (DAT3).

[0148] For example, based on the pixel values ​​of the first pixel block RB1 and the second pixel block RB2 of the first image data IDT1 (DAT1), the processor 200 can determine the pixel values ​​of the first pixel block RGB1 and the second pixel block RGB2 of the third image data IDT3 (DAT3).

[0149] As described above, the first pixel block RB1 and the second pixel block RB2 of the first image data IDT1 (DAT1) can respectively have a first motion vector MV1 and a second motion vector MV2 relationship with the first pixel block GB1 and the second pixel block GB2 of the second image data IDT2 (DAT2).

[0150] Figure 10 It is a description based on Figure 8B An illustration of the operation of the imaging device in an embodiment to synthesize global shutter image data and rolling shutter image data.

[0151] Figure 10 It is a description based on Figure 8B One or more embodiments Figure 1 A diagram illustrating the operation of the imaging device 10 in synthesizing global shutter image data and rolling shutter image data. (Refer to...) Figure 8B and Figure 10 A method for combining two frames of global shutter image data and one frame of rolling shutter image data by an imaging device 10. Figure 10 The method can be derived from Figure 1 The imaging device 10 is executed by the processor 200.

[0152] Reference Figure 10 In one or more embodiments, Figure 8B The processor 200 can (e.g., based on a first motion vector field MF1 and a second motion vector field MF2) process one frame of first image data IDT1 (DAT2) and two frames of second image data IDT2 (DAT1 and DAT3) to output third image data IDT3 (DAT4). The processing can be represented as synthesis performed by the processor 200.

[0153] For example, the two frames of the second image data IDT2 (DAT1 and DAT3) could be image data acquired before and after the first image data IDT1 (DAT2) was captured, respectively. Figure 8B The image sensor 100 can output one frame of the second image data IDT2 (DAT1) before outputting the first image data IDT1 (DAT2), and output other frames of the second image data IDT2 (DAT3) after outputting the first image data IDT1 (DAT2).

[0154] Image data may include multiple pixel data. Processor 200 may display the image data on a display device or perform image data processing based on the pixel position and pixel value of each of the multiple pixel data.

[0155] In one or more embodiments, reference is made to Figure 10 The processor 200 can process some pixel data PDT1 of the first image data IDT1 (DAT2) and one frame of the second image data IDT2 (DAT1) to generate at least a portion (e.g., pixel data PDT3) of the third image data IDT3 (DAT4). Furthermore, the processor 200 can process additional pixel data PDT2 of the first image data IDT1 (DAT2) and other frames of the second image data IDT2 (DAT3) to generate at least the remaining portion (e.g., pixel data PDT4) of the third image data IDT3 (DAT4). The method for generating the pixel data of the third image data IDT3 (DAT4) can use... Figures 9A to 9E The method.

[0156] Reference Figure 10 One or more embodiments of the invention can improve the results of image synthesis under specific shooting conditions by using both second image data IDT2 (DAT1 and DAT3) acquired before and after the shooting time of the first image data IDT1 (DAT2). For example, the rolling shutter effect can be further reduced under specific shooting conditions.

[0157] Figure 11A and Figure 11B It is a description based on Figure 8C An illustration of the operation of the imaging device in an embodiment to synthesize global shutter image data and rolling shutter image data.

[0158] Figure 11A and Figure 11B It is a description based on Figure 8C One or more embodiments Figure 1 A diagram illustrating the operation of the imaging device 10 in synthesizing global shutter image data and rolling shutter image data. (Refer to...) Figure 8C , Figure 11A and Figure 11B This describes a method for an imaging device 10 to synthesize one frame of global shutter image data and two frames of rolling shutter image data. Figure 11A and Figure 11B The method can be derived from Figure 1 The imaging device 10 is executed by the processor 200.

[0159] Reference Figure 11A In one or more embodiments, Figure 8CThe processor 200 can (e.g., based on a motion vector field MF) process two frames of first image data IDT1 (DAT1 and DAT3) and one frame of second image data IDT2 (DAT2) to output third image data IDT3 (DAT5). The processing can be represented as a synthesis performed by the processor 200.

[0160] For example, refer to Figure 8C The two frames of the first image data IDT1 (DAT1 and DAT3) can be image data acquired before and after the acquisition of the second image data IDT2 (DAT2). That is, Figure 8C The image sensor 100 can output one frame of the first image data IDT1 (DAT1) before outputting the second image data IDT2 (DAT2), and output other frames of the first image data IDT1 (DAT3) after outputting the second image data IDT2 (DAT2).

[0161] Refer again Figure 11A , Figure 8C The processor 200 can perform a first processing step, Processing 1, on two frames of the first image data IDT1 (DAT1 and DAT3) to generate a fourth image data IDT4 (DAT4), and perform a second processing step, Processing 2, on the second image data IDT2 (DAT2) and the fourth image data IDT4 (DAT4) to generate a third image data IDT3 (DAT5). The processor 200 can output the third image data IDT3 (DAT5).

[0162] In one or more embodiments, the first process (Processing 1) and the second process (Processing 2) may be different processes. The second process (Processing 2) may be similar to... Figures 9A to 9E The processing of one or more embodiments.

[0163] Figure 11B It is a description Figure 11A A diagram illustrating one or more embodiments of the first processing, Processing 1.

[0164] Figure 11B The diagram illustrates a first object OB1 composed of multiple pixel blocks of first image data IDT1 (DAT1) of a frame acquired at time T1, and a second object OB2 composed of multiple pixel blocks of first image data IDT1 (DAT3) of a frame acquired at time T2. Each pixel block may include at least one pixel data. Time T1 is earlier than time T2.

[0165] In one or more embodiments, processor 200 may perform block matching between two frames of first image data IDT1 (DAT1 and DAT3) and determine correspondences based on the block matching. For example, processor 200 may determine the motion vector field MF between two frames of first image data IDT1 (DAT1 and DAT3) based on block matching. Processor 200 may use various conventional methods to perform block matching and determine the motion vector field MF.

[0166] Reference Figure 11B The motion vectors MV1', MV2', MV3', and MV4' exemplarily shown between some pixel blocks PB1, PB2, PB3, and PB4 of the first object OB1 and some pixel blocks LB1, LB2, LB3, and LB4 of the second object OB2 can be the same motion vectors MV1, MV2, MV3, and MV4 as some motion vectors MV1, MV2, MV3, and MV4 of the motion vector field MF between two frames of the first image data IDT1 (DAT1 and DAT3). However, for the purpose of describing the first process Processing 1, the motion vectors MV1', MV2', MV3', and MV4' exemplarily shown are conceptually similar to... Figure 11B The motion vectors MV1, MV2, MV3 and MV4 are shown differently.

[0167] The first motion vector MV1' is a motion vector that describes the correspondence between the first pixel block PB1 of the first object OB1 and the first pixel block LB1 of the second object OB2. The first motion vector MV1' can be described as the position of the first pixel block PB1 of the first object OB1 acquired at time T1 moving to the position of the first pixel block LB1 of the second object OB2 at time T2.

[0168] Similarly, the second motion vector MV2', the third motion vector MV3', and the fourth motion vector MV4' can be described as the second pixel block PB2, the third pixel block PB3, and the fourth pixel block PB4 of the first object OB1 moving to the positions of the second pixel block LB2, the third pixel block LB3, and the fourth pixel block LB4 of the second object OB2, respectively.

[0169] In one or more embodiments, the processor 200 can determine the position of the first pixel block PB1 of the first object OB1 at time Ta as the position of the first pixel block of the fourth image data IDT4 (DAT4). Similarly, the processor 200 can determine the positions of the second pixel block PB2, the third pixel block PB3, and the fourth pixel block PB4 of the first object OB1 at time Ta as the positions of the second pixel block, the third pixel block, and the fourth pixel block of the fourth image data IDT4 (DAT4), respectively.

[0170] Reference Figure 11BThe method described herein is for processor 200 to calculate (obtain) the positions of the first pixel block PB1, the second pixel block PB2, the third pixel block PB3, and the fourth pixel block PB4 of the first object OB1 at time Ta.

[0171] exist Figure 11B In the first image data IDT1 (DAT1 and DAT3), two of the two frames are image data acquired in rolling shutter mode. Therefore, both the first object OB1 and the second object OB2 may exhibit a rolling shutter effect. For example, as shown in the reference... Figure 3B The pixel blocks of the first object OB1 and the second object OB2 may include pixel data generated based on photocharge integrated at different times in the column direction.

[0172] For example, as referenced Figure 3B The second pixel block LB2 of the second object OB2 can be acquired at a first time later than the time Tb (the time Tb when the photocharge is integrated) when the first pixel block LB1 is acquired. Similarly, the third pixel block LB3 and the fourth pixel block LB4 of the second object OB2 can be acquired at a second time and a third time later than the time Tb when the first pixel block LB1 is acquired, respectively.

[0173] In one or more embodiments, the processor 200 may determine that the position of the first pixel block PB1 of the first object OB1, which is moved along the first motion vector MV1' from the position of the first pixel block PB1 to the position of the first pixel block PB1 of the first object OB1 at time Ta.

[0174] As another example, in one or more embodiments, processor 200 may determine that the position of the first pixel block PB1 of the first object OB2, moved along the first motion vector MV1' for time Tb-Ta=D1, is the position of the first pixel block PB1 of the first object OB1 at time Ta. Similarly, processor 200 may calculate (obtain) the position of the second pixel block LB2 of the second object OB2, moved along the second motion vector MV2' for time D2, the position of the third pixel block LB3, moved along the third motion vector MV3' for time D3, and the position of the fourth pixel block LB4, moved along the fourth motion vector MV4' for time D4.

[0175] Time D2, D3, and D4 can be determined based on the difference between time D1 and the capture time of each row of the first image data IDT1 (DAT3) of the frame acquired at time T2.

[0176] Figure 11BEach pixel block may include at least one pixel data. When each pixel block includes multiple pixel data, each of the multiple pixel data in the same pixel block may have a different capture time for each row. However, in one or more embodiments, the first process may assume that the pixel data in the same pixel block has the same rolling shutter effect, and the processor 200 may perform the first process. For example, the processor 200 may perform the first process assuming that the pixel data in the same pixel block has been acquired simultaneously.

[0177] therefore, Figure 11B The embodiment can reduce the rolling shutter effect by using two frames of the first image data IDT1 (DAT1 and DAT3).

[0178] Subsequently, the processor 200 can process the fourth image data IDT4 (DAT4) with reduced rolling shutter effect and the data acquired in global shutter mode. Figure 11A The second image data IDT2 (DAT2) is used to perform a second process. For example, the processor 200 may be based on... Figure 11A The pixel values ​​of the pixel data in the fourth image data IDT4 (DAT4) and the pixel positions of the pixel data in the second image data IDT2 (DAT2) are used to generate Figure 11A The third image data is IDT3 (DAT5). Therefore, the processor 200 can generate image frames with further reduced rolling shutter effect.

[0179] Figure 12 This is a block diagram describing an image sensor according to one or more embodiments.

[0180] Figure 12 This is a block diagram of an image sensor 100a according to one or more embodiments. Detailed descriptions of parts that are repeated above will be omitted. Figure 12 The pixel group PXGa can correspond to Figure 5 The pixel group PXGa.

[0181] Image sensor 100a may include a stacked first substrate 10a and a second substrate 20a. The first substrate 10a and the second substrate 20a may be connected to each other via a wafer bonding process using pixel group-level Cu-Cu (C2C) interconnects. The first substrate 10a and the second substrate 20a may be electrically connected not only via in-pixel contacts IN_CT within the pixel group PXGa, but also via a C2C array located in the peripheral region of the substrates. Control signals for controlling pixel circuitry may be transmitted via the C2C array. Pixel signals from the first substrate 10a may be transmitted to the readout circuitry of the second substrate 20a via the in-pixel contacts IN_CT. For example, the second substrate 20a may include readout circuitry, logic circuitry, and interface circuitry.

[0182] In one or more embodiments, some pixel circuits may be located on the first substrate 10a, and other pixel circuits may be located on the second substrate 20a. For example, Figure 5 The first rolling shutter circuit 113a and the second rolling shutter circuit 114a may be located on the first substrate 10a, and the global selection circuit 115a and the global shutter circuit 116a may be located on the second substrate 20a.

[0183] In one or more embodiments, all pixel circuitry may be located on the second substrate 20a. For example, including Figure 5 Multiple pixels PX1, PX2, PX3 and PX4 of the photodiode may be located on the first substrate 10a, and the first rolling shutter circuit 113a, the second rolling shutter circuit 114a, the global selection circuit 115a and the global shutter circuit 116a may be located on the second substrate 20a.

[0184] Figure 13 This is a block diagram describing an image sensor according to one or more embodiments.

[0185] Figure 13 This is a block diagram of an image sensor 100b according to one or more embodiments. Detailed descriptions of parts that are repeated above will be omitted.

[0186] Reference Figure 13 The image sensor 100b may include a first substrate 10b, a second substrate 20b, and a third substrate 30b. The third substrate 30b, the second substrate 20b, and the first substrate 10b may be stacked sequentially in a direction d3 perpendicular to the plane of the substrate (the surface parallel to d1 and d2).

[0187] In one or more embodiments, Figure 5 Some of the circuits PXGa_1, PXGa_2, and PXGa_3 of the pixel group PXGa may be formed on each of the first substrate 10b and the second substrate 20b. The first portion of the pixel's circuitry PXGa_1 may be located on the first substrate 10b, and the remaining portions of the pixel's circuitry PXGa_2 and PXGa_3 may be located on the second substrate 20b. The third substrate 30b may include logic circuitry (such as readout circuitry, timing controllers, or image signal processors) and interface circuitry. The readout circuitry may include an analog-to-digital converter (ADC). In one or more embodiments, as referenced... Figure 7E The logic circuit for performing scene analysis based on the neural network of the image sensor 100 described herein may be located on the third substrate 30b.

[0188] For example, Figure 5The first rolling shutter circuit 113a and the second rolling shutter circuit 114a may be located on the first substrate 10b, and the global selection circuit 115a and the global shutter circuit 116a may be located on the second substrate 20b.

[0189] The form of the circuit for configuring the pixel group PXGa on the first substrate 10b and the second substrate 20b is not limited to this.

[0190] The first substrate 10b and the second substrate 20b can be electrically connected to each other.

[0191] In one or more embodiments, the first substrate 10b and the second substrate 20b can transmit pixel signals or control signals through through-silicon vias (TSVs) located in the peripheral regions of the first substrate 10b and the second substrate 20b.

[0192] In one or more embodiments, the first portion circuit PXGa_1 of the pixel on the first substrate 10b and the second portion circuit PXGa_2 of the pixel on the second substrate 20b can also be electrically connected via a first inter-substrate connection structure INTC_1. The inter-substrate connection structure INTC_1 can be a C2C bonded contact or a deep contact structure. A deep contact structure may include a TSV. The inter-substrate connection structure INTC_1 can electrically connect an in-pixel contact IN_CT1 electrically connected to elements of the first portion circuit PXGa_1 of the pixel to an in-pixel contact IN_CT2 electrically connected to elements of the second portion circuit PXGa_2 of the pixel.

[0193] In one or more embodiments, the first substrate 10b and / or the second substrate 20b can be electrically connected to the third substrate 30b via a TSV and / or the inter-substrate connection structure INTC_2. Signals from the first substrate 10b and / or the second substrate 20b can be transmitted to the readout circuit (or image signal processor) of the third substrate 30b via the TSV and / or the inter-substrate connection structure INTC_2.

[0194] In one or more embodiments, the second portion of the circuitry PXGa_2 of the pixel group PXGa can be electrically connected to the circuitry of the third substrate 30b via C2C bonding contacts. The second substrate connection structure INTC_2 may include C2C bonding contacts.

[0195] In one or more embodiments, the third part of the pixel group PXGa circuit PXGa_3 can be electrically connected to the circuit of the third substrate 30b via through-silicon copper (TSC).

[0196] Figure 14 It is a block diagram describing an imaging apparatus according to one or more embodiments.

[0197] Figure 14This is a block diagram of an electronic device according to one or more embodiments. Detailed descriptions of portions that are repeated above will be omitted.

[0198] The electronic device 1000 may include an imaging unit 1100, an image sensor 1200, a processor 1300, a display device 1400, and a storage device 1500.

[0199] The processor 1300 controls the overall operation of the electronic device 1000. The processor 1300 can control the position of the lens 1110 by providing control signals to the actuator 1120. Therefore, the focal length can be controlled.

[0200] Imaging unit 1100 is a light-receiving component and may include lens 1110 and actuator 1120. Lens 1110 may include multiple lenses.

[0201] The actuator 1120 can move the lens 1110 in the direction of increasing distance from the object S or in the direction of decreasing distance from the object S according to the control signal of the processor 1300.

[0202] Image sensor 1200 can generate image data and phase data based on incident light. Image sensor 1200 may include pixel array 1210, timing controller 1220, readout circuitry 1230 and image signal processor (ISP) 1240.

[0203] The pixels of the pixel array 1210 may include at least one photoelectric conversion element.

[0204] The pixels of the pixel array 1210 according to one or more embodiments can operate in either a rolling shutter mode or a global shutter mode. The image signal processor 1240 can generate a mode control signal MC based on a shooting mode control signal IMG_MOD sent by the processor 1300. The pixels can operate in either a rolling shutter mode or a global shutter mode based on the mode control signal MC sent by the image signal processor 1240.

[0205] Each pixel in pixel array 1210 can be a reference. Figure 4 The pixel group PXGa is described. Each pixel in the pixel array 1210 may include references. Figure 5 The circuit described for pixel group PXGa.

[0206] The image signal processor 1240 can provide a mode control signal MC to the timing controller 1220. The timing controller 1220 can control the operation of the pixel array 1210 based on the mode control signal MC.

[0207] The image sensor 1200 can operate alternately in rolling shutter mode and global shutter mode to output at least one frame of rolling shutter image data and at least one frame of global shutter image data.

[0208] For reference Figures 9A to 11B The processor 1300 can synthesize at least one frame of rolling shutter image data and at least one frame of global shutter image data.

[0209] Figure 15 This is a flowchart describing an operation method of an imaging device according to one or more embodiments. Detailed descriptions of portions that overlap with the foregoing will be omitted. Figure 1 The imaging device 10 performs Figure 15 The operation method of the image sensor.

[0210] During operation S110, the imaging device 10 may receive user input commanding the acquisition of images or videos. In one or more embodiments, the imaging device 10 may be an electronic device equipped with an image sensor (such as, for example, a mobile phone, laptop computer, security camera, surveillance camera, tablet computer, or smartphone). The imaging device 10 may receive user input from the user for acquiring still images or videos through a user interface without particular limitations (such as a graphical user interface or a capture button).

[0211] When operating S120, Figure 1 The processor 200 of the imaging device 10 can send a first command to the image sensor in response to user input, commanding the acquisition of an image or video. In one or more embodiments, the first command may be a reference... Figures 7B to 7E The described shooting command is SHOT_CMD. The shooting command SHOT_CMD can be a command to acquire still images or videos. In one or more embodiments, the processor 200 may send a mode control command along with a first command. The mode control command can be one of a plurality of mode control commands. For example, the mode control command can be a reference... Figures 7B to 7E The described mode control command is one of IMG_MOD1, IMG_MOD2, and IMG_MOD3.

[0212] In operation S130 Figure 1 The image sensor 100 can generate first image data in rolling shutter mode in response to a first command, and can output the first image data. In operation S140, Figure 1The image sensor 100 can generate second image data in global shutter mode in response to a first command, and can output the second image data. Operations S130 and S140 can be executed in any order. For example, the image sensor 100 can execute operation S140 after executing operation S130, or execute operation S130 after executing operation S140.

[0213] Can be based on reference Figures 7B to 7E One or more embodiments are described to perform operations S130 and S140.

[0214] In operating S150, Figure 1 The processor 200 can generate third image data based on the first image data and the second image data. For example, the processor 200 can generate third image data based on a reference. Figures 9A to 11B One or more embodiments described synthesize at least one frame of rolling shutter image data and at least one frame of global shutter image data, and generate third image data.

[0215] An imaging apparatus and an operating method thereof according to one or more embodiments can generate images with reduced rolling shutter effect and noise. Therefore, the image quality can be improved.

[0216] Although embodiments have been described with reference to the accompanying drawings, those skilled in the art will understand that various changes in form and detail may be made therein without departing from the spirit and scope defined by the appended claims and their equivalents.

Claims

1. An imaging device, comprising: An image sensor, including a pixel array comprising a matrix of pixels, is configured to output first image data acquired in a rolling shutter mode and second image data acquired in a global shutter mode. as well as At least one processor is configured to process first image data and second image data, and output third image data based on the processing.

2. The imaging device according to claim 1, wherein, The image sensor is also configured to output at least one of first image data and second image data based on the shooting command of the at least one processor.

3. The imaging device according to claim 1, wherein, The image sensor is configured to alternately output first image data and second image data based on the shooting command of the at least one processor.

4. The imaging device according to claim 1, wherein, Each of the first and second image data includes information indicating a shooting mode, either a rolling shutter mode or a global shutter mode.

5. The imaging device according to claim 1, wherein, The at least one processor is further configured to generate third image data based on the pixel value of the first pixel data of the first image data and the pixel position of the second pixel data of the second image data.

6. The imaging device according to claim 1, wherein, Based on the first position of the first pixel data of the first image data and the second position of the second pixel data of the second image data, the at least one processor is further configured to: Determine the relationship between the first position and the second position; The third position is obtained by transforming the first pixel data of the first image data based on the relationship to obtain the first position; and Third image data is generated based on the third location.

7. The imaging device according to claim 1, wherein, The at least one processor is further configured to: Process two frames of the first image data and one frame of the second image data; and The third image data is output based on the processing of the two frames of the first image data and the one frame of the second image data.

8. The imaging device according to claim 7, wherein, The at least one processor is further configured to: The fourth image data is generated based on the motion vector field between the two frames of the first image data; and The pixel data of the third image data is generated based on the pixel values ​​of the pixel data of the fourth image data and the pixel positions of the pixel data of the second image data.

9. The imaging device according to claim 1, wherein, The image sensor is also configured to output a fourth image data in global shutter mode, and The at least one processor is further configured to process first image data, second image data, and fourth image data, and output third image data based on the processing of the first image data, second image data, and fourth image data.

10. The imaging device according to claim 9, wherein, The at least one processor is further configured to: Process the first image data and the second image data to generate at least a first portion of the third image data, and The first image data and the fourth image data are processed to generate a second part of the third image data, in addition to the first part of the third image data.

11. The imaging device according to claim 9, wherein, The image sensor is also configured to output second image data before outputting first image data, and to output fourth image data after outputting first image data.

12. The imaging apparatus according to any one of claims 1 to 11, wherein, The at least one processor is also configured to send one of a first mode control command and a second mode control command to the image sensor based on illumination. The image sensor is configured to output first image data and second image data based on a first mode control command, and to output first image data based on a second mode control command. The at least one processor is further configured to generate a second mode control command at an illuminance lower than the illuminance at which the first mode control command was generated.

13. The imaging apparatus according to any one of claims 1 to 11, wherein, The image sensor is configured as follows: Based on the preview command received from the at least one processor, the first image data acquired in rolling shutter mode is output, and Based on the shooting command received from the at least one processor, the system outputs first image data acquired in rolling shutter mode and second image data acquired in global shutter mode.

14. An imaging device, comprising: An image sensor includes a pixel array comprising a matrix of pixels, the image sensor being configured to generate first image data in a rolling shutter mode and second image data in a global shutter mode based on a shooting command received from at least one processor, and to output the first image data and the second image data. A memory device is configured to store first image data, second image data, and at least one command executed by the at least one processor, the memory device being electrically connected to the at least one processor; as well as The at least one processor is configured to send a shooting command to an image sensor based on user input of an image or video based on a command, and to generate third image data by changing the pixel position of the pixel data of the first image data based on the second image data or by changing the pixel value of the pixel data of the second image data based on the first image data.

15. The imaging device according to claim 14, wherein, The at least one processor is further configured to determine the pixel position of the pixel data of the third image data based on the pixel position of the pixel data of the second image data.

16. The imaging device according to claim 14, wherein, The at least one processor is further configured to determine the pixel value of the pixel data of the third image data based on the pixel value of the pixel data of the first image data.

17. The imaging device according to claim 14, wherein, The at least one processor is further configured to: determine a motion vector field between the first image data and the second image data, and determine the pixel position of the pixel data of the third image data based on the motion vector field.

18. The imaging device according to claim 14, wherein, The at least one processor is further configured to send mode control commands to the image sensor based on illumination, and The image sensor is also configured to generate first image data and second image data based on mode control commands.

19. A method of operating an imaging device, comprising: Receive user input commands to obtain images or videos; A first command to acquire an image or video is sent to the image sensor based on user input via at least one processor. First image data is generated in rolling shutter mode using the image sensor based on a first command. The first image data is output through the image sensor; The second image data is generated in global shutter mode based on the first command using the image sensor. The second image data is output through the image sensor; as well as The at least one processor generates third image data based on the first image data and the second image data.

20. The method of operating the imaging device according to claim 19, wherein, The step of outputting the first image data occurs before the step of outputting the second image data, or the step of outputting the first image data occurs after the step of outputting the second image data.

Images