Image sensing system and method of operating the same

Abstract

To provide an image sensing system capable of implementing a power save mode, and an operation method thereof.SOLUTION: An image sensing system 1000 may include a pixel array including first to fourth pixels, a first column line connected to the first and second pixels, a second column line connected to the third and fourth pixels, a readout circuit connected to the first and second column lines to receive an output signal and output image data based on the output signal, and an image signal processor 100 configured to generate metadata associated with a motion vector based on the image data. The start time of the integration time for the first pixel and the third pixel may be the same, and the start time of the integration time for the second pixel and the fourth pixel may be the same.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image sensing system and a method of operation thereof. [Background technology]

[0002] An image sensing device is a semiconductor device that converts optical information into an electrical signal. Such image sensing devices include charge-coupled devices (CCDs) and complementary metal-oxide semiconductor (CMOS) image sensing devices.

[0003] A CMOS image sensor is abbreviated as CIS (CMOS image sensor). A CIS has multiple pixels arranged two-dimensionally. Each pixel includes, for example, a photodiode (PD). The photodiode converts incident light into an electrical signal.

[0004] 2. Description of the Related Art In recent years, with the development of the computer and communication industries, there has been an increasing demand for high-performance image sensors in a variety of fields, such as digital cameras, camcorders, smartphones, game consoles, security cameras, medical micro cameras, and robots. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2023-122576 Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention has been made in view of the above-mentioned conventional techniques, and an object of the present invention is to provide an image sensing system capable of implementing a power save mode and an operating method thereof. [Means for solving the problem]

[0007] In order to achieve the above object, an image sensing system according to one aspect of the present invention includes: a pixel array having a first sub-pixel array including first and second pixels and a second sub-pixel array including third and fourth pixels; a first column line connected to the first and second pixels; a second column line connected to the third and fourth pixels and different from the first column line; a readout circuit connected to the first and second column lines to receive output signals from the first to fourth pixels and output image data based on the output signals; and an image signal processor that performs interpolation based on the image data to generate metadata related to motion vectors of objects included in the image data, wherein the readout circuit reads out the first pixel and the second pixel in that order for pixels connected to the first column line, and reads out the third pixel and the fourth pixel in that order for pixels connected to the second column line, and wherein starting points of integration times for the first pixel and the third pixel are the same, and starting points of integration times for the second pixel and the fourth pixel are the same.

[0008] In order to achieve the above object, according to another aspect of the present invention, there is provided an image sensing system comprising: an image sensor configured to capture an image of an object and generate image data; an image signal processor configured to receive the image data from the image sensor, perform image processing, and generate metadata relating to a motion vector of the object based on a result of the image processing; and a timing generator control circuit configured to receive the metadata from the image signal processor and generate a control signal for controlling a timing generator based on the metadata, wherein the image sensor comprises a pixel array including a plurality of pixels, a row driver circuit, at least one row line connected to the row driver circuit and extending in a first direction, and first, second, third, and fourth pixels connected to the at least one row line. a first column line connected to the first pixel and the third pixel and extending in a second direction intersecting the first direction; a second column line connected to the second pixel and the fourth pixel and extending in the second direction, the second column line being different from the first column line; a readout circuit connected to the first column line and the second column line, receiving output signals from the first to fourth pixels and outputting the image data based on the output signals; and a timing generator configured to transmit an operation timing reference signal to the row driver circuit based on the control signal received from the timing generator control circuit, wherein the image signal processor performs image processing on the image data based on a readout order between the first pixel and the third pixel and a readout order between the second pixel and the fourth pixel.

[0009] In order to achieve the above object, one aspect of the present invention provides a method for operating an image sensing system having an image signal processor, the method comprising the steps of: sensing an imaged object by an image sensor to generate image data; receiving the image data and performing image processing by the image signal processor; generating metadata relating to a motion vector of the object based on a result of the image processing by the image signal processor; receiving the metadata by a timing generator control circuit and generating a control signal for controlling a timing generator based on the received metadata; and timing the start of an integration time of a pixel included in the image sensor based on the control signal by the timing generator. and adjusting a start point of integration time of the pixels by the timing generator, wherein the step of adjusting the start point of integration time of the pixels by the timing generator includes a step of expanding, by the timing generator, an interval between the first point in time and the second point in time based on the control signal for the first pixel and the third pixel, among the first, second, third, and fourth pixels included in the image sensor, whose start points of integration time are the same at a first point in time, and for the second pixel and the fourth pixel, among the first to fourth pixels, whose start points of integration time are the same at a second point in time after the first point in time, wherein the first pixel and the second pixel are connected to a first column line, and the third pixel and the fourth pixel are connected to a second column line different from the first column line. Specific details of other embodiments are included in the detailed description and drawings. [Effects of the Invention]

[0010] According to the present invention, the image sensor can be driven in a rolling shutter mode for pixels that share an analog-to-digital converter among a plurality of pixels included in a pixel array, and can be driven in a global shutter mode for pixels that do not share an analog-to-digital converter and have the same phase in different pixel arrays. Also, when the moving speed of an object is not fast, the sensing operation speed of the image sensor can be adjusted to be relatively slow, and the image sensor can be driven in a power save mode. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a diagram illustrating an image sensing system according to an embodiment. [Figure 2] FIG. 1 is a diagram illustrating a conceptual layout of an image sensor according to an embodiment. [Figure 3] 1 is a diagram illustrating an image sensor according to an embodiment; [Figure 4] FIG. 2 is a diagram illustrating a pixel array according to an embodiment. [Figure 5] FIG. 5 is a cross-sectional view of the pixel array taken along line AA in FIG. 4. [Figure 6] 1 is a diagram illustrating an image sensor according to an embodiment; [Figure 7] FIG. 7 is a diagram for explaining the pixel in FIG. 6. [Figure 8] FIG. 5 is a diagram for explaining the pixel array of FIG. [Figure 9] FIG. 2 is a diagram for explaining a readout operation sequence of a first example of an image sensor according to an embodiment. [Figure 10] 10A and 10B are diagrams illustrating a readout operation sequence of a second example of an image sensor according to an embodiment. [Figure 11] FIG. 10 is a diagram for explaining a readout operation sequence of a third example of an image sensor according to an embodiment. [Figure 12]1A and 1B are diagrams for conceptually explaining that an image sensor according to an embodiment operates in a global shutter mode for pixels having a first phase. [Figure 13] 10A and 10B are diagrams for conceptually explaining that an image sensor according to an embodiment operates in a global shutter mode for pixels having a second phase. [Figure 14] 10A and 10B are diagrams for conceptually explaining that an image sensor according to an embodiment operates in a global shutter mode for pixels having a third phase. [Figure 15] 10A and 10B are diagrams for conceptually explaining that an image sensor according to an embodiment operates in a global shutter mode for a pixel having a fourth phase. [Figure 16] 10A and 10B are diagrams for conceptually explaining that an image sensor according to an embodiment operates in a rolling shutter manner with respect to pixels included in a first sub-pixel array. [Figure 17] 10A and 10B are diagrams for conceptually explaining that an image sensor according to an embodiment operates in a rolling shutter manner with respect to pixels included in a second sub-pixel array. [Figure 18] 10A and 10B are diagrams for conceptually explaining that an image sensor according to an embodiment operates in a rolling shutter mode with respect to pixels included in a third sub-pixel array. [Figure 19] 10A and 10B are diagrams for conceptually explaining that an image sensor according to an embodiment operates in a rolling shutter mode with respect to pixels included in a fourth sub-pixel array. [Figure 20] 10A and 10B are diagrams illustrating a frame image signal according to an embodiment. [Figure 21] 1 is a diagram illustrating an image sensing system according to an embodiment. [Figure 22] 1 is a flowchart illustrating a method of operating an image sensing system according to an embodiment. [Figure 23]1 is a flowchart illustrating a method of operating an image signal processor according to an embodiment. [Figure 24] 24 is a diagram for explaining the operation method of the image signal processor according to FIG. 23. FIG. [Figure 25] 24 is a diagram for explaining the operation method of the image signal processor according to FIG. 23. FIG. [Figure 26] 24 is a diagram for explaining the operation method of the image signal processor according to FIG. 23. FIG. [Figure 27] 24 is a diagram for explaining the operation method of the image signal processor according to FIG. 23. FIG. [Figure 28] 1 is a flowchart illustrating a method of operating an image signal processor according to an embodiment. [Figure 29] 29 is a diagram for explaining the operation method of the image signal processor according to FIG. 28. [Figure 30] 29 is a diagram for explaining the operation method of the image signal processor according to FIG. 28. [Figure 31] 1 is a diagram illustrating motion vector calculation in an image signal processor according to an embodiment; [Figure 32] 4A and 4B are diagrams illustrating a first frame image signal generated by an image sensor according to an embodiment. [Figure 33] 10A and 10B are diagrams illustrating a second frame image signal generated by an image sensor according to an embodiment. [Figure 34] 10 is a diagram illustrating a start point of an integration time for each sub-pixel array for a first frame image signal of an image sensor according to an embodiment. [Figure 35] 10 is a diagram illustrating a start point of an integration time for each sub-pixel array for a second frame image signal of an image sensor, according to an embodiment. [Figure 36]10A and 10B are diagrams illustrating an operation of controlling a start point of an integration time for each sub-pixel array for a second frame image signal of an image sensor, according to an embodiment. [Figure 37] FIG. 1 illustrates an image sensing system according to one embodiment. [Figure 38] 1 is a diagram illustrating an electronic device including a multi-camera module according to an embodiment. [Figure 39] FIG. 39 is a diagram for explaining the camera module of FIG. 38. DETAILED DESCRIPTION OF THE INVENTION

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, specific examples of embodiments for carrying out the image sensing system and the operating method thereof of the present invention will be described in detail with reference to the drawings.

[0013] FIG. 1 is a diagram illustrating an image sensing system according to an embodiment.

[0014] Referring to FIG. 1, an image sensing system 1000 includes an image sensor 200, an image signal processor 100, and a timing generator control circuit 300.

[0015] The image sensor 200 senses an image of an object and generates image data IDATA. The image sensor 200 is connected to the image signal processor 100 and provides the image data IDATA to the image signal processor 100. The image signal processor 100 receives the image data IDATA from the image sensor 200 and performs image processing on the image data IDATA.

[0016] The image signal processor 100 performs image processing on the image data IDATA contained in the image sensor 200. However, the relative locations of the image signal processor 100 and the image sensor 200 may be changed depending on the embodiment. For example, the image signal processor 100 may be located separately from the image sensor 200.

[0017] The image signal processor 100 performs interpolation on the image data IDATA. For example, the image signal processor 100 performs interpolation such as upscaling or phase correction on the image data IDATA. Phase correction is also called phase shift. The image signal processor 100 calculates a motion vector of an object based on image data generated by performing interpolation on the image data IDATA.

[0018] For example, the image signal processor 100 calculates the velocity and acceleration of an object by comparing the image data generated by interpolating the image data IDATA over time according to the movement of the object. The image signal processor 100 also generates information about the path of movement of the object by comparing the image data generated by interpolating the image data IDATA over time according to the movement of the object. In this way, the image signal processor 100 calculates the motion vector of the object and generates metadata (MD) including at least one of information about the velocity, acceleration, and path of movement of the object.

[0019] The image signal processor 100 transmits the metadata MD to the timing generator control circuit 300. The timing generator control circuit 300 generates a control signal CS for controlling the timing generator included in the image sensor 200 based on the received metadata MD.

[0020] FIG. 2 is a diagram illustrating a conceptual layout of an image sensor according to an embodiment.

[0021] 2, the image sensor 200 includes a first region S1 and a second region S2 stacked in a third direction Z. The first region S1 and the second region S2 extend in a first direction X and a second direction Y intersecting the third direction Z as shown, and components of the image sensor 200 are arranged in the first region S1 and the second region S2.

[0022] In the following description, the upper surface or upper portion will be described based on the third direction Z, and the lower surface or lower portion will be described based on the direction opposite to the third direction Z.

[0023] A third region including a memory may be disposed below the second region S2. In this case, the memory disposed in the third region receives image data from the first region S1 and the second region S2, stores or processes the image data, and retransmits the image data to the first region S1 and the second region S2. In this case, the memory may include memory devices such as dynamic random access memory (DRAM), static random access memory (SRAM), spin transfer torque magnetic random access memory (STT-MRAM), and flash memory. When the memory includes, for example, a DRAM, the memory may receive and process image data at a relatively high speed. In another embodiment, the memory may be disposed in the second region S2.

[0024] The first region S1 includes a pixel array region PA and a first peripheral region PH1, and the second region S2 includes a logic circuit region LC, a second peripheral region PH2, and an image signal processor (100, shown in FIG. 1). The first region S1 and the second region S2 are stacked one above the other.

[0025] In the first region S1, the pixel array region PA is the same as the pixel array 270 described below with reference to Figure 3. The pixel array 270 includes a plurality of unit pixels arranged in a matrix, each of which includes a photodiode and a transistor.

[0026] The first peripheral area PH1 includes a plurality of pads arranged around the pixel array area PA, and transmits and receives electrical signals to and from an external device.

[0027] In the second region S2, the logic circuit region LC includes electronic elements including a plurality of transistors, which are electrically connected to the pixel array region PA to provide a signal or control an output signal for each unit pixel in the pixel array region PA.

[0028] In the logic circuit region LC, for example, a timing generator 220, a row driver circuit 210, a ramp signal generating circuit 230, a readout circuit 240, a column driver circuit 280, an analog-to-digital converter (ADC) 250, a buffer circuit 260, etc., which will be described later with reference to Figure 3, are arranged. In the logic circuit region LC, for example, blocks other than the pixel array 270 are arranged.

[0029] In the second region S2, a second peripheral region PH2 is also arranged in a region corresponding to the first peripheral region PH1 of the first region S1, but the embodiment is not limited to this.

[0030] When the image signal processor 100 is included in the image sensor 200, the image signal processor 100 is disposed in the second region S2 together with the logic circuit region LC and the second peripheral region PH2. In this case, the image signal processor 100 receives image data (IDATA, shown in FIG. 1) from the buffer circuit 260 and performs image processing on the received image data IDATA.

[0031] FIG. 3 is a diagram illustrating an image sensor according to an embodiment.

[0032] 3, the image sensor 200 includes a pixel array region PA and a logic circuit region LC. Here, the pixel array region PA is included in the first region S1, and the logic circuit region LC is included in the second region S2. The pixel array region PA performs analog signal processing on analog signals, and the logic circuit region LC performs analog signal processing and digital signal processing on analog signals transferred from the pixel array region PA.

[0033] The pixel array area PA includes a pixel array 270. The logic circuit area LC also includes a timing generator 220, a row driver circuit 210, a ramp signal generating circuit 230, a readout circuit 240, a column driver circuit 280, a buffer circuit 260, and the like.

[0034] The image sensor 200 senses an object captured through a lens under the control of the image signal processor 100, and the image signal processor 100 outputs the image sensed and output by the image sensor 200 to a display. In this case, the display includes all devices that output images. For example, the display includes computers, smartphones, and other image output terminals.

[0035] The pixel array 270 includes a number of pixels in a matrix configuration, each connected to a number of row lines and a number of column lines. Each of the pixels includes a red pixel for converting light in the red spectral region into an electrical signal, a green pixel for converting light in the green spectral region into an electrical signal, and a blue pixel for converting light in the blue spectral region into an electrical signal. A color filter array is arranged above each of the pixels constituting the pixel array 270 to transmit light in a specific spectral region.

[0036] The pixel array 270 includes a number of photoelectric conversion elements, such as photodiodes, pinned photodiodes, etc. The pixel array 270 senses light using the photoelectric conversion elements and converts the light into electrical signals to generate video signals.

[0037] The timing generator 220 outputs operation timing reference signals such as control signals or clock signals to each of the row driver circuit 210, the analog-to-digital converter 250, the ramp signal generation circuit 230, and the column driver circuit 280, thereby controlling the operation or timing of the row driver circuit 210, the analog-to-digital converter 250, the ramp signal generation circuit 230, and the column driver circuit 280.

[0038] The row driver circuit 210 drives the pixel array 270. The row driver circuit 210 causes the pixels of the pixel array 270 to output reset and image signal components. Here, the row driver circuit 210 does not select a specific row of the pixel array 270. For example, the row driver circuit 210 causes signals to be output from all pixels of the pixel array 270 during a specific time period. That is, all pixels of the pixel array 270 output signals simultaneously. In this case, the image sensor 200 is called a digital pixel sensor (DPS). However, embodiments according to the technical concept of the present invention are not limited thereto, and the row driver circuit 210 may also select a specific row of the pixel array 270.

[0039] The ramp signal generation circuit 230 generates and transmits a ramp signal used in the readout circuit 240. For example, the readout circuit 240 includes a correlated double sampler CDS, a comparator, an analog-to-digital converter 250, etc., and the ramp signal generation circuit 230 generates and transmits a ramp signal used in the correlated double sampler CDS, the comparator, the analog-to-digital converter 250, etc.

[0040] The readout circuit 240 samples the pixel signals provided by the pixel array 270, compares them with the ramp signal, and then converts the analog image signals into digital image signals based on the comparison results.

[0041] The analog-to-digital converter 250 compares the reference signal provided by the ramp signal generating circuit 230 with the pixel signal provided by the pixel array 270, and outputs a signal representing the comparison result. The analog-to-digital converter 250 counts the signal representing the comparison result and outputs it to the column driver circuit 280.

[0042] The column driver circuit 280 temporarily stores the provided digital signal and performs an operation on the provided digital signal. The column driver circuit 280 provides the operated digital signal to the image signal processor 100 via the buffer circuit 260.

[0043] Here, the analog-to-digital converter 250 processes both analog and digital signals. Specifically, a part of the analog-to-digital converter 250 converts the pixel signal into a digital signal, and another part of the analog-to-digital converter 250 compares the converted digital signal with a reference signal and outputs a signal representing the comparison result. However, embodiments according to the technical concept of the present invention are not limited thereto, and correlated double sampling may be performed in a separate block.

[0044] The buffer circuit 260 includes, for example, a latch unit. The buffer circuit 260 temporarily stores image data IDATA to be provided to the outside and transmits the image data IDATA to the image signal processor 100 (shown in FIG. 3). The image data IDATA provided from the buffer circuit 260 to the image signal processor 100 undergoes a certain image processing process by the image signal processor 100. After this image processing process, the data output from the image signal processor 100 is transmitted to an external memory or an external device.

[0045] FIG. 4 is a diagram illustrating a pixel array according to an embodiment.

[0046] Referring to FIG. 4, a pixel array PA includes a plurality of unit pixels PX. The unit pixels PX are arranged two-dimensionally. For example, the unit pixels PX are repeatedly arranged in a first direction X and a second direction Y. The unit pixels PX are arranged at regular intervals. For example, the pixel array PA may be arranged in a Bayer pattern. However, embodiments according to the technical concept of the present invention are not limited thereto, and the pixel array PA may be arranged in a tetra pattern, a nona pattern, or the like. In the following description, an embodiment in which the pixel array PA is arranged in a quad Bayer pattern will be described as an example. However, it goes without saying that embodiments according to the technical concept of the present invention are not limited thereto.

[0047] Referring to FIG. 4, the pixel array PA includes a plurality of unit pixels PX. The lenses LS cover the plurality of unit pixels PX. For example, one lens LS covers four unit pixels PX. The corresponding plurality of unit pixels PX include the same type of color filter. For example, the plurality of unit pixels PX covered by one lens LS include one of red, green, and blue color filters. Here, the color filters included in the pixel array PA have an RGB Quad Bayer pattern, but are not limited thereto.

[0048] FIG. 5 is a cross-sectional view of the pixel array taken along line AA in FIG.

[0049] 5, the pixel array PA includes a unit pixel PX1 and a unit pixel PX2, which are arranged adjacent to each other.

[0050] The pixel array PA includes a substrate (6a, 6b), photoelectric transistors (8a, 8b), an anti-reflection film 7, a side anti-reflection film 4, color filters (3a, 3b), an upper planarization film 2, a lower planarization film 5, and a lens LS.

[0051] The substrates (6a, 6b) may be, for example, P-type or N-type bulk substrates, or P-type or N-type epitaxial layers grown on P-type bulk substrates, or N-type or P-type epitaxial layers grown on N-type bulk substrates. In addition to semiconductor substrates, organic plastic substrates may also be used as the substrates (6a, 6b).

[0052] The photosensitive transistors (8a, 8b) are photodiodes, phototransistors, photogates, pinned photodiodes, or combinations thereof.

[0053] The anti-reflection coating 7 and the side surface anti-reflection coating 4 prevent external light incident on the lens LS from penetrating into the G1 region and the G2 region, respectively. The anti-reflection coating 7 and the side surface anti-reflection coating 4 are made of an insulating film such as a silicon oxide film, a silicon nitride film, a silicon oxynitride film, a resin, or a combination or stack of these, but the embodiment is not limited to this.

[0054] The upper planarization film 2 and the lower planarization film 5 are formed flat with the color filters 3a and 3b interposed therebetween. The upper planarization film 2 and the lower planarization film 5 may include at least one of a silicon oxide-based material, a silicon nitride-based material, a resin, or a combination thereof, but the embodiment is not limited thereto.

[0055] 6 is a diagram illustrating an image sensor according to an embodiment, and FIG. 7 is a diagram illustrating a pixel of FIG.

[0056] 6 and 7, the image sensor 200 includes a row driver circuit 210, row lines (ROW1 to ROW(m)), column lines (COL1 to COL(n)), a pixel array 270, a ramp signal generating circuit 230, analog-to-digital converters (250_1, 250_2, ..., 250_(n-1), 250n), a buffer circuit 260, etc.

[0057] The row driver circuit 210 drives the pixel array 270 row by row. The row driver circuit 210 generates a transmission control signal TS, a reset control signal RS, a selection control signal SEL, etc. and provides them to the pixels PX of the pixel array 270.

[0058] However, the embodiment is not limited to this, and the row driver circuit 210 may output signals from all pixels of the pixel array 270 during a specific time period without selecting a specific row of the pixel array 270.

[0059] The pixel array 270 includes a plurality of pixels PX, which are arranged in a grid pattern along a number of rows and columns. The pixel array 270 uses the plurality of pixels PX to sense light and convert it into electrical signals to generate image signals.

[0060] A plurality of row lines (ROW1 to ROW(m)) extend in a first direction D1. The plurality of row lines (ROW1 to ROW(m)) are sequentially arranged in a second direction D2. For example, the first row line ROW1 is spaced apart from the second row line ROW2 in the second direction D2. A plurality of column lines (COL1 to COL(n)) extend in the second direction D2. The plurality of column lines (COL1 to COL(n)) are sequentially arranged in the first direction D1. For example, the second column line COL2 is spaced apart from the first column line COL1 in the first direction D1. However, the embodiment of the present invention is not limited thereto.

[0061] A plurality of pixels PX are connected to row lines (ROW1 to ROW(m)) and column lines (COL1 to COL(n)). For example, one pixel PX is connected to both the first row line ROW1 and the first column line COL1. The pixel PX is located at the intersection of the first row line ROW1 and the first column line COL1. Therefore, the plurality of pixels PX are arranged in a lattice pattern (i.e., a matrix shape).

[0062] 7, a pixel PX includes a photodiode PD, a transfer transistor TX, a reset transistor RX, a source follower SF, and a selection transistor SX, where the pixel PX is a unit constituting the pixel array 270 or the pixel array area PA.

[0063] One end of the transfer transistor TX is connected to the photodiode PD, and the other end is connected to a floating diffusion region FD. A control electrode of the transfer transistor TX receives a transmission control signal TS. Here, light incident on the image sensor 200 is converted into an electrical signal via the photodiode PD. The converted electrical signal is transmitted to the floating diffusion region FD via the transfer transistor TX.

[0064] One end of the reset transistor RX is connected to the power supply voltage VDD, and the other end is connected to the floating diffusion region FD. A control electrode of the reset transistor RX receives a reset control signal RS. One end of the source follower SF is connected to the power supply voltage VDD, and the other end is connected to one end of the select transistor SX. A control electrode of the source follower SF is connected to the floating diffusion region FD. The other end of the select transistor SX is connected to a column line (COL1 to COL(n)), and a control electrode receives a select control signal SEL.

[0065] Control signals (TS, RS, SEL) for controlling the transistors (TX, RX, SX) are output from the row driver circuit 210. An output signal Vout of the selection transistor SX is supplied to the column lines (COL1 to COL(n)). The output signal Vout corresponds to an analog signal. That is, the output signal Vout output from the pixel PX is converted into a digital signal through the readout circuit 240 and transmitted to the image signal processor 100 as image data IDATA.

[0066] 6 are signal lines that transmit a transmission control signal TS, a reset control signal RS, and a selection control signal SEL from the row driver circuit 210 to the pixel PX. Also, the column lines (COL1 to COL(n)) in FIG. 6 are signal lines that transmit an output signal Vout of the selection transistor SX.

[0067] 6, the plurality of column lines (COL1 to COL(n)) are connected to analog-to-digital converters (250_1, 250_2, ..., 250_(n-1), 250_n), respectively. Here, the analog-to-digital converters (250_1, 250_2, ..., 250_(n-1), 250_n) are also connected to the ramp signal generation circuit 230. That is, the analog-to-digital converters (250_1, 250_2, ..., 250_(n-1), 250_n) receive ramp signals from the ramp signal generation circuit 230 and receive output signals Vout from the plurality of column lines (COL1 to COL(n)). The analog-to-digital converters (250_1, 250_2, ..., 250_(n-1), 250_n) perform correlation double sampling (CDS) operations, counting operations, etc. to convert the output signals Vout, which are analog signals, into digital signals. Here, the analog-to-digital converters 250_1, 250_2, ..., 250_(n-1), 250_n are included in the readout circuit 240 as shown in FIG. 3. The readout circuit 240 including the analog-to-digital converters 250_1, 250_2, ..., 250_(n-1), 250_n is spaced apart from the pixel array 270 in a direction opposite to the second direction D2. The analog-to-digital converters 250_1, 250_2, ..., 250_(n-1), 250_n are sequentially arranged along the first direction D1. That is, the analog-to-digital converters 250_1, 250_2, ..., 250_(n-1), 250_n are arranged to correspond to each pixel PX.

[0068] The buffer circuit 260 is connected to the plurality of analog-digital converters (250_1, 250_2, ..., 250_(n-1), 250_n) and receives converted digital signals from the analog-digital converters (250_1, 250_2, ..., 250_(n-1), 250_n). The buffer circuit 260 is disposed in a direction opposite to the second direction D2 from the analog-digital converters (250_1, 250_2, ..., 250_(n-1), 250_n).

[0069] FIG. 8 is a diagram for explaining the pixel array of FIG.

[0070] 8, the pixel array PA includes a first subpixel array SA1, a second subpixel array SA2, a third subpixel array SA3, and a fourth subpixel array SA4. The first subpixel array SA1 includes green pixels (PX_G1, PX_G2, PX_G3, PX_G4). The second subpixel array SA2 includes red pixels (PX_R1, PX_R2, PX_R3, PX_R4). The third subpixel array SA3 includes blue pixels (PX_B1, PX_B2, PX_B3, PX_B4). The fourth subpixel array SA4 includes green pixels (PX_G1', PX_G2', PX_G3', PX_G4').

[0071] The green pixels (PX_G1, PX_G2, PX_G3, PX_G4) included in the first subpixel array SA1 are covered by a single lens (LS, shown in FIG. 4) and include green filters. The red pixels (PX_R1, PX_R2, PX_R3, PX_R4) included in the second subpixel array SA2 are covered by a single lens LS and include red color filters. The blue pixels (PX_B1, PX_B2, PX_B3, PX_B4) included in the third subpixel array SA3 are covered by a single lens LS and include blue color filters. The green pixels (PX_G1, PX_G2, PX_G3, PX_G4) included in the fourth subpixel array SA4 are covered by a single lens LS and include green filters.

[0072] The green pixels (PX_G1, PX_G2, PX_G3, PX_G4) included in the first subpixel array SA1 share one column line and one analog-to-digital converter. The red pixels (PX_R1, PX_R2, PX_R3, PX_R4) included in the second subpixel array SA2 share one column line and one analog-to-digital converter. The blue pixels (PX_B1, PX_B2, PX_B3, PX_B4) included in the third subpixel array SA3 share one column line and one analog-to-digital converter. The green pixels (PX_G1', PX_G2', PX_G3', PX_G4') included in the fourth subpixel array SA4 share one column line and one analog-to-digital converter.

[0073] For example, referring to FIG. 6, multiple pixels included in the same subpixel array are connected to one column line (e.g., column line COL1) among the column lines (COL1 to COL(n)), and therefore the multiple pixels share the analog-to-digital converter 250_1 connected to column line COL1.

[0074] In this way, since multiple pixels included in the same sub-pixel array share one analog-to-digital converter, the image sensor can operate in a rolling shutter mode during a readout operation for multiple pixels included in the same sub-pixel array.

[0075] For example, the green pixels (PX_G1, PX_G2, PX_G3, PX_G4) included in the first subpixel array SA1 have different integration time start points. The red pixels (PX_R1, PX_R2, PX_R3, PX_R4) included in the second subpixel array SA2 have different integration time start points. The blue pixels (PX_B1, PX_B2, PX_B3, PX_B4) included in the third subpixel array SA3 have different integration time start points. The green pixels (PX_G1', PX_G2', PX_G3', PX_G4') included in the fourth subpixel array SA4 have different integration time start points.

[0076] The green pixels (PX_G1, PX_G2, PX_G3, PX_G4) included in the first subpixel array SA1, the red pixels (PX_R1, PX_R2, PX_R3, PX_R4) included in the second subpixel array SA2, the blue pixels (PX_B1, PX_B2, PX_B3, PX_B4) included in the third subpixel array SA3, and the green pixels (PX_G1', PX_G2', PX_G3', PX_G4') included in the fourth subpixel array SA4 are read out by different analog-to-digital converters, respectively. Therefore, the image sensor (200, shown in FIG. 1) can operate in a global shutter mode during the readout operation for the multiple pixels included in the different subpixel arrays. When the image sensor operates in the global shutter mode, the start point of the integration time for the multiple pixels is the same. The operation of the rolling shutter mode and the global shutter mode of the image sensor will be described later with reference to FIG. 12, etc.

[0077] 9 to 11 are diagrams illustrating various examples of readout operation sequences of an image sensor according to an embodiment.

[0078] In the following description, it is assumed that the first direction X is the downward direction, the opposite direction to the first direction X is the upward direction, the second direction Y is the rightward direction, and the opposite direction to the second direction Y is the leftward direction.

[0079] 9, the readout circuit 240 simultaneously performs a readout operation on the first to fourth subpixel arrays (SA1 to SA4). At this time, the order of the readout operation on the multiple pixels included in the same subpixel array is set in advance. For example, as shown in FIG. 9, the readout circuit 240 performs a readout operation on the multiple pixels included in the same subpixel array in the order of the pixel located in the upper left, the pixel located in the upper right, the pixel located in the lower left, and the pixel located in the lower right.

[0080] For example, the readout operation is performed on the first subpixel array SA1 in the order of green pixel PX_G1, green pixel PX_G2, green pixel PX_G3, and green pixel PX_G4. The readout operation is performed on the second subpixel array SA2 in the order of red pixel PX_R1, red pixel PX_R2, red pixel PX_R3, and red pixel PX_R4. The readout operation is performed on the third subpixel array SA3 in the order of blue pixel PX_B1, blue pixel PX_B2, blue pixel PX_B3, and blue pixel PX_B4. The readout operation is performed on the fourth subpixel array SA4 in the order of green pixel PX_G1', green pixel PX_G2', green pixel PX_G3', and green pixel PX_G4'.

[0081] The phase of the pixels is determined according to the readout order of the readout circuit 240 for the multiple pixels included in each subpixel array. For example, the first pixel to be read out among the multiple pixels included in each subpixel array has a first phase (Phase 1). The second pixel to be read out among the multiple pixels included in each subpixel array has a second phase (Phase 2). The third pixel to be read out among the multiple pixels included in each subpixel array has a third phase (Phase 3). The fourth pixel to be read out among the multiple pixels included in each subpixel array has a fourth phase (Phase 4).

[0082] For example, the green pixel PX_G1 of the first subpixel array SA1, the red pixel PX_R1 of the second subpixel array SA2, the blue pixel PX_B1 of the third subpixel array SA3, and the green pixel PX_G1' of the fourth subpixel array SA4 have the first phase (Phase1).

[0083] Furthermore, the green pixel PX_G2 of the first subpixel array SA1, the red pixel PX_R2 of the second subpixel array SA2, the blue pixel PX_B2 of the third subpixel array SA3, and the green pixel PX_G2' of the fourth subpixel array SA4 have a second phase (Phase2).

[0084] Furthermore, the green pixel PX_G3 of the first subpixel array SA1, the red pixel PX_R3 of the second subpixel array SA2, the blue pixel PX_B3 of the third subpixel array SA3, and the green pixel PX_G3' of the fourth subpixel array SA4 have a third phase (Phase3).

[0085] Furthermore, the green pixel PX_G4 of the first subpixel array SA1, the red pixel PX_R4 of the second subpixel array SA2, the blue pixel PX_B4 of the third subpixel array SA3, and the green pixel PX_G4' of the fourth subpixel array SA4 have a fourth phase (Phase4).

[0086] In this way, pixels included in the same subpixel array and subjected to sequential readout operations are defined as pixels having different phases. That is, pixels connected to the same column line and sharing an analog-to-digital converter, and whose image sensor operates in a rolling shutter mode during the readout operation, are pixels having different phases. Pixels included in the same subpixel array and having different phases have different integration time start points.

[0087] Pixels included in different subpixel arrays and having the same readout operation sequence within each subpixel array are defined as pixels having the same phase relationship. That is, pixels connected to different column lines and not sharing an analog-to-digital converter, so that the image sensor operates in a global shutter mode during the readout operation, are pixels having the same phase. Pixels included in different subpixel arrays and having the same phase have the same start point of integration time.

[0088] Next, referring to FIG. 10, unlike FIG. 9, the readout circuit 240 performs the readout operation on multiple pixels included in the same subpixel array in the order of the pixel located in the upper left, the pixel located in the lower left, the pixel located in the upper right, and the pixel located in the lower right.

[0089] For example, the readout operation is performed on the first subpixel array SA1 in the order of green pixel PX_G1, green pixel PX_G3, green pixel PX_G2, and green pixel PX_G4. The readout operation is performed on the second subpixel array SA2 in the order of red pixel PX_R1, red pixel PX_R3, red pixel PX_R2, and red pixel PX_R4. The readout operation is performed on the third subpixel array SA3 in the order of blue pixel PX_B1, blue pixel PX_B3, blue pixel PX_B2, and blue pixel PX_B4. The readout operation is performed on the fourth subpixel array SA4 in the order of green pixel PX_G1', green pixel PX_G3', green pixel PX_G2', and green pixel PX_G4'.

[0090] At this time, the first pixel to be read out among the plurality of pixels included in each subpixel array has a first phase (Phase 1). The second pixel to be read out among the plurality of pixels included in each subpixel array has a second phase (Phase 2). The third pixel to be read out among the plurality of pixels included in each subpixel array has a third phase (Phase 3). The fourth pixel to be read out among the plurality of pixels included in each subpixel array has a fourth phase (Phase 4).

[0091] For example, the green pixel PX_G1 of the first subpixel array SA1, the red pixel PX_R1 of the second subpixel array SA2, the blue pixel PX_B1 of the third subpixel array SA3, and the green pixel PX_G1' of the fourth subpixel array SA4 have a first phase (Phase 1). The green pixel PX_G3 of the first subpixel array SA1, the red pixel PX_R3 of the second subpixel array SA2, the blue pixel PX_B3 of the third subpixel array SA3, and the green pixel PX_G3' of the fourth subpixel array SA4 have a second phase (Phase 2). The green pixel PX_G2 of the first subpixel array SA1, the red pixel PX_R2 of the second subpixel array SA2, the blue pixel PX_B2 of the third subpixel array SA3, and the green pixel PX_G2' of the fourth subpixel array SA4 have a third phase (Phase 3). Furthermore, the green pixel PX_G4 of the first subpixel array SA1, the red pixel PX_R4 of the second subpixel array SA2, the blue pixel PX_B4 of the third subpixel array SA3, and the green pixel PX_G4' of the fourth subpixel array SA4 have a fourth phase (Phase4).

[0092] As such, the readout order of the readout circuit 240 for multiple pixels included in the same sub-pixel array may vary depending on the embodiment, and therefore, the types of pixels included in each of the first phase (Phase 1) to fourth phase (Phase 4) are different.

[0093] 11, unlike in FIGS. 9 and 10, the readout circuit 240 performs the readout operation on a plurality of pixels included in some subpixel arrays in the order of the upper left pixel, the upper right pixel, the lower left pixel, and the lower right pixel. Also, the readout circuit 240 performs the readout operation on a plurality of pixels included in other subpixel arrays in the order of the upper left pixel, the lower left pixel, the upper right pixel, and the lower right pixel.

[0094] For example, the readout operation is performed on the first subpixel array SA1 in the order of green pixel PX_G1, green pixel PX_G2, green pixel PX_G3, and green pixel PX_G4. The readout operation is performed on the second subpixel array SA2 in the order of red pixel PX_R1, red pixel PX_R3, red pixel PX_R2, and red pixel PX_R4. The readout operation is performed on the third subpixel array SA3 in the order of blue pixel PX_B1, blue pixel PX_B2, blue pixel PX_B3, and blue pixel PX_B4. The readout operation is performed on the fourth subpixel array SA4 in the order of green pixel PX_G1', green pixel PX_G3', green pixel PX_G2', and green pixel PX_G4'.

[0095] At this time, the first pixel to be read out among the plurality of pixels included in each subpixel array has a first phase (Phase 1). The second pixel to be read out among the plurality of pixels included in each subpixel array has a second phase (Phase 2). The third pixel to be read out among the plurality of pixels included in each subpixel array has a third phase (Phase 3). The fourth pixel to be read out among the plurality of pixels included in each subpixel array has a fourth phase (Phase 4).

[0096] For example, the green pixel PX_G1 of the first subpixel array SA1, the red pixel PX_R1 of the second subpixel array SA2, the blue pixel PX_B1 of the third subpixel array SA3, and the green pixel PX_G1' of the fourth subpixel array SA4 have the first phase (Phase1).

[0097] Furthermore, the green pixel PX_G2 of the first subpixel array SA1, the red pixel PX_R3 of the second subpixel array SA2, the blue pixel PX_B2 of the third subpixel array SA3, and the green pixel PX_G3' of the fourth subpixel array SA4 have a second phase (Phase2).

[0098] Furthermore, the green pixel PX_G3 of the first subpixel array SA1, the red pixel PX_R2 of the second subpixel array SA2, the blue pixel PX_B3 of the third subpixel array SA3, and the green pixel PX_G2' of the fourth subpixel array SA4 have a third phase (Phase3).

[0099] Furthermore, the green pixel PX_G4 of the first subpixel array SA1, the red pixel PX_R4 of the second subpixel array SA2, the blue pixel PX_B4 of the third subpixel array SA3, and the green pixel PX_G4' of the fourth subpixel array SA4 have a fourth phase (Phase4).

[0100] As such, the readout order of the readout circuit 240 for multiple pixels included in the same sub-pixel array may vary depending on the embodiment, and therefore, the types of pixels included in each of the first phase (Phase 1) to fourth phase (Phase 4) are different.

[0101] Furthermore, the readout order of the pixels included in the plurality of sub-pixel arrays (SA1 to SA4) included in the pixel array PA differs for each sub-pixel array.

[0102] FIG. 12 is a diagram for conceptually explaining that an image sensor according to an embodiment operates in a global shutter mode for pixels having a first phase.

[0103] In the following, an example will be described in which the readout circuit 240 of the image sensor 200 performs a readout operation on multiple pixels included in the same subpixel array in the order of the pixel located in the upper left, the pixel located in the upper right, the pixel located in the lower left, and the pixel located in the lower right, as described with reference to FIG.

[0104] 12, the image sensor 200 operates in a global shutter mode for the pixels (PX_G1, PX_R1, PX_B1, PX_G1′) having the first phase. In the global shutter mode, the image sensor 200 sequentially performs a reset operation for resetting charges accumulated in the floating diffusion nodes of the pixels (PX_G1, PX_R1, PX_B1, PX_G1′) having the first phase, a storage operation for storing photocharges generated in the photoelectric conversion elements, and a readout operation.

[0105] The integration time refers to the time required to substantially accumulate photocharges generated in a photoelectric conversion element, such as a photodiode, included in each of the pixels PX. The integration time is also referred to as charge accumulation time. The integration time refers to the time interval from when the image sensor 200 opens the shutter, i.e., when the photocharges begin to be exposed to light, to when the image sensor 200 closes the shutter, i.e., when the photocharges stop being exposed to light.

[0106] Readout time means the time it takes for a pixel signal corresponding to the photocharge generated in each of the pixels (PX_G1, PX_R1, PX_B1, PX_G1') having the first phase to be output from each of the pixels (PX_G1, PX_R1, PX_B1, PX_G1') having the first phase.

[0107] The image sensor 200 controls the pixels (PX_G1, PX_R1, PX_B1, PX_G1') having the first phase so that the reset time (Reset Time) and the integration time (integration time) are the same. As a result, the start points of the integration times of the pixels (PX_G1, PX_R1, PX_B1, PX_G1') having the first phase (Phase 1) are all the same.

[0108] In this way, the image sensor 200 operates in global shutter mode for each pixel (PX_G1, PX_R1, PX_B1, PX_G1') having the first phase, and all signals photoelectrically converted by all photoelectric conversion elements in one frame image are transmitted to the floating diffusion node at once, and then the signal of the corresponding pixel is output from the row.

[0109] FIG. 13 is a diagram for conceptually explaining that an image sensor according to an embodiment operates in a global shutter mode for pixels having a second phase.

[0110] In the following, explanations that overlap with the above-described embodiment will be omitted, and differences will be mainly described.

[0111] 13, the image sensor 200 drives the pixels (PX_G2, PX_R2, PX_B2, PX_G2') having the second phase in a global shutter mode. The image sensor 200 controls the pixels (PX_G2, PX_R2, PX_B2, PX_G2') having the second phase so that the reset time, which is the time for performing a reset operation, and the integration time, which is the time for performing an accumulation operation, are the same. As a result, the start points of the integration times of the pixels (PX_G2, PX_R2, PX_B2, PX_G2') having the second phase (Phase 2) are all the same.

[0112] FIG. 14 is a diagram for conceptually explaining that an image sensor according to an embodiment operates in a global shutter mode for pixels having a third phase.

[0113] 14, the image sensor 200 drives the pixels (PX_G3, PX_R3, PX_B3, PX_G3') having the third phase in a global shutter mode. The image sensor 200 controls the pixels (PX_G3, PX_R3, PX_B3, PX_G3') having the third phase so that the reset time (Reset Time) and the integration time (Integration Time) are the same. As a result, the start points of the integration times of the pixels (PX_G3, PX_R3, PX_B3, PX_G3') having the third phase (Phase 3) are all the same.

[0114] FIG. 15 is a diagram for conceptually explaining that an image sensor according to an embodiment operates in a global shutter mode for pixels having a fourth phase.

[0115] 15, the image sensor 200 drives the pixels (PX_G4, PX_R4, PX_B4, PX_G4') having the fourth phase in a global shutter mode. The image sensor 200 controls the pixels (PX_G4, PX_R4, PX_B4, PX_G4') having the fourth phase so that the reset time, during which a reset operation is performed, and the integration time, during which an accumulation operation is performed, are the same. As a result, the start points of the integration times of the pixels (PX_G4, PX_R4, PX_B4, PX_G4') having the fourth phase (Phase 4) are all the same.

[0116] FIG. 16 is a diagram conceptually illustrating that an image sensor according to an embodiment operates in a rolling shutter manner with respect to pixels included in a first sub-pixel array.

[0117] 16, the image sensor 200 drives the pixels (PX_G1, PX_G2, PX_G3, PX_G4) included in the first subpixel array SA1 in rolling shutter mode. In the rolling shutter mode, the image sensor 200 sequentially executes a reset operation for resetting charges accumulated in the floating diffusion nodes, an accumulation operation for accumulating photocharges generated in the photoelectric conversion elements, and a readout operation. However, unlike the global shutter mode described with reference to FIGS. 12 to 15, the reset operation, accumulation operation, and readout operation for each of the pixels (PX_G1, PX_G2, PX_G3, PX_G4) included in the first subpixel array SA1 have a time difference for each pixel (PX_G1, PX_G2, PX_G3, PX_G4).

[0118] For example, the image sensor 200 sequentially executes the reset operation, accumulation operation, and readout operation according to the readout order (i.e., the order of green pixel PX_G1, green pixel PX_G2, green pixel PX_G3, and green pixel PX_G4) for the plurality of pixels (PX_G1, PX_G2, PX_G3, PX_G4) included in the first subpixel array SA1. As a result, the start points of the reset time, the start points of the integration time, and the start points of the readout time for the plurality of pixels (PX_G1, PX_G2, PX_G3, PX_G4) included in the first subpixel array SA1 are different for each pixel (PX_G1, PX_G2, PX_G3, PX_G4).

[0119] FIG. 17 is a diagram conceptually illustrating that an image sensor according to an embodiment operates in a rolling shutter manner with respect to pixels included in a second sub-pixel array.

[0120] 17, the image sensor 200 operates in a rolling shutter mode for the pixels (PX_R1, PX_R2, PX_R3, PX_R4) included in the second subpixel array SA2. In the rolling shutter mode, the image sensor 200 sequentially executes a reset operation for resetting charges accumulated in the floating diffusion nodes, an accumulation operation for accumulating photocharges generated in the photoelectric conversion elements, and a readout operation. However, unlike the global shutter mode, the reset operation, accumulation operation, and readout operation for each of the pixels (PX_R1, PX_R2, PX_R3, PX_R4) included in the second subpixel array SA2 have a time difference for each pixel (PX_R1, PX_R2, PX_R3, PX_R4).

[0121] For example, the image sensor 200 sequentially performs the reset operation, accumulation operation, and readout operation according to the readout order (i.e., the order of red pixel PX_R1, red pixel PX_R2, red pixel PX_R3, and red pixel PX_R4) for the multiple pixels (PX_R1, PX_R2, PX_R3, PX_R4) included in the second subpixel array SA2. As a result, the start points of the reset time, integration time, and readout time for the multiple pixels (PX_R1, PX_R2, PX_R3, PX_R4) included in the second subpixel array SA2 are different for each pixel (PX_R1, PX_R2, PX_R3, PX_R4).

[0122] FIG. 18 is a diagram conceptually illustrating that an image sensor according to an embodiment operates in a rolling shutter mode with respect to pixels included in the third sub-pixel array.

[0123] 18, the image sensor 200 drives the pixels (PX_B1, PX_B2, PX_B3, PX_B4) included in the third subpixel array SA3 in rolling shutter mode. In rolling shutter mode, the image sensor 200 sequentially executes a reset operation to reset charges accumulated in the floating diffusion nodes, an accumulation operation to accumulate photocharges generated in the photoelectric conversion elements, and a readout operation. However, unlike the global shutter mode, the reset operation, accumulation operation, and readout operation for each of the pixels (PX_B1, PX_B2, PX_B3, PX_B4) included in the third subpixel array SA3 have a time difference for each pixel (PX_B1, PX_B2, PX_B3, PX_B4).

[0124] For example, the image sensor 200 sequentially executes the reset operation, accumulation operation, and readout operation according to the readout order (i.e., the order of blue pixel PX_B1, blue pixel PX_B2, blue pixel PX_B3, and blue pixel PX_B4) for the plurality of pixels (PX_B1, PX_B2, PX_B3, PX_B4) included in the third subpixel array SA3. As a result, the start points of the reset time, integration time, and readout time for the plurality of pixels (PX_B1, PX_B2, PX_B3, PX_B4) included in the third subpixel array SA3 are different for each pixel (PX_B1, PX_B2, PX_B3, PX_B4).

[0125] FIG. 19 is a diagram conceptually illustrating that an image sensor according to an embodiment operates in a rolling shutter manner with respect to pixels included in a fourth sub-pixel array.

[0126] 19, the image sensor 200 drives the pixels (PX_G1′, PX_G2′, PX_G3′, PX_G4′) included in the fourth subpixel array SA4 in rolling shutter mode. In rolling shutter mode, the image sensor 200 sequentially executes a reset operation to reset charges accumulated in the floating diffusion nodes, an accumulation operation to accumulate photocharges generated in the photoelectric conversion elements, and a readout operation. However, unlike the global shutter mode, the reset operation, accumulation operation, and readout operation for each of the pixels (PX_G1′, PX_G2′, PX_G3′, PX_G4′) included in the fourth subpixel array SA4 have a time difference for each pixel (PX_G1′, PX_G2′, PX_G3′, PX_G4′).

[0127] For example, the image sensor 200 sequentially performs the reset operation, accumulation operation, and readout operation according to the readout order (i.e., the order of green pixel PX_G1′, green pixel PX_G2′, green pixel PX_G3′, and green pixel PX_G4′) for the plurality of pixels (PX_G1′, PX_G2′, PX_G3′, and PX_G4′) included in the fourth subpixel array SA4. As a result, the start points of the reset time, integration time, and readout time for the plurality of pixels (PX_G1′, PX_G2′, PX_G3′, and PX_G4′) included in the fourth subpixel array SA4 are different for each pixel.

[0128] FIG. 20 is a diagram illustrating a frame image signal according to an embodiment.

[0129] 20, a frame image signal (Frame Image) is a signal output by the image sensor 200 after sensing light from the pixel array PA of FIG. 4. For example, light passes through the color filters (3a, 3b, shown in FIG. 5) of the pixel array PA and reaches the photoelectric transistors (8a, 8b, shown in FIG. 5), and the frame image signal (Frame Image) is output from the logic circuit region (LC, shown in FIG. 2). That is, the frame image signal (Frame Image) is included in the image data IDATA shown in FIG. 1, etc.

[0130] For example, the frame image signal (Frame Image) includes a first green pixel value G1 output by sensing light transmitted through a green color filter 3a. The frame image signal (Frame Image) also includes a second green pixel value G2 output by sensing light transmitted through a green color filter 3b. That is, the green pixel values ​​(G1 to G4), blue pixel values ​​(B1 to B4), red pixel values ​​(R1 to R4), and green pixel values ​​(G1' to G4') shown in Fig. 20 are image data IDATA output by the image sensor 200 after sensing light transmitted through color filters having the corresponding colors.

[0131] The pixel values ​​of the frame image signal (Frame Image) are arranged to correspond to the colors of the color filters of the pixel array PA as shown in Fig. 20. However, Fig. 20 merely shows the arrangement of pixel values ​​according to the position of each unit pixel PX, and the storage positions of the pixel values ​​of the actually output frame image signal (Frame Image) are not limited to the positions shown in the figure.

[0132] FIG. 21 is a diagram illustrating an image sensing system according to an embodiment.

[0133] Referring to FIG. 21, the image sensing system 1000 includes an image sensor 200, which includes a timing generator 220, a plurality of pixels PX, an analog-to-digital converter ADC, a buffer circuit 260, an image signal processor 100, and a timing generator control circuit 300.

[0134] The pixels are grouped according to whether they share an analog-to-digital converter ADC. As shown in Fig. 21, each of the grouped pixels PX shares one analog-to-digital converter ADC. The pixels PX included in one group correspond to the pixels included in one sub-pixel array described with reference to Figs. 1 to 20. The image data IDATA read out via the readout circuit 240 is buffered in the buffer circuit 260 and then provided to the image signal processor 100.

[0135] The image signal processor 100 processes the image data IDATA to calculate the motion vector of the object sensed by the image sensor 200. The image signal processor 100 generates metadata MD relating to the motion vector of the object based on the calculation result and provides the metadata MD to the timing generator control circuit 300.

[0136] The timing generator control circuit 300 outputs a control signal CS for controlling the timing generator 220 based on the metadata MD, and provides it to the timing generator 220. The timing generator 220 outputs a signal for controlling the operation timing of components included in the image sensor 200 (for example, a row driver circuit (210, shown in FIG. 3), etc.) based on the control signal CS, and provides it to the above components.

[0137] 21 shows the image sensor 200 including both the image signal processor 100 and the timing generator control circuit 300, the embodiment is not limited to this. For example, at least one of the image signal processor 100 and the timing generator control circuit 300 may be arranged separately from the image sensor 200.

[0138] FIG. 22 is a flowchart illustrating a method of operating an image sensing system according to an embodiment.

[0139] 21 and 22, the image sensor 200 senses an image of an object to generate image data (S100), and then transmits the generated image data IDATA (S101). The image signal processor 100 receives the image data IDATA (S102) and performs interpolation on the image data IDATA (S103). The image signal processor 100 then extracts characteristic information of the object based on the result of interpolating the image data IDATA (S104).

[0140] The image signal processor 100 then calculates a motion vector of the object based on the characteristic information of the object (S105), and generates metadata MD of the object based on the calculated motion vector (S106). The image signal processor 100 then transmits the metadata MD to the timing generator control circuit 300 (S107), and the timing generator control circuit 300 receives the metadata MD (S108) and generates a control signal based on the metadata MD (S109). The timing generator control circuit 300 then transmits a control signal CS to the timing generator 220 (S110), and the timing generator 220 receives the control signal CS (S111) and controls at least one component included in the logic circuit region (LC, shown in FIG. 2) based on the control signal CS (S112).

[0141] 23 is a flowchart illustrating an operation method of the image signal processor according to an embodiment, and FIGS. 24 to 27 are diagrams illustrating an operation method of the image signal processor according to FIG.

[0142] The operation of the image signal processor according to an embodiment will be described below with reference to FIGS.

[0143] 23 and 24, the image signal processor 100 receives image data IDATA from the image sensor 200 (S201). The image data IDATA is a frame image signal (Frame Image) shown in FIG. 20. The frame image signal (Frame Image) is image data IDATA corresponding to one frame. The object 400 moves diagonally from the upper left to the lower right through a plurality of pixels (PX_G1, PX_G2, PX_G3, PX_G4, PX_R1, PX_R2, PX_R3, PX_R4, PX_B1, PX_B2, PX_B3, PX_B4, PX_G1', PX_G2', PX_G3', PX_G4') arranged in a matrix along the first direction X and the second direction Y as time passes through T1, T2, T3, and T4.

[0144] At this time, the pixels (PX_G1, PX_R1, PX_B1, PX_G1') having the first phase (Phase 1) to be read out first in each sub-pixel array (SA1 to SA4) are read out at time T1, the pixels (PX_G2, PX_R2, PX_B2, PX_G2') having the second phase (Phase 2) to be read out second are read out at time T2, the pixels (PX_G3, PX_R3, PX_B3, PX_G3') having the third phase (Phase 3) to be read out third are read out at time T3, and the pixels (PX_G4, PX_R4, PX_B4, PX_G4') having the fourth phase (Phase 4) to be read out fourth are read out at time T4.

[0145] That is, at time T1 when pixels (PX_G1, PX_R1, PX_B1, PX_G1') having the first phase (Phase 1) are read out, the object 400 is located at the upper left with respect to a plurality of pixels (PX_G1, PX_G2, PX_G3, PX_G4, PX_R1, PX_R2, PX_R3, PX_R4, PX_B1, PX_B2, PX_B3, PX_B4, PX_G1', PX_G2', PX_G3', PX_G4') and is located at the upper left with respect to a plurality of pixels (PX_G1, PX_R1, PX_R2, PX_R3, PX_R4, PX_B1, PX_B2, PX_B3, PX_B4, PX_G1', PX_G2', PX_G3', PX_G4'). At time T2 when pixels PX_G2, PX_R2, PX_B2, and PX_G2' having the third phase (P ase2) are read out, the object 400 is positioned lower and rightward from the position at time T1 with respect to a plurality of pixels (PX_G1, PX_G2, PX_G3, PX_G4, PX_R1, PX_R2, PX_R3, PX_R4, PX_B1, PX_B2, PX_B3, PX_B4, PX_G1', PX_G2', PX_G3', and PX_G4'). At time T3 when the pixel (PX_G3, PX_R3, PX_B3, PX_G3') having the pixel (PX_G1, PX_G2, PX_G3, PX_G4, PX_R1, PX_R2, PX_R3, PX_R4, PX_B1, PX_B2, PX_B3, PX_B4, PX_G1', PX_G2', PX_G3', PX_G4') is read out, the object 400 is positioned lower and right than its position at time T2, and is in the fourth position. At time T4 when pixels (PX_G4, PX_R4, PX_B4, PX_G4') having phase (Phase 4) are read out, object 400 is positioned lower and to the right of multiple pixels (PX_G1, PX_G2, PX_G3, PX_G4, PX_R1, PX_R2, PX_R3, PX_R4, PX_B1, PX_B2, PX_B3, PX_B4, PX_G1', PX_G2', PX_G3', PX_G4') relative to its position at time T3.

[0146] 23 and 25, the image signal processor 100 divides the image data IDATA for each phase to generate a plurality of first image signals (IS1a to IS1d) (S202). Referring to Fig. 25, the image signal processor 100 samples the image data IDATA for each phase to generate a first image signal IS1a corresponding to time T1, a first image signal IS1b corresponding to time T2, a first image signal IS1c corresponding to time T3, and a first image signal IS1d corresponding to time T4.

[0147] Next, referring to FIGS. 23 and 26, the image signal processor 100 performs upscaling on each of the plurality of first image signals (IS1a to IS1d) to generate a plurality of second image signals (IS2a to IS2d) (S203).

[0148] Referring to Figure 26, the image data on the left is a diagram showing the first image signal IS1a corresponding to time T1, the first image signal IS1b corresponding to time T2, the first image signal IS1c corresponding to time T3, and the first image signal IS1d corresponding to time T4, while the image data on the right is a diagram showing the second image signal IS2a corresponding to time T1, the second image signal IS2b corresponding to time T2, the second image signal IS2c corresponding to time T3, and the second image signal IS2d corresponding to time T4.

[0149] The first image signals IS1a to IS1d are the result of outputting only pixel values ​​of pixels that have the same readout time, i.e., the same phase, for multiple pixels in the pixel array, so there are portions in the image data where pixel values ​​are not output. For example, referring to the first image signal IS1a corresponding to time T1, at time T1, only the upper left pixels (PX_G1, PX_R1, PX_B1, PX_G1') in the first to fourth sub-pixel arrays SA1 to SA4 are read out, so the upper right pixels (PX_G2, PX_R2, PX_B2, PX_G2'), the lower left pixels (PX_G3, PX_R3, PX_B3, PX_G3'), and the lower right pixels (PX_G4, PX_R4, PX_B4, PX_G4') in the first to fourth sub-pixel arrays SA1 to SA4 have not yet been read out. Therefore, the pixel value of the pixel has not yet been output, and the first image signal IS1a corresponding to the time point T1 does not include the pixel value of the pixel.

[0150] Therefore, before the image signal processor 100 processes the image data IDATA received from the image sensor 200 and calculates the motion vector of the object, it first performs interpolation on the image data IDATA. For example, the image signal processor 100 performs upscaling on each of the first image signals (IS1a to IS1d) shown on the left side of Fig. 26 to generate the second image signals (IS2a to IS2d) shown on the right side of Fig. 26.

[0151] Referring to the second image signal IS2a corresponding to time T1, pixel values ​​not included in the first image signal IS1a corresponding to time T1 are calculated using pixel values ​​of pixels adjacent to the corresponding pixel whose pixel values ​​are known. For example, in the first image signal IS1a corresponding to time T1, the green pixel PX_G2 has not yet been read out, so its pixel value is unknown. At this time, the image signal processor 100 calculates a pixel value G1a of the green pixel PX_G2 as a value similar to the pixel value (G1, R1) using the pixel values ​​(G1, R1) of the green pixel PX_G1 and red pixel PX_R1, which are adjacent to the green pixel PX_G2 and have output pixel values. For example, the pixel value G1a may be any value between the pixel values ​​G1 and R1.

[0152] In this manner, the image signal processor 100 calculates pixel values ​​of pixels (PX_G2, PX_G3, PX_G4, PX_R2, PX_R3, PX_R4, PX_B2, PX_B3, PX_B4, PX_G2', PX_G3', PX_G4') whose pixel values ​​are unknown at time T1, and generates a second image signal IS2a corresponding to time T1 including output pixel values ​​(G1, R1, B1, G1') and calculated pixel values ​​(G1a, G1b, G1c, R1a, R1b, R1c, B1a, B1b, B1c, G1'a, G1'b, G1'c).

[0153] In a similar manner, at time T2, the image signal processor 100 calculates pixel values ​​for pixels (PX_G1, PX_G3, PX_G4, PX_R1, PX_R3, PX_R4, PX_B1, PX_B3, PX_B4, PX_G1', PX_G3', PX_G4') whose pixel values ​​are unknown, and generates a second image signal IS2b corresponding to time T2 which includes output pixel values ​​(G2, R2, B2, G2') and calculated pixel values ​​(G2a, G2b, G2c, R2a, R2b, R2c, B2a, B2b, B2c, G2'a, G2'b, G2'c).

[0154] Similarly, at time T3, image signal processor 100 calculates pixel values ​​for pixels (PX_G1, PX_G2, PX_G4, PX_R1, PX_R2, PX_R4, PX_B1, PX_B2, PX_B4, PX_G1', PX_G2', PX_G4') whose pixel values ​​are unknown, to generate a second image signal IS2c corresponding to time T3 that includes output pixel values ​​(G3, R3, B3, G3') and calculated pixel values ​​(G3a, G3b, G3c, R3a, R3b, R3c, B3a, B3b, B3c, G3'a, G3'b, G3'c). At time T4, the image signal processor 100 calculates pixel values ​​of pixels (PX_G1, PX_G2, PX_G3, PX_R1, PX_R2, PX_R3, PX_B1, PX_B2, PX_B3, PX_G1', PX_G2', PX_G3') whose pixel values ​​are unknown, and generates a second image signal IS2d corresponding to time T4 which includes output pixel values ​​(G4, R4, B4, G4') and calculated pixel values ​​(G4a, G4b, G4c, R4a, R4b, R4c, B4a, B4b, B4c, G4'a, G4'b, G4'c).

[0155] 23 and 27, the image signal processor 100 compares a plurality of second image signals (IS2a to IS2d) to extract characteristic information of the object 400 (S204), and calculates a motion vector of the object 400 based on the extracted characteristic information of the object 400 (S205). The image signal processor 100 compares the plurality of second image signals (IS2a to IS2d) by time point. For example, referring to FIG. 27, the image signal processor 100 compares the second image signal IS2a corresponding to time point T1 with the second image signal IS2b corresponding to time point T2 to extract characteristic information of the object 400 and calculate the motion vector of the object 400.

[0156] The image signal processor 100 knows information about the time difference between time T1 and time T2, and compares the position of the object 400 in the second image signal IS2a corresponding to time T1 with the position of the object 400 in the second image signal IS2a corresponding to time T2 to know the distance the object 400 has moved in the first direction X and the distance the object 400 has moved in the second direction Y during the time difference from time T1 to time T2.

[0157] For example, the image signal processor 100 extracts characteristic information of the object 400, such that the time difference between time T1 and time T2 is 1 ms and the object 400 has moved by one pixel during the time difference from time T1 to time T2. In this case, the image signal processor 100 calculates that the speed of the object 400 is 1 pixel / ms based on the characteristic information of the object 400. The information about the speed of the object 400 calculated in this way is included in the metadata MD related to the motion vector of the object 400.

[0158] Similarly, the image signal processor 100 compares the second image signal IS2b corresponding to time T2 with the second image signal IS2c corresponding to time T3 to extract characteristic information of the object 400 and calculates the motion vector of the object 400, and compares the second image signal IS2c corresponding to time T3 with the second image signal IS2d corresponding to time T4 to extract characteristic information of the object 400 and calculates the motion vector of the object 400.

[0159] For example, the image signal processor 100 compares the position of the object 400 in the second image signal IS2b corresponding to time T2 with the position of the object 400 in the second image signal IS2c corresponding to time T3 to determine the movement distance of the object 400 in the first direction X and the movement distance of the object 400 in the second direction Y during the time difference from time T2 to time T3. Furthermore, since the image signal processor 100 knows information about the time difference of the object 400 between time T2 and time T3, it calculates the velocity of the object 400 during the time interval from time T2 to time T3 based on the extracted characteristic information of the object 400. Thereafter, the image signal processor 100 calculates the acceleration of the object 400 by comparing the velocity information of the object 400 during the time interval from time T1 to time T2 with the velocity information of the object 400 during the time interval from time T2 to time T3.

[0160] In addition, the image signal processor 100 analyzes the movement path of the object 400 from time T1 to time T4 to generate information about the predicted movement path of the object 400. The information about the acceleration of the object 400 and the information about the movement path of the object 400 generated in this manner are also included in the metadata MD about the motion vector of the object 400 (S206).

[0161] Next, the image signal processor 100 transmits the generated metadata MD to the timing generator control circuit 300 (S207).

[0162] 28 is a flowchart illustrating an operation method of an image signal processor according to an embodiment, and FIGS. 29 and 30 are diagrams illustrating an operation method of the image signal processor according to FIG.

[0163] Referring to Figure 28, the step (S301) in which the image signal processor 100 receives image data IDATA from the image sensor 200 and the step (S302) in which the image data IDATA is divided into the same phases to generate a plurality of first image signals (IS1a to IS1d) are the same as the operations described with reference to Figures 23 to 25, so duplicated content will be omitted below.

[0164] 28 and 29, the image signal processor 100 performs phase correction on each of the plurality of first image signals (IS1a to IS1d) to generate a plurality of third image signals (IS3a to IS3d) (S303).

[0165] Referring to Figure 29, the image data on the left is a diagram showing the first image signal IS1a corresponding to time T1, the first image signal IS1b corresponding to time T2, the first image signal IS1c corresponding to time T3, and the first image signal IS1d corresponding to time T4, while the image data on the right is a diagram showing the third image signal IS3a corresponding to time T1, the third image signal IS3b corresponding to time T2, the third image signal IS3c corresponding to time T3, and the third image signal IS3d corresponding to time T4, respectively.

[0166] The first image signals (IS1a to IS1d) shown on the left side of Figure 29 are the result of only outputting pixel values ​​for pixels that have the same readout time, i.e., the same phase, for multiple pixels included in the pixel array, so there are parts in the image data where pixel values ​​are not output.

[0167] Therefore, before the image signal processor 100 processes the image data IDATA received from the image sensor 200 and calculates the motion vector of the object 400, it first performs interpolation on the image data IDATA. For example, the image signal processor 100 performs phase correction on each of the first image signals (IS1a to IS1d) shown on the left side of Fig. 29 to generate the third image signals (IS3a to IS3d) shown on the right side of Fig. 29.

[0168] 29, the first image signal IS1a corresponding to time T1 includes only pixel values ​​(G1, R1, B1, G1') output from the upper left pixel (PX_G1, PX_R1, PX_B1, PX_G1') of the plurality of pixels included in each of the first to fourth sub-pixel arrays (SA1 to SA4). The first image signal IS1b corresponding to time T2 includes only pixel values ​​(G2, R2, B2, G2') output from the upper right pixel (PX_G2, PX_R2, PX_B2, PX_G2') of the plurality of pixels included in each of the first to fourth sub-pixel arrays (SA1 to SA4). The first image signal corresponding to time T3 includes only pixel values ​​(G3, R3, B3, G3') output from the pixel (PX_G3, PX_R3, PX_B3, PX_G3') located at the bottom left of the plurality of pixels included in each of the first to fourth sub-pixel arrays (SA1 to SA4). The first image signal corresponding to time T4 includes only pixel values ​​(G4, R4, B4, G4') output from the pixel (PX_G4, PX_R4, PX_B4, PX_G4') located at the bottom right of the plurality of pixels included in each of the first to fourth sub-pixel arrays (SA1 to SA4).

[0169] As such, since the pixel values ​​contained in each of the first image signals (IS1a to IS1d) at different times are output from pixels located at different positions, the image signal processor 100 compares the multiple first image signals (IS1a to IS1d) at different times and performs interpolation to correct the phase of the pixels outputting pixel values ​​within each of the multiple first image signals (IS1a to IS1d) before extracting characteristic information of the object 400.

[0170] For example, for a first image signal IS1a corresponding to time T1, the image signal processor 100 shifts the phase of the green pixel PX_G1, which is outputting a pixel value among the green pixels (PX_G1, PX_G2, PX_G3, PX_G4) included in the first sub-pixel array SA1, to the center of the first sub-pixel array SA1 in the first image signal IS1a corresponding to time T1. Also, for a first image signal IS1a corresponding to time T1, the image signal processor 100 shifts the phase of the red pixel PX_R1, which is outputting a pixel value among the red pixels (PX_R1, PX_R2, PX_R3, PX_R4) included in the second sub-pixel array SA2, to the center of the second sub-pixel array SA2.

[0171] Furthermore, for the first image signal IS1a corresponding to time T1, the image signal processor 100 shifts the phase of the blue pixel PX_B1, which is outputting a pixel value among the blue pixels (PX_B1, PX_B2, PX_B3, PX_B4) included in the third subpixel array SA3, to the center of the third subpixel array SA3 in the first image signal IS1a corresponding to time T1. Furthermore, for the first image signal IS1a corresponding to time T1, the image signal processor 100 shifts the phase of the green pixel PX_G1', which is outputting a pixel value among the green pixels (PX_G1', PX_G2', PX_G3', PX_G4') included in the fourth subpixel array SA4, to the center of the fourth subpixel array SA4 in the first image signal IS1a corresponding to time T1. As a result, the image signal processor 100 generates the third image signal IS3a corresponding to time T1 shown on the right side of FIG.

[0172] The third image signal IS3a corresponding to time T1 includes a phase-corrected pixel value G1_PS of a green pixel, a phase-corrected pixel value R1_PS of a red pixel, a phase-corrected pixel value B1_PS of a blue pixel, and a phase-corrected pixel value G1'_PS of a green pixel, where the phase-corrected pixel value G1_PS is equal to the pixel value G1 included in the first image signal IS1a corresponding to time T1, the phase-corrected pixel value R1_PS of a red pixel is equal to the pixel value R1 included in the first image signal IS1a corresponding to time T1, the phase-corrected pixel value B1_PS of a blue pixel is equal to the pixel value B1 included in the first image signal IS1a corresponding to time T1, and the phase-corrected pixel value G1'_PS is equal to the pixel value G1' included in the first image signal IS1a corresponding to time T1. However, the image signal processor 100 recognizes that the position of the green pixel outputting the pixel value G1_PS in the third image signal IS3a has been changed to the center of the first sub-pixel array SA1, i.e., the portion where the plurality of pixels (PX_G1, PX_G2, PX_G3, PX_G4) are adjacent to each other. Also, the image signal processor 100 recognizes that the position of the red pixel outputting the pixel value R1_PS in the third image signal IS3a has been changed to the center of the second sub-pixel array SA2, i.e., the portion where the plurality of pixels (PX_R1, PX_R2, PX_R3, PX_R4) are adjacent to each other.

[0173] The image signal processor 100 also recognizes that the position of the blue pixel that outputs pixel value B1_PS in the third image signal IS3a has been changed to the center of the third sub-pixel array SA3, i.e., the portion where multiple pixels (PX_B1, PX_B2, PX_B3, PX_B4) are adjacent to each other. The image signal processor 100 also recognizes that the position of the green pixel that outputs pixel value G1'_PS in the third image signal IS3a has been changed to the center of the fourth sub-pixel array SA4, i.e., the portion where multiple pixels (PX_G1', PX_G2', PX_G3', PX_G4') are adjacent to each other.

[0174] In a similar manner, the image signal processor 100 performs phase-correcting interpolation on the first image signal IS1b corresponding to time T2 to generate a third image signal IS3b corresponding to time T2, performs phase-correcting interpolation on the first image signal IS1c corresponding to time T3 to generate a third image signal IS3c corresponding to time T3, and performs phase-correcting interpolation on the first image signal IS1d corresponding to time T4 to generate a third image signal IS3d corresponding to time T4.

[0175] Referring to the multiple third image signals (IS3a to IS3d) shown on the right side of Figure 29, the phases of the green pixels of the first sub-pixel array SA1, the red pixels of the second sub-pixel array SA2, the blue pixels of the third sub-pixel array SA3, and the green pixels of the fourth sub-pixel array SA4, which output pixel values ​​in the third image signals, are the same at each time point.

[0176] Therefore, referring to Figures 28 and 30, the image signal processor 100 compares a plurality of third image signals (IS3a to IS3d) to extract characteristic information of the object 400 (S304), and calculates a motion vector of the object based on the extracted characteristic information of the object (S305).

[0177] Furthermore, the image signal processor 100 generates metadata based on the motion vector of the object 400 (S306), and transmits the generated metadata MD to the timing generator control circuit 300 (S307).

[0178] For example, referring to FIG. 30, the image signal processor 100 compares the third image signal IS3a corresponding to time T1 with the third image signal IS3b corresponding to time T2 to determine that the object 400 has moved from the upper left to the lower right based on the phase-corrected green pixel of the first subpixel array SA1 during the time difference from time T1 to time T2.

[0179] In this way, the image signal processor 100 calculates the motion vector of the object 400 based on the extracted characteristic information of the object 400, and generates metadata MD including information such as speed and acceleration.

[0180] Among the operations of the image signal processor 100, the explanation of steps S304 to S307 is the same as the explanation of steps S204 to S207 explained with reference to FIG. 23, etc., and therefore, a duplicate explanation will be omitted below.

[0181] 28 to 30, the "phase" in step S302 of FIG. 28 and the "phase" in step S303 of FIG. 28 have different meanings. For example, the "phase" in step S302 of FIG. 28 is determined according to the readout order of the readout circuit for multiple pixels included in the same subpixel array. That is, multiple pixels included in the same subpixel array are not read out simultaneously but are read out sequentially in a specific order, resulting in pixels having different "phases." On the other hand, the "phase" in step S303 of FIG. 28 refers to the position where the pixel value is included in the image data IDATA output by the image sensor 200, i.e., the physical position occupied in the pixel array by the pixel whose pixel value read out at a specific time point is output as image data IDATA.

[0182] FIG. 31 is a diagram illustrating motion vector calculation in an image signal processor according to an embodiment.

[0183] 31, object 400 moves during the time interval between time T1 and time T2. At this time, image signal processor 100 identifies the coordinates of object 400 in the image data corresponding to time T1 as f1 and the coordinates of object 400 in the image data corresponding to time T2 as f2.

[0184] In one embodiment, the image signal processor 100 calculates the movement amount dx in the x direction, the movement amount dy in the y direction, and the time difference dx between time T1 and time T2 of the object 400 based on the following equation 1, and generates metadata regarding the motion vector of the object 400 based on this.

[0185] <Formula 1> f1:I(x,y,t)=f2:I(x+dx,y+dy,t+dt)

[0186] 32 and 33 are diagrams illustrating a first and second frame image signals generated by an image sensor according to an embodiment.

[0187] 32 and 33, the first frame image signal (Frame Image 1) in FIG. 32 is image data IDATA output by the image sensor 200 during the first frame time period, and the second frame image signal (Frame Image 2) in FIG. 33 is image data IDATA output by the image sensor 200 during the second frame time period after the first frame time period.

[0188] In the first and second frame time periods, the readout circuit 240 of the image sensor 200 performs the readout operation on a plurality of pixels included in the same sub-pixel array in the same order. For example, as shown in Figures 32 and 33, the readout circuit 240 performs the readout operation on the pixel located in the upper left, pixel located in the upper right, pixel located in the lower left, and pixel located in the lower right in that order.

[0189] Referring to FIG. 32, green pixel values ​​(G1, G2, G3, G4) are image data respectively output from green pixels (PX_G1, PX_G2, PX_G3, PX_G4) of the first sub-pixel array SA1 during the first frame time period, and red pixel values ​​(R1, R2, R3, R4) are image data respectively output from red pixels (PX_R1, PX_R2, PX_R3, PX_R4) of the second sub-pixel array SA2 during the first frame time period. The blue pixel values ​​(B1, B2, B3, B4) are image data output from the blue pixels (PX_B1, PX_B2, PX_B3, PX_B4) of the third sub-pixel array SA3 during the first frame time period, and the green pixel values ​​(G1', G2', G3', G4') are image data output from the green pixels (PX_G1', PX_G2', PX_G3', PX_G4') of the fourth sub-pixel array SA4 during the first frame time period.

[0190] Referring to FIG. 33, green pixel values ​​(G1_a, G2_a, G3_a, G4_a) are image data respectively output from green pixels (PX_G1, PX_G2, PX_G3, PX_G4) of the first sub-pixel array SA1 during the second frame time period, and red pixel values ​​(R1_a, R2_a, R3_a, R4_a) are image data respectively output from red pixels (PX_R1, PX_R2, PX_R3, PX_R4) of the second sub-pixel array SA2 during the second frame time period. The blue pixel values ​​(B1_a, B2_a, B3_a, B4_a) are image data output from the blue pixels (PX_B1, PX_B2, PX_B3, PX_B4) of the third sub-pixel array SA3 during the second frame time period, and the green pixel values ​​(G1'_a, G2'_a, G3'_a, G4'_a) are image data output from the green pixels (PX_G1', PX_G2', PX_G3', PX_G4') of the fourth sub-pixel array SA4 during the second frame time period.

[0191] The pixel value output by the same pixel during the first frame time interval and the pixel value output during the second frame time interval are different from each other depending on the movement of the object 400, which is the sensing target of the image sensor 200, during the first frame time interval and the subsequent second frame time interval. For example, the pixel value (G1, shown in FIG. 32) output from the green pixel PX_G1 during the first frame time interval is different from the pixel value (G1_a, shown in FIG. 33) output from the green pixel PX_G1 during the second frame time interval.

[0192] FIG. 34 is a diagram illustrating a start point of an integration time for each sub-pixel array for a first frame image signal of an image sensor, according to an embodiment.

[0193] Referring to Figures 32 and 34, the first frame image signal (Frame Image1) is a collection of pixel values ​​(G1, R1, B1, G1') output from pixels (PX_G1, PX_R1, PX_B1, PX_G1') where charge accumulation, i.e., integration, begins at time T1, pixel values ​​(G2, R2, B2, G2') output from pixels (PX_G2, PX_R2, PX_B2, PX_G2') where integration begins at time T2, pixel values ​​(G3, R3, B3, G3') output from pixels (PX_G3, PX_R3, PX_B3, PX_G3') where integration begins at time T3, and pixel values ​​(G4, R4, B4, G4') output from pixels (PX_G4, PX_R4, PX_B4, PX_G4') where integration begins at time T4.

[0194] At this time, the start point of the integration time for the pixels (PX_G1, PX_R1, PX_B1, PX_G1') having the first phase (Phase 1) is time T1. That is, the time at which the photocharges contained in the pixels (PX_G1, PX_R1, PX_B1, PX_G1') having the first phase (Phase 1) that are read out first in each subpixel array begin to accumulate is the same as time T1.

[0195] The start of the integration time for the pixels (PX_G2, PX_R2, PX_B2, PX_G2') having the second phase (Phase 2) is time T2, which is after time T1. That is, the time at which the photocharges contained in the pixels having the second phase (Phase 2) that are read out next to the pixels having the first phase (Phase 1) in each subpixel array begin to accumulate is the same as time T2.

[0196] In this way, since multiple pixels included in the same subpixel array share an analog-to-digital converter, there is a time difference of time interval TI1 between the start point T1 of the integration time of the pixels (PX_G1, PX_R1, PX_B1, PX_G1') having the first phase (Phase 1) and the start point T2 of the integration time of the pixels (PX_G2, PX_R2, PX_B2, PX_G2') having the second phase (Phase 2).

[0197] The start of the integration time for the pixels (PX_G3, PX_R3, PX_B3, PX_G3') having the third phase (Phase 3) is time T3, which is after time T2. That is, the time at which the photocharges contained in the pixels having the third phase (Phase 3) that are read out next to the pixels having the second phase (Phase 2) in each subpixel array begin to accumulate is the same as time T3.

[0198] In this way, since multiple pixels included in the same subpixel array share an analog-to-digital converter, there is a time difference of time interval TI1 between the start point T2 of the integration time of the pixels (PX_G2, PX_R2, PX_B2, PX_G2') having the second phase (Phase 2) and the start point T3 of the integration time of the pixels (PX_G3, PX_R3, PX_B3, PX_G3') having the third phase (Phase 3).

[0199] The start of the integration time for the pixels (PX_G4, PX_R4, PX_B4, PX_G4') having the fourth phase (Phase 4) is time T4, which is after time T3. That is, the time at which the photocharges contained in the pixels having the fourth phase (Phase 4) that are read out next to the pixels having the third phase (Phase 3) in each subpixel array begin to accumulate is the same as time T4.

[0200] In this way, since multiple pixels included in the same subpixel array share an analog-to-digital converter, there is a time difference of time interval TI1 between the start point T3 of the integration time of the pixels (PX_G3, PX_R3, PX_B3, PX_G3') having the third phase (Phase 3) and the start point T4 of the integration time of the pixels (PX_G4, PX_R4, PX_B4, PX_G4') having the fourth phase (Phase 4).

[0201] At this time, the start points of the pixel integration times are controlled by the timing generator 220 of the image sensor 200. That is, the timing generator 220 provides operation timing control signals to the row driver circuit 210 to adjust the start points of the integration times of the pixels (PX_G1, PX_R1, PX_B1, PX_G1') having the first phase (Phase 1) to time T1, the start points of the integration times of the pixels (PX_G2, PX_R2, PX_B2, PX_G2') having the second phase (Phase 2) to time T2, the start points of the integration times of the pixels (PX_G3, PX_R3, PX_B3, PX_G3') having the third phase (Phase 3) to time T3, and the start points of the integration times of the pixels (PX_G4, PX_R4, PX_B4, PX_G4') having the fourth phase (Phase 4) to time T4.

[0202] In addition, the timing generator 220 adjusts the time difference between the start point T1 of the integration time of a pixel having the first phase (Phase 1) and the start point T2 of the integration time of a pixel having the second phase (Phase 2), the time difference between the start point T2 of the integration time of a pixel having the second phase (Phase 2) and the start point T3 of the integration time of a pixel having the third phase (Phase 3), and the time difference between the start point T3 of the integration time of a pixel having the third phase (Phase 3) and the start point T4 of the integration time of a pixel having the fourth phase (Phase 4) so ​​that they are all the same within the time interval TI1.

[0203] FIG. 35 is a diagram illustrating a start point of an integration time for each sub-pixel array for a second frame image signal of an image sensor according to an embodiment.

[0204] 33 and 35, the second frame image signal (Frame Image2) includes pixel values ​​(G1_a, R1_a, B1_a, G1'_a) output from pixels (PX_G1, PX_R1, PX_B1, PX_G1') where charge accumulation, i.e., integration, begins at time T1', pixel values ​​(G2_a, R2_a, B2_a, G2'_a) output from pixels (PX_G2, PX_R2, PX_B2, PX_G2') where integration begins at time T2', and pixel values ​​(G2_a, R2_a, B2_a, G2'_a) output from pixels (PX_G2, PX_R2, PX_B2, PX_G2') where integration begins at time T2'. '_a), pixel values ​​(G3_a, R3_a, B3_a, G3'_a) output from pixels (PX_G3, PX_R3, PX_B3, PX_G3') where integration begins at time T3', and pixel values ​​(G4_a, R4_a, B4_a, G4'_a) output from pixels (PX_G4, PX_R4, PX_B4, PX_G4') where integration begins at time T4'.

[0205] At this time, the start point of the integration time for the pixels (PX_G1, PX_R1, PX_B1, PX_G1') having the first phase (Phase 1) is time T1'. That is, the time when the photocharges contained in each of the pixels (PX_G1, PX_R1, PX_B1, PX_G1') having the first phase (Phase 1) that are read out first in each subpixel array start to accumulate is the same as time T1'.

[0206] The start of the integration time for the pixels (PX_G2, PX_R2, PX_B2, PX_G2') having the second phase (Phase 2) is time T2', which is after time T1'. That is, the time at which the photocharges contained in the pixels having the second phase (Phase 2) that are read out next to the pixels having the first phase (Phase 1) in each subpixel array begin to accumulate is the same as time T2'.

[0207] In this way, since multiple pixels included in the same subpixel array share an analog-to-digital converter, there is a time difference of time interval TI1 between the start point T1' of the integration time of the pixels (PX_G1, PX_R1, PX_B1, PX_G1') having the first phase (Phase 1) and the start point T2' of the integration time of the pixels (PX_G2, PX_R2, PX_B2, PX_G2') having the second phase (Phase 2).

[0208] The start of the integration time for the pixels (PX_G3, PX_R3, PX_B3, PX_G3') having the third phase (Phase 3) is time T3', which is after time T2'. That is, the time at which the photocharges contained in the pixels having the third phase (Phase 3) that are read out next to the pixels having the second phase (Phase 2) in each subpixel array begin to accumulate is the same as time T3'.

[0209] In this way, since multiple pixels included in the same subpixel array share an analog-to-digital converter, there is a time difference of time interval TI1 between the start point T2' of the integration time of the pixels (PX_G2, PX_R2, PX_B2, PX_G2') having the second phase (Phase 2) and the start point T3' of the integration time of the pixels (PX_G3, PX_R3, PX_B3, PX_G3') having the third phase (Phase 3).

[0210] The start of the integration time for the pixels (PX_G4, PX_R4, PX_B4, PX_G4') having Phase 4 (Phase 4) is time T4', which is after time T3'. That is, the time at which the photocharges contained in the pixels having Phase 4 that are read out next to the pixels having Phase 3 in each subpixel array begin to accumulate is the same as time T4'.

[0211] In this way, since multiple pixels included in the same subpixel array share an analog-to-digital converter, a time difference of time interval TI1 occurs between the start point T3' of the integration time of the pixels (PX_G3, PX_R3, PX_B3, PX_G3') having the third phase (Phase 3) and the start point T4' of the integration time of the pixels (PX_G4, PX_R4, PX_B4, PX_G4') having the fourth phase (Phase 4).

[0212] At this time, the timing generator 220 provides operation timing control signals to the row driver circuit 210 to adjust the start point of the integration time of pixels (PX_G1, PX_R1, PX_B1, PX_G1') having the first phase (Phase 1) to time T1', the start point of the integration time of pixels (PX_G2, PX_R2, PX_B2, PX_G2') having the second phase (Phase 2) to time T2', the start point of the integration time of pixels (PX_G3, PX_R3, PX_B3, PX_G3') having the third phase (Phase 3) to time T3', and the start point of the integration time of pixels (PX_G4, PX_R4, PX_B4, PX_G4') having the fourth phase (Phase 4) to time T4'.

[0213] In addition, the timing generator 220 adjusts the time difference between the start point T1' of the integration time of a pixel having the first phase (Phase 1) and the start point T2' of the integration time of a pixel having the second phase (Phase 2), the time difference between the start point T2' of the integration time of a pixel having the second phase (Phase 2) and the start point T3' of the integration time of a pixel having the third phase (Phase 3), and the time difference between the start point T3' of the integration time of a pixel having the third phase (Phase 3) and the start point T4' of the integration time of a pixel having the fourth phase (Phase 4) so ​​that they are all the same within the time interval TI1.

[0214] In addition, the timing generator 220 adjusts the time difference TI1 between the start points of the integration time of pixels for each phase of the first frame image signal (Frame Image 1) described with reference to Figures 32 and 34 and the time difference TI1 between the start points of the integration time of pixels for each phase of the second frame image signal (Frame Image 2) so that they are the same.

[0215] 21 and 34, after the integration time for the pixels (PX_G4, PX_R4, PX_B4, PX_G4') having the fourth phase that are read out last among the pixels included in the pixel array has ended, at time T5, the image sensor 200 transmits image data IDATA to the image signal processor 100. At this time, the image data IDATA is the first frame image signal (Frame Image1) of FIGS.

[0216] The image signal processor 100 receives image data IDATA and performs image processing on the image data IDATA to generate metadata MD. The image signal processor 100 provides the metadata MD to a timing generator control circuit 300, which generates a control signal CS based on the metadata MD. The timing generator control circuit 300 provides the control signal CS to a timing generator 220, which generates an operation timing control signal based on the control signal CS to control the operation timing of the components included in the logic circuit area LC of the image sensor 200 and provides the operation timing control signal to the components.

[0217] Hereinafter, with reference to FIG. 36, an operation of the timing generator 220 generating an operation timing control signal to control the start point of the integration time for each sub-pixel array for the second frame image signal (Frame Image2) will be described.

[0218] FIG. 36 is a diagram illustrating an operation of controlling a start point of integration time for each sub-pixel array for a second frame image signal of an image sensor, according to an embodiment.

[0219] 36, the timing generator 220 adjusts the time difference between the start point T1′ of the integration time of a pixel having a first phase (Phase 1) and the start point T2′ of the integration time of a pixel having a second phase (Phase 2), the time difference between the start point T2′ of the integration time of a pixel having a second phase (Phase 2) and the start point T3′ of the integration time of a pixel having a third phase (Phase 3), and the time difference between the start point T3′ of the integration time of a pixel having a third phase (Phase 3) and the start point T4′ of the integration time of a pixel having a fourth phase (Phase 4) so ​​that they are all equal to one another at a time interval TI2. In this case, the time interval TI2 is greater than the time interval TI1 in FIG.

[0220] For example, if the metadata MD generated by the image signal processor 100 includes information that the moving speed of the object 400 is not relatively fast, the timing generator control circuit 300 generates a control signal CS that controls the timing generator control circuit 300 to increase the time difference between the start points of the integration times of multiple pixels included in the same sub-pixel array based on the metadata MD including the above information.

[0221] The timing generator 220 receives the control signal CS from the timing generator control circuit 300 and provides an operation timing control signal to the row driver circuit 210, etc., to increase the time interval TI1 between the start points of integration times of pixels included in the same sub-pixel array when generating the first frame image signal (Frame Image1) to a time interval TI2 when generating the second frame image signal (Frame Image2). As a result, the speed at which the image sensor 200 generates the image data IDATA is slower when generating the second frame image signal (Frame Image2) than when generating the first frame image signal (Frame Image1).

[0222] As such, in an embodiment of the present invention, for pixels among a plurality of pixels included in a pixel array that share an analog-to-digital converter, the image sensor is driven using the rolling shutter method, and for pixels that do not share an analog-to-digital converter and have the same readout order within different pixel arrays and therefore have the same phase, the image sensor is driven using the global shutter method.

[0223] In this case, the image signal processor calculates the motion vector of the object by using the rolling shutter method for multiple pixels included in the same subpixel array, which causes a time difference in the readout time. The image signal processor then provides a feedback signal to the timing generator based on the calculated motion vector, and the timing generator adjusts the readout operation timing of the multiple pixels in response to the feedback. As a result, as described with reference to FIG. 36, if the moving speed of the object 400 is not fast, the sensing operation speed of the image sensor 200 can be adjusted to be relatively slow, allowing the image sensor to be operated in a power saving mode. Furthermore, by adjusting the difference in the start point of the integration time between pixels having different phases, the frame rate can be adjusted without shortening the overall length of the integration time (e.g., the integration time length P1 in FIG. 35 and the integration time length P2 in FIG. 36 are the same).

[0224] FIG. 37 is a diagram illustrating an image sensing system according to one embodiment.

[0225] 37, the image sensing system includes an object 400 and an image sensor 200. In this case, the image signal processor described with reference to FIGS. 1 to 36 is included in the image sensor 200. In one embodiment, the image sensing system further includes a distance measurement device 500. The distance measurement device 500 measures the distance between the image sensor 200 and the object 400, and provides information about the measured distance to the image signal processor included in the image sensor 200. The image signal processor extracts characteristic information of the object 400 from the provided distance information, calculates a motion vector of the object 400, and uses the calculation result to generate metadata.

[0226] Fig. 38 is a diagram illustrating an electronic device including a multi-camera module according to an embodiment, and Fig. 39 is a diagram illustrating the camera module of Fig. 38.

[0227] Referring to FIG. 38, the electronic device 1000 includes a camera module group 1100, an application processor 1200, a power management IC (PMIC) 1300, external storage 1400, and a display (not shown).

[0228] The camera module group 1100 includes multiple camera modules (1100a, 1100b, and 1100c). Although the drawings show an embodiment in which three camera modules (1100a, 1100b, and 1100c) are arranged, the embodiment is not limited thereto. In one embodiment, the camera module group 1100 may be modified to include only two camera modules. In another embodiment, the camera module group 1100 may be modified to include n camera modules (n is a natural number greater than or equal to 4).

[0229] The detailed configuration of camera module 1100b will be described in more detail below with reference to FIG. 39, but the following description can also be applied to other camera modules (1100a, 1100c) according to the embodiment.

[0230] Referring to FIG. 39, the camera module 1100b includes a prism 2105, an optical path folding element (hereinafter referred to as “OPFE”) 2110, an actuator 2130, an image sensing device 2140, and a storage unit 2150.

[0231] The prism 2105 includes a reflecting surface 2107 made of a light-reflecting material, and changes the path of light L incident from the outside.

[0232] In one embodiment, the prism 2105 changes the path of light L incident in a first direction X to a second direction Y perpendicular to the first direction X. The prism 2105 also changes the path of light L incident in the first direction X to the perpendicular second direction Y by rotating a reflective surface 2107 made of a light-reflecting material around a central axis 2106 in a direction A or by rotating the central axis 2106 in a direction B. At this time, the OPFE 2110 also moves in a third direction Z perpendicular to the first direction X and the second direction Y.

[0233] In one embodiment, as shown, the maximum rotation angle of prism 2105 in the A direction is less than or equal to 15 degrees in the plus (+) A direction and greater than 15 degrees in the minus (-) A direction, although the embodiment is not limited thereto.

[0234] In other embodiments, prism 2105 may move in the plus (+) or minus (-) B direction by 20 degrees in or out, or between 10 and 20 degrees, or between 15 and 20 degrees, where the angle of movement may be the same angle in the plus (+) or minus (-) B direction, or may move to an approximately equal angle within a range of 1 degree in or out.

[0235] In one embodiment, the prism 2105 moves the reflective surface 2107 of the light-reflecting material in a third direction (eg, Z direction) parallel to the extension direction of the central axis 2106 .

[0236] The OPFE 2110 includes, for example, m (where m is a natural number) groups of optical lenses. The m lenses move in the second direction Y to change the optical zoom ratio of the camera module 1100b. For example, if the basic optical zoom ratio of the camera module 1100b is Z, when the m optical lenses included in the OPFE 2110 are moved, the optical zoom ratio of the camera module 1100b is changed to an optical zoom ratio of 3Z or 5Z or more.

[0237] The actuator 2130 moves the OPFE 2110 or the optical lens (hereinafter referred to as the optical lens) to a specific position. For example, the actuator 2130 adjusts the position of the optical lens so that the image sensor 2142 is located at the focal length of the optical lens for accurate sensing.

[0238] Image sensing device 2140 includes an image sensor 2142, control logic 2144, and memory 2146. Image sensor 2142 senses an image of a sensing target using light L provided through an optical lens. In one embodiment, image sensor 2142 includes image sensor 200 described above.

[0239] The control logic 2144 controls the overall operation of the camera module 1100b, for example, in response to control signals provided via the control signal line CSLb.

[0240] The memory 2146 stores information necessary for the operation of the camera module 1100b, such as correction data 2147. The correction data 2147 includes information necessary for the camera module 1100b to generate image data using light L provided from an external device. The correction data 2147 includes, for example, information regarding the degree of rotation, the focal length, and the optical axis. If the camera module 1100b is implemented in the form of a multi-state camera in which the focal length changes depending on the position of the optical lens, the correction data 2147 includes information related to the focal length value and autofocusing for each position (or state) of the optical lens.

[0241] The storage unit 2150 stores image data sensed through the image sensor 2142. The storage unit 2150 is disposed outside the image sensing device 2140 and is implemented in a stacked form on a sensor chip constituting the image sensing device 2140. In one embodiment, the storage unit 2150 is implemented as an EEPROM (Electrically Erasable Programmable Read-Only Memory), but the embodiment is not limited thereto. The storage unit 2150 may be implemented as a chip.

[0242] 38 and 39, in one embodiment, each of the multiple camera modules (1100a, 1100b, 1100c) includes an actuator 2130. As a result, each of the multiple camera modules (1100a, 1100b, 1100c) includes correction data 2147 that may be the same as or different from one another due to the operation of the actuator 2130 included therein.

[0243] In one embodiment, one of the multiple camera modules (1100a, 1100b, 1100c) (e.g., 1100b) is a folded lens type camera module including the above-mentioned prism 2105 and OPFE 2110, and the remaining camera modules (e.g., 1100a, 1100c) are vertical type camera modules that do not include the prism 2105 and OPFE 2110, but the embodiment is not limited thereto.

[0244] In one embodiment, one of the camera modules 1100a, 1100b, and 1100c (e.g., 1100c) is a vertical depth camera that extracts depth information using, for example, infrared rays (IR). In this case, the application processor 1200 merges image data provided from the vertical depth camera with image data provided from a different camera module (e.g., 1100a or 1100b) to generate a 3D depth image.

[0245] In one embodiment, at least two camera modules (e.g., 1100a, 1100c) of the plurality of camera modules (1100a, 1100b, 1100c) have different fields of view (fields of view) from each other, for example, but not limited to, the optical lenses of at least two camera modules (e.g., 1100a, 1100c) of the plurality of camera modules (1100a, 1100b, 1100c) are different from each other.

[0246] In one embodiment, the camera modules (1100a, 1100b, 1100c) each have a different viewing angle, and in this case, the optical lenses included in the camera modules (1100a, 1100b, 1100c) are also different, but this is not limiting.

[0247] In one embodiment, the multiple camera modules (1100a, 1100b, 1100c) are physically separated from one another. That is, the multiple camera modules (1100a, 1100b, 1100c) do not share the sensing area of ​​a single image sensor 2142, but an independent image sensor 2142 is disposed within each of the multiple camera modules (1100a, 1100b, 1100c).

[0248] 38 again, the application processor 1200 includes an image processing unit 1210, a memory controller 1220, and an internal memory 1230. The application processor 1200 is implemented separately from the multiple camera modules (1100a, 1100b, 1100c). For example, the application processor 1200 and the multiple camera modules (1100a, 1100b, 1100c) are implemented separately from each other as separate semiconductor chips.

[0249] The image processing device 1210 includes a number of sub-image processors (1212a, 1212b, 1212c), an image generator 1214, and a camera module controller 1216.

[0250] The image processing device 1210 includes a plurality of sub-image processors (1212a, 1212b, 1212c) whose number corresponds to the number of the camera modules (1100a, 1100b, 1100c).

[0251] Image data generated from each camera module (1100a, 1100b, 1100c) is provided to the corresponding sub-image processor (1212a, 1212b, 1212c) via separate image signal lines (ISLa, ISLb, ISLc). For example, image data generated from camera module 1100a is provided to sub-image processor 1212a via image signal line ISLa, image data generated from camera module 1100b is provided to sub-image processor 1212b via image signal line ISLb, and image data generated from camera module 1100c is provided to sub-image processor 1212c via image signal line ISLc. Such image data transmission is performed using, for example, a camera serial interface (CSI) based on MIPI (Mobile Industry Processor Interface), but the embodiment is not limited thereto.

[0252] In other embodiments, one sub-image processor may be arranged to correspond to multiple camera modules. For example, sub-image processor 1212a and sub-image processor 1212c may be implemented as a single integrated sub-image processor, rather than being implemented separately as shown, and image data provided from camera module 1100a and camera module 1100c may be selected via a selection element (e.g., a multiplexer) and then provided to the integrated sub-image processor.

[0253] The image data provided to each of the sub-image processors (1212a, 1212b, 1212c) is provided to an image generator 1214. The image generator 1214 generates an output image using the image data provided from each of the sub-image processors (1212a, 1212b, 1212c) according to image generating information or a mode signal.

[0254] Specifically, the image generator 1214 generates an output image by merging at least a portion of the image data generated from the camera modules 1100a, 1100b, and 1100c, each having a different viewing angle, in response to the image generation information or mode signal. The image generator 1214 may also generate an output image by selecting one of the image data generated from the camera modules 1100a, 1100b, and 1100c, each having a different viewing angle, in response to the image generation information or mode signal.

[0255] In one embodiment, the image generation information includes a zoom signal or zoom factor, and in one embodiment, the mode signal is based on a mode selected by, for example, a user.

[0256] If the image generation information is a zoom signal (zoom factor) and each camera module (1100a, 1100b, 1100c) has a different field of view (viewing angle), the image generator 1214 performs different operations depending on the type of zoom signal. For example, if the zoom signal is a first signal, the image generator 1214 merges the image data output from camera module 1100a and the image data output from camera module 1100c, and then generates an output image using the merged image signal and the image data output from camera module 1100b that was not used in the merging. If the zoom signal is a second signal different from the first signal, the image generator 1214 does not merge the image data, but instead selects one of the image data output from each camera module (1100a, 1100b, 1100c) to generate an output image. However, the embodiment is not limited to this, and various modifications can be made to the method of processing image data as needed.

[0257] In one embodiment, the image generator 1214 receives multiple image data with different exposure times from at least one of the multiple sub-image processors (1212a, 1212b, 1212c) and performs HDR (high dynamic range) processing on the multiple image data to generate merged image data with an increased dynamic range.

[0258] The camera module controller 1216 provides control signals to each of the camera modules (1100a, 1100b, 1100c). The control signals generated by the camera module controller 1216 are provided to the corresponding camera modules (1100a, 1100b, 1100c) via separate control signal lines (CSLa, CSLb, CSLc).

[0259] One of the multiple camera modules (1100a, 1100b, 1100c) can be designated as a master camera (e.g., 1100a) in response to image generation information including a zoom signal or a mode signal, and the remaining camera modules (e.g., 1100b and 1100c) can be designated as slave cameras. Such information is included in a control signal and provided to the corresponding camera modules (1100a, 1100b, 1100c) via separate control signal lines (CSLa, CSLb, CSLc).

[0260] The camera modules operating as the master and slave are changed depending on the zoom factor or the operation mode signal. For example, when the viewing angle of camera module 1100a is wider than that of camera module 1100c and the zoom factor indicates a low zoom ratio, camera module 1100c operates as the master and camera module 1100a operates as the slave. Conversely, when the zoom factor indicates a high zoom ratio, camera module 1100a operates as the master and camera module 1100c operates as the slave.

[0261] In one embodiment, the control signals provided from the camera module controller 1216 to each of the camera modules 1100a, 1100b, and 1100c include a sync enable signal. For example, if the camera module 1100b is the master camera and the camera modules 1100a and 1100c are slave cameras, the camera module controller 1216 transmits the sync enable signal to the camera module 1100b. Upon receiving the sync enable signal, the camera module 1100b generates a sync signal based on the received sync enable signal and provides the generated sync signal to the camera modules 1100a and 1100c via a sync signal line SSL. The camera modules 1100b and the camera modules 1100a and 1100c transmit image data to the application processor 2200 in synchronization with the sync signal.

[0262] In one embodiment, the control signals provided by the camera module controller 1216 to the camera modules (1100a, 1100b, 1100c) include mode information via a mode signal, and based on such mode information, the camera modules (1100a, 1100b, 1100c) operate in a first operating mode or a second operating mode associated with a sensing speed.

[0263] In a first operating mode, the camera modules (1100a, 1100b, 1100c) may generate image signals at a first rate (e.g., generate image signals at a first frame rate), encode the image signals at a second rate higher than the first rate (e.g., encode image signals at a second frame rate higher than the first frame rate), and transmit the encoded image signals to the application processor 1200. In this case, the second rate is 30 times or less than the first rate.

[0264] The application processor 1200 may store the received image signal, i.e., the encoded image signal, in the internal memory 1230 or the external storage 1400 of the application processor 1200, and then read and decode the encoded image signal from the memory 1230 or the storage 1400, and display image data generated based on the decoded image signal. For example, a corresponding sub-processor among the multiple sub-processors (1212a, 1212b, 1212c) of the image processing device 1210 performs decoding and image processing on the decoded image signal. For example, the image data generated based on the decoded image signal is displayed on a display.

[0265] In the second operating mode, the multiple camera modules (1100a, 1100b, 1100c) may generate image signals at a third rate lower than the first rate (e.g., generate image signals at a third frame rate lower than the first frame rate) and transmit the image signals to the application processor 1200. The image signals provided to the application processor 1200 are unencoded signals. The application processor 1200 performs image processing on the received image signals or stores the image signals in the memory 1230 or the storage 1400.

[0266] The PMIC 1300 supplies power, such as a power supply voltage, to each of the multiple camera modules (1100a, 1100b, and 1100c). For example, under the control of the application processor 1200, the PMIC 1300 supplies a first power to the camera module 1100a via a power signal line PSLa, a second power to the camera module 1100b via a power signal line PSLb, and a third power to the camera module 1100c via a power signal line PSLc.

[0267] The PMIC 1300 generates power and adjusts the power level corresponding to each of the camera modules (1100a, 1100b, 1100c) in response to a power control signal PCON from the application processor 1200. The power control signal PCON includes a power adjustment signal for each operation mode of the camera modules (1100a, 1100b, 1100c). For example, the operation mode may include a low power mode. In this case, the power control signal PCON includes information about the camera module operating in the low power mode and the power level to be set. The power levels provided to each of the camera modules (1100a, 1100b, 1100c) may be the same or different from each other. Furthermore, the power levels may be dynamically changed.

[0268] Although the embodiments of the present invention have been described in detail above with reference to the drawings, the present invention is not limited to the above-described embodiments and can be modified in various ways without departing from the technical concept of the present invention. [Explanation of symbols]

[0269] 2 Upper planarization film 3a, 3b Color filters 4 Side anti-reflection coating 5 Lower planarization film 6a, 6b board 7 Anti-reflection coating 8a, 8b Photoelectric transistor 100 Image Signal Processor 200 image sensors 210 Row driver circuit 220 Timing Generator 230 Ramp signal generating circuit 240 Readout Circuit 250, 250_1 to 250_n Analog-to-Digital Converter (ADC) 260 Buffer Circuit 270 pixel array 280 Column driver circuit 300 Timing generator control circuit 400 objects 500 Distance Measuring Device 1000 Image Sensing System 1100 Camera Module Group 1100a, 1100b, 1100c camera modules 1200 Application Processor 1210 Image Processing Device 1212a, 1212b, 1212c Sub-Image Processors 1213 Multiplexer (Mux) 1214 Image Generator 1216 Camera Module Controller 1220 memory controller 1230 internal memory 1300 Power Management IC (PMIC) 1400 External Storage 2105 Prism 2106 Center axis 2107 Reflective surface 2110 Optical Path Bending Element (OPFE) 2130 Actuator 2140 Image Sensing Device 2142 Image Sensor 2144 Control Logic 2146 memory 2147 correction data 2150 Preservation Department B1~B4, B1a~B4a, B1b~B4b, B1c~B4c Blue pixel values COL1~COL(n) column lines CS control signal CSLa, CSLb, CSLc control signal lines FD Floating diffusion area G1~G4, G1'~G4', G1a~G4a, G1b~G4b, G1c~G4c, G1'a~G4'a, G1'b~G4'b, G1'c~G4'c Green pixel value IS1a~IS1d 1st image signal IS2a~IS2d Second image signals ISLa, ISLb, ISLc image signal lines LC logic circuit area LS lens MD Metadata P1, P2 integration time length PA Pixel Array PCON Power Control Signal PD photodiode PH1 First Peripheral Area PSLa, PSLb, PSLc power signal lines PX pixels PX_B1~PX_B4 Blue pixels PX_G1~PX_G4, PX_G1'~PX_G4' green pixels PX_R1~PX_R4 Red pixels R1~R4, R1a~R4a, R1b~R4b, R1c~R4c Red pixel values ROW1~ROW(m) Row line RS Reset control signal RX reset transistor S1, S2 1st, 2nd area SA1 to SA4 1st to 4th sub-pixel arrays SEL Selection control signal SF Source Follower SX select transistor Time points T1 to T4 TS transmission control signal TX Transfer Transistor VDD power supply voltage Vout output signal

Claims

1. a pixel array having a first sub-pixel array including a first pixel and a second pixel and a second sub-pixel array including a third pixel and a fourth pixel; a first column line connected to the first pixel and the second pixel; a second column line connected to the third pixel and the fourth pixel and different from the first column line; a readout circuit connected to the first and second column lines to receive output signals from the first to fourth pixels and to output image data based on the output signals; an image signal processor that performs interpolation based on the image data to generate metadata related to motion vectors of objects included in the image data; The readout circuit reading out the first pixel and the second pixel in order from the pixels connected to the first column line; reading out the third pixel and the fourth pixel in order from the pixels connected to the second column line; The start points of the integration times for the first pixel and the third pixel are the same; The image sensing system, wherein the start points of integration times for the second pixel and the fourth pixel are the same.

2. the image signal processor divides the image data into identical phases to generate first and second image signals; the first image signal includes a first pixel value output by sensing light transmitted through a first color filter corresponding to the first pixel, and a second pixel value output by sensing light transmitted through a second color filter corresponding to the third pixel; 2. The image sensing system of claim 1, wherein the second image signal includes a third pixel value output by sensing light transmitted through a third color filter corresponding to the second pixel, and a fourth pixel value output by sensing light transmitted through a fourth color filter corresponding to the fourth pixel.

3. the image signal processor upscales the first image signal to generate a third image signal, and upscales the second image signal to generate a fourth image signal; the third image signal includes a fifth pixel value between the first pixel value and the second pixel value; the fifth pixel value corresponds to the second pixel; the fourth image signal includes a sixth pixel value between the third pixel value and the fourth pixel value; The image sensing system of claim 2 , wherein the sixth pixel value corresponds to the third pixel value.

4. 4. The image sensing system of claim 3, wherein the image signal processor compares the third image signal with the fourth image signal to extract characteristic information of the object, and calculates a motion vector of the object based on the extracted characteristic information.

5. The image sensing system of claim 4 , wherein the metadata about the object's motion vector includes at least one of information about the object's velocity, acceleration, and path of movement.

6. the image signal processor performs phase correction on the first image signal to generate a third image signal, and performs phase correction on the second image signal to generate a fourth image signal; a phase of the first pixel in the third image signal and a phase of the second pixel in the fourth image signal are the same; 3. The image sensing system of claim 2, wherein the phase of the third pixel in the third image signal and the phase of the fourth pixel in the fourth image signal are the same.

7. 7. The image sensing system of claim 6, wherein the image signal processor compares the third image signal with the fourth image signal to extract characteristic information of the object, and calculates a motion vector of the object based on the extracted characteristic information.

8. The image sensing system of claim 7 , wherein the metadata about the object's motion vector includes at least one of information about the object's velocity, acceleration, and path of movement.

9. The readout circuit a first analog-to-digital converter connected to the first column line and configured to convert a signal on the first column line into a first digital signal; 2. The image sensing system of claim 1, further comprising: a second analog-to-digital converter connected to the second column line to convert a signal on the second column line into a second digital signal.

10. a timing generator that generates an operation timing reference signal for the pixel array; 2. The image sensing system of claim 1, further comprising: a timing generator control circuit that receives the metadata from the image signal processor and generates a control signal to control the timing generator based on the metadata.

11. 11. The image sensing system of claim 10, wherein the timing generator receives the control signal from the timing generator control circuit and adjusts an interval between a start point of an integration time for the first pixel and the third pixel and a start point of an integration time for the second pixel and the fourth pixel based on the received control signal.

12. 12. The image sensing system of claim 11, wherein the timing generator adjusts, based on the control signal, an interval between a start point of an integration time for the first pixel and the third pixel and a start point of an integration time for the second pixel and the fourth pixel so as to increase.

13. an image sensor configured to image an object to generate image data; an image signal processor that receives the image data from the image sensor, performs image processing, and generates metadata related to the motion vectors of the object based on the results of the image processing; a timing generator control circuit that receives the metadata from the image signal processor and generates a control signal for controlling a timing generator based on the metadata; The image sensor includes: a pixel array including a plurality of pixels; a row driver circuit; at least one row line connected to the row driver circuit and extending in a first direction; a first pixel, a second pixel, a third pixel, and a fourth pixel connected to the at least one row line; a first column line connected to the first pixel and the third pixel and extending in a second direction intersecting the first direction; a second column line connected to the second pixel and the fourth pixel and extending in the second direction, the second column line being different from the first column line; a readout circuit connected to the first column line and the second column line to receive output signals from the first to fourth pixels and to output the image data based on the output signals; a timing generator configured to transmit an operation timing reference signal to the row driver circuit based on the control signal received from the timing generator control circuit; the image signal processor performs image processing on the image data based on a readout order between the first pixel and the third pixel and a readout order between the second pixel and the fourth pixel.

14. the first column line receives at least one of a first output signal from the first pixel and a second output signal from the third pixel; 14. The image sensing system of claim 13, wherein the readout circuit includes a first analog-to-digital converter connected to the first column line, receiving at least one of the first output signal and the second output signal, and converting the received at least one of the first output signal and the second output signal into a first digital signal.

15. the second column line receives at least one of a third output signal from the second pixel and a fourth output signal from the fourth pixel; 15. The image sensing system of claim 14, wherein the readout circuit further comprises a second analog-to-digital converter connected to the second column line, receiving at least one of the third output signal and the fourth output signal, and converting the received at least one of the third output signal and the fourth output signal into a second digital signal.

16. The readout circuit reading out the pixels connected to the first column line in the order of the first pixel and the third pixel; reading out the second pixel and the fourth pixel in order from the pixels connected to the second column line; The start points of the integration times for the first pixel and the third pixel are the same; The image sensing system of claim 13 , wherein the start points of the integration times for the second pixel and the fourth pixel are the same.

17. 1. A method of operating an image sensing system comprising an image signal processor, comprising: sensing an image of an object by an image sensor to generate image data; receiving the image data and performing image processing by the image signal processor; generating, by the image signal processor, metadata relating to the motion vectors of the object based on the results of the image processing; receiving the metadata by a timing generator control circuit and generating a control signal for controlling a timing generator based on the received metadata; and adjusting, by the timing generator, a start point of an integration time of a pixel included in the image sensor based on the control signal; adjusting the start point of integration time of the pixels by the timing generator includes expanding, by the timing generator, an interval between the first point in time and the second point in time for the first pixel and the third pixel, among first, second, third, and fourth pixels included in the image sensor, whose start points of integration time are the same at a first point in time, and the second pixel and the fourth pixel, among the first to fourth pixels, whose start points of integration time are the same at a second point in time after the first point in time, based on the control signal; the first pixel and the second pixel are connected to a first column line; The third pixel and the fourth pixel are connected to a second column line different from the first column line.

18. The image processing step by the image signal processor includes dividing the image data into identical phases to generate first and second image signals; the first image signal includes a first pixel value output by sensing light transmitted through a first color filter corresponding to the first pixel, and a second pixel value output by sensing light transmitted through a second color filter corresponding to the third pixel; 18. The method of claim 17, wherein the second image signal includes a third pixel value output by sensing light transmitted through a third color filter corresponding to the second pixel, and a fourth pixel value output by sensing light transmitted through a fourth color filter corresponding to the fourth pixel.

19. generating the metadata by the image signal processor, upscaling the first image signal to generate a third image signal; upscaling the second image signal to generate a fourth image signal; comparing the third image signal with the fourth image signal to extract characteristic information of the object; and calculating a motion vector of the object based on the extracted characteristic information; the third image signal includes a fifth pixel value between the first pixel value and the second pixel value; the fifth pixel value corresponds to the second pixel; the fourth image signal includes a sixth pixel value between the third pixel value and the fourth pixel value; 20. The method of claim 18, wherein the sixth pixel value corresponds to the third pixel value.

20. generating the metadata by the image signal processor, performing a phase correction on the first image signal to generate a third image signal; performing a phase correction on the second image signal to generate a fourth image signal; comparing the third image signal with the fourth image signal to extract characteristic information of the object; and calculating a motion vector of the object based on the extracted characteristic information; a phase of the first pixel in the third image signal and a phase of the second pixel in the fourth image signal are the same; 20. The method of claim 18, wherein a phase of the third pixel in the third image signal and a phase of the fourth pixel in the fourth image signal are the same.

Images