Image sensor, device including the same, and operating method for image sensor

Abstract

To provide an image sensor, a device including the image sensor, and a method for operating the image sensor.SOLUTION: An image sensor according to the present invention is an image sensor that operates in a global shutter mode and includes a pixel array including a plurality of pixels, a timing controller configured to control the period of an output signal corresponding to N pieces of frame data output from the pixels, and (N-1) frame buffers configured to save (N-1) pieces of frame data.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to image sensors, devices including the same, and methods of operating image sensors, and more particularly to image sensors including frame buffers. [Background technology]

[0002] An image sensor is a device that captures two-dimensional or three-dimensional images of an object. Image sensors generate images of an object using photoelectric conversion elements that react to the intensity of light reflected from the object. Recent developments in the computer and communications industries have led to an increasing demand for image sensors with improved performance in a variety of electronic devices, including digital cameras, camcorders, PCS (Personal Communication Systems), game consoles, security cameras, medical microcameras, and mobile phones.

[0003] An image sensor can generate high- or low-light image data by storing photocharges corresponding to multiple exposure times to create a High Dynamic Range (HDR) image. To generate an HDR image, a frame buffer is required to store multiple frames of data. Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide an image sensor that can reduce the amount of frame buffer usage. [Means for solving the problem]

[0005] To solve the above technical problems, an image sensor according to the technical idea of ​​the present invention is disclosed.

[0006] The image sensor operates in a global shutter mode and includes a pixel array including a plurality of pixels, a timing controller configured to control periods of output signals corresponding to N frame data output from the plurality of pixels, and N-1 frame buffers configured to store N-1 frame data, respectively.

[0007] To solve the above technical problems, an imaging device according to the technical idea of ​​the present invention is disclosed.

[0008] The image device includes an image sensor configured to read out N consecutive frame data, and an image signal processor configured to merge and process the N frame data, wherein the image sensor is configured to operate in a global shutter mode, and the image sensor can control the readout speed of at least one frame data among the N frame data to be different from each other.

[0009] To solve the above technical problems, a method for operating an image sensor according to the technical idea of ​​the present invention is disclosed.

[0010] The method for operating the image sensor includes the steps of outputting N frame data from the image sensor, storing N-1 frame data of the N frame data in a frame buffer, and storing a portion of the frame data that is not stored in the frame buffer in a line buffer. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a schematic configuration of an image sensor according to an embodiment of the present invention. [Figure 2] FIG. 10 is a block diagram illustrating an image sensor according to another embodiment. [Figure 3]1 is a diagram illustrating a global shutter mode operation of an image sensor according to an exemplary embodiment of the present invention; [Figure 4A] 10 is a diagram for explaining ISP processing of multiple frames according to a comparative example; [Figure 4B] 1 is a diagram illustrating ISP processing of multiple frames according to an embodiment of the present invention. [Figure 5] 2 is a diagram illustrating an operation method of an image sensor according to an embodiment of the present invention; [Figure 6] 3 is a flowchart illustrating a method of operating an image sensor according to an embodiment of the present invention. [Figure 7] 3 is a flowchart illustrating a method of operating an image sensor according to an embodiment of the present invention. [Figure 8A] 10 is a diagram showing ADC output data, readout data from a frame buffer, and ISP output data according to a comparative example. [Figure 8B] 1 is a diagram illustrating ADC output data, readout data from a frame buffer, and ISP output data according to an embodiment of the present invention. [Figure 9] 1 is a diagram illustrating signals corresponding to ADC output data and signals for reading out the same according to an embodiment of the present invention. [Figure 10] 1 is a diagram illustrating signals corresponding to ADC output data and signals for reading out the same according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0012] Various embodiments of the present invention will now be described with reference to the accompanying drawings.

[0013] FIG. 1 is a block diagram showing a schematic configuration of an image sensor according to an embodiment of the present invention.

[0014] The pixel PX shown in FIG. 1 may be a digital pixel that performs a global shutter function.

[0015] The image sensor 10 may be mounted in an electronic device having an image or light sensing function. For example, the image sensor 10 may be mounted in electronic devices such as a camera, a smartphone, a wearable device, an Internet of Things (IoT), a tablet PC (Personal Computer), a PDA (Personal Digital Assistant), a PMP (Portable Multimedia Player), and a navigation device. The image sensor 10 may also be mounted in electronic devices provided as components in vehicles, furniture, manufacturing equipment, doors, various measuring instruments, and the like.

[0016] The image sensor 10 includes a pixel array 100, a pixel driver (row driver) 200, a ramp signal generator 300, a timing controller 400, a digital signal processing unit 500, and an interface circuit 600. The pixel array 100 includes a plurality of pixels PX, each of which is configured to sense an external optical signal and output a digital output signal DOUT corresponding to the sensed optical signal.

[0017] The pixels PX can sense optical signals using light-sensing elements and convert the sensed optical signals into digital output signals DOUT, which are electrical signals. Each of the pixels PX can sense light in a specific spectral range. For example, the pixels PX include a red pixel that converts light in the red spectral range into an electrical signal, a green pixel that converts light in the green spectral range into an electrical signal, and a blue pixel that converts light in the blue spectral range into an electrical signal. A color filter that transmits light in the specific spectral range and a microlens for focusing the light may be arranged on top of each of the pixels PX.

[0018] Each pixel PX includes a photodetector circuit 110, an analog-to-digital converter (ADC) 120, and a memory 130. In some embodiments, the pixel array 100 includes a first semiconductor substrate and a second semiconductor substrate that are stacked together. For example, the photodetector circuit 110 of each pixel PX is formed on the first semiconductor substrate, and the ADC 120, which converts signals generated by the photodetector circuit 110 into digital signals, is formed on a second semiconductor substrate different from the first semiconductor substrate. The photodetector circuits formed on the first semiconductor substrate are connected to the ADCs formed on the second semiconductor substrate via electrical connection means to transmit the detection signals. In some embodiments, the electrical connection means includes through-silicon vias (TSVs) that penetrate the first semiconductor substrate and / or metal-to-metal connection structures formed on the first and second semiconductor substrates, respectively. In some embodiments, the memory 130 is formed on the second semiconductor substrate. The photodetector circuit 110 includes a light-sensing element and can convert an externally sensed optical signal into an electrical signal, i.e., an analog detection signal. For example, the light-sensing element may include a photodiode, a phototransistor, a port gate, a pinned photodiode, etc. The detection signal includes a detection signal due to a reset operation of the pixel PX and a detection signal due to a light-detection operation of the pixel PX.

[0019] The ADC 120 can convert the detection signal output from the photodetection circuit 110 into a digital signal. In some embodiments, the ADC 120 converts the detection signal into a digital signal by comparing the detection signal with a ramp signal RAMP. The memory 130 can store the converted digital signal. The memory 130 can output a digital output signal DOUT to the digital signal processing unit 500.

[0020] The pixel driver 200 can output a control signal CTRL for controlling a plurality of pixels PX included in the pixel array 100. In response to the control signal CTRL output from the pixel driver 200, each of the plurality of pixels PX can generate a detection signal, convert the detection signal into a digital signal using a ramp signal RAMP, store the digital signal, and output the stored digital signal as a digital output signal DOUT.

[0021] The ramp signal generator 300 can generate and output a ramp signal RAMP to the pixel array 100. The ramp signal RAMP is provided to the pixel array 100 (e.g., the ADC 120 in the pixel PX) and used as a reference signal to be compared with the detected signal. In an exemplary embodiment, the ramp signal RAMP is a steadily decreasing or increasing signal (e.g., an increasing / decreasing signal with a single and / or linear slope).

[0022] The timing controller 400 can control the overall operation of the image sensor 10. For example, the timing controller 400 can control the operation timing of the image sensor 10 based on control information received from an external device (e.g., an image signal processor (ISP), an application processor (AP), etc.) via the interface circuit 600. The pixel driver 200 and the ramp signal generator 300 can generate a control signal CTRL and a ramp signal RAMP based on the timing signal provided from the timing controller 400.

[0023] According to an example, the digital output signal DOUT output from the pixel array 100 is output as frame data corresponding to the number set in the image sensor 10. According to an example, when the number set in the image sensor 10 is N, the pixel array 100 can output N pieces of frame data. The timing controller 400 according to the present invention can control the output speed of the frame data output from the pixel array 100. The timing controller 400 can control the output speed of the last frame data among the N pieces of frame data output from the pixel array 100 to be slower than the output speeds of the other frame data. A more specific operation of the timing controller 400 will be described later.

[0024] The digital signal processor 500 performs digital signal processing on the digital output signal DOUT received from the pixel array 100 and can provide final image data ID to an external device. The digital output signal DOUT includes a reset value resulting from the reset operation of the pixel PX and an image signal value resulting from the light detection operation of the pixel PX.

[0025] The digital signal processor 500 performs an operation on the reset value and the image signal value to determine a final digital value corresponding to the optical signal sensed at one pixel PX. The final digital values ​​determined at each of the multiple pixels PX are combined to generate final image data ID. That is, a correlated double sampling operation is provided through the digital output signal DOUT generated by the digital signal processing operation of the digital signal processor 500 and the operation of the ADC 120 included in the pixel PX.

[0026] Digital signal processing unit 500 may include frame buffers 501. According to one example, the number of frame buffers 501 included in digital signal processing unit 500 is different from the number of frame data processed by image sensor 10. When the number of frame data processed by image sensor 10 is N, the number of frame buffers 501 included in digital signal processing unit 500 is N-1, where N is a natural number greater than or equal to 2.

[0027] The digital signal processing unit 500 may include a line buffer 502. The size of the line buffer 502 is smaller than that of the frame buffer 501. The line buffer 502 can store line data corresponding to some lines of the last frame data among multiple frame data.

[0028] According to the present invention, in order to reduce the buffer size required for frame buffer processing, the image data of the last frame can be output to have a variable 1h size. According to the present invention, by reducing the time required to store the Nth frame in the buffer, the FPS (frames per second) can be improved.

[0029] The interface circuit 600 is configured to receive control information from an external device or to output a final image data ID. In an exemplary embodiment, the interface circuit 600 can exchange such information with the external device based on a predetermined protocol.

[0030] FIG. 2 is a block diagram illustrating an image sensor according to another embodiment.

[0031] The pixel PX' shown in Figure 2 is a digital pixel capable of global shutter operation. Compared to the image sensor 10 in Figure 1, the image sensor 10' in Figure 2 does not include an ADC 510 inside each pixel PX', but includes an ADC 510 outside the pixel array 100'. In the description of Figure 2, duplicated descriptions of the same reference numerals as in Figure 1 will be omitted.

[0032] 2, the image sensor 10′ includes a pixel array 100′, a pixel driver 200′, a ramp signal generator 300′, a timing controller 400′, a readout circuit 500′, and an interface circuit 600′. The pixel array 100′ includes a plurality of pixels PX′, each of which is configured to sense an external optical signal and output a pixel signal PXS corresponding to the sensed optical signal.

[0033] In the pixel array 100′, a plurality of pixels PX′ are arranged in a matrix of a plurality of rows and a plurality of columns. The image sensor 10′ can control the photocharge accumulation timing of pixels PX′ arranged in different rows to be the same in a global shutter mode, thereby eliminating image distortion due to differences in the photocharge accumulation timing.

[0034] The pixel PX' may include a photodetection circuit 110' and a pixel signal generation circuit 120'. The photodetection circuit 110' includes a light-sensing element and can convert an externally sensed light signal into an electrical signal, i.e., an analog detection signal. The detection signal includes a detection signal generated by a reset operation of the photodetection circuit 110' and a detection signal generated by a photodetection operation of the photodetection circuit 110'.

[0035] The pixel signal generating circuit 120' can receive the detection signal, generate a pixel signal PXS corresponding to the detection signal, and output the pixel signal PXS via the column line.

[0036] The pixel driver 200′ can output a control signal CTRL′ for controlling a plurality of pixels PX′ included in the pixel array 100′. In response to the control signal CTRL′ output from the pixel driver 200′, each of the plurality of pixels PX′ can operate in a plurality of operation modes according to illumination. In an exemplary embodiment, the pixel driver 200′ can determine when the control signal CTRL′ output to each of the plurality of pixels PX′ is activated or deactivated to operate in a global shutter mode.

[0037] The ramp signal generator 300′ generates a ramp signal RAMP′ and can provide the ramp signal RAMP′ to a readout circuit 500′, such as an ADC 510. The ramp signal RAMP′ is a signal for converting an analog signal into a digital signal, and is generated to have a triangular wave form.

[0038] The readout circuit 500' may include an ADC 510, a memory 520, a frame buffer 501', and a line buffer 502'. The ADC 510 samples and holds the pixel signal PXS provided by the pixel array 100', and performs correlated double sampling, double sampling the reset signal and the image signal, and outputting a level corresponding to the difference between the samples. The ADC 510 is provided with a ramp signal RAMP', compares the reset signal and the image signal with the ramp signal RAMP', and outputs a comparison result signal.

[0039] The ADC 510 can convert the comparison result signal into a digital signal. The memory 520 can latch the digital signal and sequentially output the latched image data ID. The frame buffer 501' can store N-1 frame data out of N frame data, and the line buffer 502' can store line data, which is a portion of the frame data that is not stored in the frame buffer out of the N frame data.

[0040] FIG. 3 is a diagram illustrating the operation of a global shutter mode of an image sensor according to an exemplary embodiment of the present invention.

[0041] 1 and 2, the image sensor 100 can operate in a global shutter mode.

[0042] One frame period FP includes a first period P1 and a second period P2, during which a reset operation, an exposure operation, and a global signal dumping operation are simultaneously performed on multiple pixels PX of the pixel array 110, i.e., multiple rows (e.g., from the first row R1 to the nth row Rn) of the pixel array 100, and a readout operation is sequentially performed on the multiple rows of the pixel array 110 in the second period P2. The second period P2 is referred to as a frame readout period.

[0043] The first period P1 includes a reset period, an integration period, and a global signal dumping period (GSDP). During the reset period, the pixels PX perform a reset operation to remove charges accumulated in the photodiodes (and floating diffusion nodes), and during the integration period, they perform an integration operation to generate and accumulate photocharges corresponding to optical signals received by the photodiodes. During the global signal dumping period (GSDP), the pixels PX can store, in at least two capacitors arranged therein, a reset signal according to a reset level of the floating diffusion node and an image signal corresponding to the photocharges accumulated in the photodiodes, respectively.

[0044] During the second period P2, a rolling readout operation is performed in which the readout operation performed during the readout period is performed sequentially by row. For example, after the readout operation is performed on the first row R1 in the pixel array 100, the readout operation is performed on the next row, the second row R2. Then, after the readout operation is performed on the second row R2, the readout operation is performed on the next row, the third row R3. During the readout operation, the reset signal and the image signal stored in at least two capacitors are output from each pixel PX as a pixel signal during a global signal dumping period (GSDP).

[0045] FIG. 4A is a diagram illustrating ISP processing of multiple frames according to a comparative example.

[0046] Referring to FIG. 4A, areas corresponding to the first frame F1, the second frame F2, the third frame F3, and the fourth frame F4 are disclosed, and a diagram of performing image signal processing using the first frame F1 to the fourth frame F4 is shown.

[0047] 4A, the length of the first frame F1 in the first direction 12 is the line length. The length of the first frame F1 in the first direction 12 corresponds to the total number of lines included in the first frame F1. The length of the first frame F1 in the second direction 14 is the frame length.

[0048] Referring to FIG. 4A, the first frame F1 is a first transform region 1 st A / D, horizontal blank area H-Blk, and first lead-out area 1 st According to one example, the first transformation region 1 st The A / D is an area corresponding to a section for converting data corresponding to the first frame F1 from analog to digital. stR / O is an area corresponding to the section for reading out the data corresponding to the first frame F1 that is being converted to digital. The horizontal blank area H-Blk is an area corresponding to the section for preparing for analog-to-digital conversion of the next frame line.

[0049] The second frame F2 is the second transformation region 2 nd A / D, horizontal blank area H-Blk, and second lead-out area 2 nd The third frame F3 includes the third transformation region 3 rd A / D, horizontal blank area H-blk, and third lead-out area 3 rd The fourth frame F4 includes the fourth transformation region 4 th A / D, horizontal blank area H-blk, and fourth lead-out area 4 th The second to fourth conversion areas, horizontal blank area, and second to fourth lead-out areas included in the second frame F2 to fourth frame F4 have the same configuration as the first conversion area, horizontal blank area, and first lead-out area included in the first frame F1, so a duplicated description will be omitted.

[0050] 4A shows an example in which ISP processing is performed after the analog-to-digital conversion of the first frame F1, the second frame F2, the third frame F3, and the fourth frame F4 is completed and the readout of the converted data is completed. Thus, the total frame length is calculated by adding the length of the frame on which ISP processing is performed to the frame length corresponding to the four frames.

[0051] To perform analog-to-digital conversion of the first frame F1, the second frame F2, the third frame F3, and the fourth frame F4, four frame buffers are required to store the first frame F1, the second frame F2, the third frame F3, and the fourth frame F4, respectively. Referring to FIG. 4A, in an image sensor including digital pixels with multiple analog-to-digital converters, all frames must be output from the analog-to-digital converters to process each frame data. Furthermore, when processing multiple frames, the A / D intervals are different from each other, so a corresponding number of frame buffers are required. Using frame buffers requires significant cost in terms of digital logic, and increasing the number of frame buffers increases the chip size, raising costs and power consumption. Therefore, a readout method for an image sensor that can minimize the number of frame buffers used is needed.

[0052] FIG. 4B is a diagram illustrating ISP processing of multiple frames according to an embodiment of the present invention.

[0053] 4B, regions corresponding to a first frame F1, a second frame F2, a third frame F3, and a fourth frame F4' are shown. The first frame F1, the second frame F2, and the third frame F3 are the same as those described in FIG. 4A, and therefore will not be described again.

[0054] Referring to FIG. 4B, the fourth frame F4′ is a fourth transformation region 4 th A / D, 4th readout area 4 th 4B, the length of the vertical blank region H-Blk included in the fourth frame F4′ may be greater than the lengths of the vertical blank regions H-blk included in the first frame F1, the second frame F2, and the third frame F3.

[0055] Referring to FIG. 4B, the first frame F1, the second frame F2, and the third frame F3 are read out sequentially, and the frame data corresponding to each frame is stored in the frame buffer. The fourth frame F4' is stored in the fourth transformation area 4. th After A / D is complete, the data can be read out in real time without being stored in a frame buffer. At this time, the frame data corresponding to the fourth frame F4' is read out line by line and stored in a line buffer. The readout time of the frame data corresponding to the fourth frame F4' can be adjusted to sequentially read out and synthesize each line of the first frame F1, second frame F2, third frame F3, and fourth frame F4'.

[0056] 4A and 4B, the total frame length according to FIG. 4B is shorter than the total frame length according to FIG. 4A. According to the present invention, an image sensor capable of synthesizing multiple frames while reducing the number of frame buffers used is disclosed. More specific embodiments of the present invention will be described below with reference to the following.

[0057] FIG. 5 is a diagram illustrating an operation method of an image sensor according to an embodiment of the present invention.

[0058] Timing controller 401 in Figure 5 corresponds to timing controllers 400 and 400' shown in Figures 1 and 2. Frame buffers 501a, 501b, 501c, and 501n in Figure 5 correspond to frame buffers 501 and 501' shown in Figures 1 and 2. Line buffer 502a in Figure 5 corresponds to line buffers 502 and 502' shown in Figures 1 and 2.

[0059] 5, a timing controller 401 can control the output period of frame data corresponding to each of a plurality of frames. According to one example, the timing controller 401 can control the output period of frame data corresponding to N frames to be different from each other. The timing controller 401 can control the output period of Nth frame data among the frame data corresponding to the N frames to be different from the output period of the other frame data.

[0060] For example, the first frame data, second frame data, third frame data, ..., nth frame data are all frame data determined depending on the order of the frame data to be output sequentially. For example, the first frame data refers to the frame data that is output first, and the nth frame data refers to the frame data that is output last.

[0061] According to one example, first frame data is stored in the first frame buffer 501a. Second frame data is stored in the second frame buffer 501b. Third frame data is stored in the third frame buffer 501c. N-1th frame data is stored in the n-1th frame buffer 501n. Nth frame data is stored in the line buffer 502a.

[0062] According to one example, the output periods of the first frame data, the second frame data, the third frame data, ..., the nth frame data may each be controlled by the timing controller 401.

[0063] According to the present invention, by controlling the output period of the nth frame data so that it is longer than the output period of the remaining frame data, the nth frame data that has only been partially output is stored in the line buffer 502a, and the nth frame data is read out in real time, thereby reducing the amount of frame buffer usage.

[0064] According to one example, a frame controller 503 may be connected to the frame buffers 501a, ..., 501n and the line buffer 502a. According to one example, the frame controller 503 is a controller included in the frame buffer 501. The frame controller 503 may control the data stored in the frame buffers 501a, ..., 501n and the line buffer 502a to be read out line by line. The data read out line by line is merged by the image signal processor. According to one example, the image signal processor may perform HDR blending. HDR blending is a technology for enhancing the dynamic range by blending multiple frames having different luminance values ​​at similar times into one frame.

[0065] FIG. 6 is a flowchart illustrating a method of operating an image sensor according to one embodiment of the present invention.

[0066] 6, the timing controller may control the output speed of at least one frame data among N consecutive frame data to be different from each other. In this case, the frame data speed is the readout speed of data corresponding to each frame. In the present invention, the expression "outputting frame data at different speeds" has the same meaning as the expression "outputting signals corresponding to frame data at different output periods."

[0067] 6, N-1 frame data out of N consecutive frame data may be stored in a frame buffer. Therefore, when N frame data are read out, the image sensor includes N-1 frame buffers. According to one example, N-1 frame data out of N consecutive frame data are stored in respective frame buffers, and a portion of the remaining frame data is stored in a line buffer.

[0068] Referring to step S300 of Figure 6, lines of N consecutive frame data may be sequentially read out. According to one example, line data corresponding to one line of consecutive frame data may be sequentially read out.

[0069] 6, a plurality of frame data corresponding to each line may be synthesized and then subjected to ISP processing. In one example, N pieces of line data correspond to the first line of N consecutive frame data. The N pieces of first line data may be synthesized to output the first line data of the final frame.

[0070] FIG. 7 is a flowchart illustrating a method of operating an image sensor according to one embodiment of the present invention.

[0071] Step S110 in Fig. 7 is included in step S100 in Fig. 6. Steps S210 and S220 in Fig. 7 are included in step S200 in Fig. 6.

[0072] 7, the timing controller may output N-th frame data at a slower rate among N consecutive frame data. For example, the output rates of the first (1st) frame data to the (N-1th) frame data are all the same, and the output rate of the N-th frame data is slower than the output rate of the first (1st) frame data to the (N-1th) frame data.

[0073] Referring to step S210 of FIG. 7, the first frame data to the (N-1)th frame data among N consecutive frame data may be stored in the corresponding frame buffers.

[0074] Referring to step S220 of FIG. 7, a portion of the N-th frame data among N consecutive frame data may be stored in a line buffer.

[0075] According to the present invention, when N consecutive frames are to be synthesized, only N-1 frame buffers are required, thereby reducing the amount of frame buffer usage. The output speed and readout method of frame data according to the present invention will be described in more detail through the following embodiments.

[0076] FIG. 8A is a diagram showing ADC output data, readout data from a frame buffer, and ISP output data according to a comparative example.

[0077] 8A, a diagram illustrating outputting data corresponding to each frame output from the ADC (ADC out Frame buffer), outputting readout data from the frame buffer (Frame buffer Readout), and then generating ISP output data (Multiple Frame ISP Output) is shown. In the diagram of FIG. 8A, the X axis represents time. In the embodiment of FIG. 8A, the ISP process is performed when four frames are output from the image sensor.

[0078] The data corresponding to each frame output from the ADC (ADC out Frame buffer) is output sequentially for each frame, with the output interval FO1 corresponding to the first frame, the output interval FO2 corresponding to the second frame, the output interval FO3 corresponding to the third frame, and the output interval FO4 corresponding to the fourth frame being output sequentially.

[0079] The output section FO1 corresponding to the first frame includes a conversion section A / D and a lead-out section RO. In the conversion section A / D, the signal corresponding to the first frame is converted from analog to digital, and in the lead-out section, each line 1 corresponding to the first frame is output sequentially. Each line 1 is output in accordance with the number of lines included in the first frame.

[0080] The output section FO2 corresponding to the second frame includes a conversion section A / D and a lead-out section RO. In the conversion section A / D, the signal corresponding to the second frame is converted from analog to digital, and in the lead-out section, each line 2 corresponding to the second frame is output sequentially. Each line 2 is output in accordance with the number of lines included in the second frame.

[0081] The output section FO3 corresponding to the third frame includes a conversion section A / D and a lead-out section RO. In the conversion section A / D, the signal corresponding to the third frame is converted from analog to digital, and in the lead-out section, each line 3 corresponding to the third frame is output sequentially. Each line 3 is output in accordance with the number of lines included in the third frame.

[0082] The output section FO4 corresponding to the fourth frame includes a conversion section A / D and a lead-out section RO. In the conversion section A / D, the signal corresponding to the fourth frame is converted from analog to digital, and in the lead-out section, each line 4 corresponding to the fourth frame is output sequentially. Each line 4 is output in accordance with the number of lines included in the fourth frame.

[0083] The output segments FO1, FO2, FO3, and FO4 corresponding to each frame are output to and stored in the corresponding frame buffers, so in the embodiment of Figure 8A, four frame buffers are required, one for each of the output segments FO1, FO2, FO3, and FO4.

[0084] In the present invention, 1h time is 1 horizontal time. For example, when the frame rate is Z (Z is a natural number greater than or equal to 2), 1 horizontal time is determined as 1 / (Z*n), where n is the number of rows in the pixel array 110. 1 horizontal time refers to the time required to perform analog-to-digital conversion on pixel signals corresponding to 1 line or pixel signals output from pixels included in 1 row. In the present invention, 1h size is the length in the X-axis direction corresponding to 1 horizontal time.

[0085] After data is output to the four frame buffers, the same line in each frame is controlled to be read out sequentially, and the data is merged to correspond to each line, thereby performing ISP processing. In the step of outputting data read out from the frame buffer (Frame buffer Readout), line data can be read out from buffers corresponding to each of the frames stored in the frame buffer and transmitted to ISP hardware. Referring to FIG. 8A, the first line 1 of the first frame, the first line 2 of the second frame, the first line 3 of the third frame, and the first line 4 of the fourth frame can be read out sequentially, thereby reading out data of all frames corresponding to the first line. When data of all frames corresponding to the first line has been read out, it can be merged and the merged image data corresponding to the first line can be output.

[0086] Referring to the embodiment of Figure 8A, for processing by an image signal processor requiring multiple frames, data corresponding to each frame image is stored in a frame buffer and then processed sequentially. According to one example, HDR merge can be performed using multiple frames. Referring to Figure 8A, outputs corresponding to four frames can be stored in a frame buffer, and then lines of each frame can be output consecutively for processing.

[0087] In the comparative example, the first through fourth frames are all output in the same 1h size and stored in a frame buffer. In one example, an image sensor operating in global shutter mode had to store all frames in a frame buffer before use. Four frame buffers were required to store data corresponding to the first through fourth frames.

[0088] FIG. 8B is a diagram showing ADC output data, readout data from a frame buffer, and ISP output data according to an embodiment of the present invention.

[0089] 8B, a diagram for explaining the process of outputting data corresponding to each frame output from the ADC (ADC out Frame buffer), outputting readout data from the frame buffer (Frame buffer Readout), and then generating ISP output data (Multiple Frame ISP Output) is shown. In the diagram of FIG. 8B, the X axis represents time. In the embodiment of FIG. 8B, the ISP process is performed for four frames.

[0090] The data corresponding to each frame output from the ADC is output sequentially for each frame, with an output section FO1' corresponding to the first frame, an output section FO2' corresponding to the second frame, an output section FO3' corresponding to the third frame, and an output section FO4' corresponding to the fourth frame being output sequentially.

[0091] The output section FO1' corresponding to the first frame includes a conversion section A / D and a lead-out section RO. In the conversion section A / D, the signal corresponding to the first frame is converted from analog to digital, and in the lead-out section, each line 1 corresponding to the first frame is output sequentially. Each line 1 is output in accordance with the number of lines included in the first frame.

[0092] The output section FO2' corresponding to the second frame includes a conversion section A / D and a lead-out section RO. In the conversion section A / D, the signal corresponding to the second frame is converted from analog to digital, and in the lead-out section, each line 2 corresponding to the second frame is output sequentially. Each line 2 is output in accordance with the number of lines included in the second frame.

[0093] The output section FO3' corresponding to the third frame includes a conversion section A / D and a lead-out section RO. In the conversion section A / D, the signal corresponding to the third frame is converted from analog to digital, and in the lead-out section, each line 3 corresponding to the third frame is output sequentially. Each line 3 is output in accordance with the number of lines included in the third frame.

[0094] The output section FO4' corresponding to the fourth frame includes a conversion section A / D and a lead-out section RO. In the conversion section A / D, the signal corresponding to the fourth frame is converted from analog to digital. The lead-out section includes a horizontal blank section Hblank between each line 4 corresponding to the fourth frame. By including the horizontal blank section Hblank between each line 4 in the lead-out section corresponding to the fourth frame, the output speed of the line data 4 corresponding to the fourth frame can be controlled to be slow. In one example, the output length of each line 4 is four times the 1h size. In another example, the output length of each line 4 may be more than four times the 1h size. In one example, the 1h size is the size at which line data corresponding to the first to third frames is output. In one example, the actual data output for the fourth frame has a 1h size, but the horizontal blank section Hblank can be increased by about 3h to control the total 1h size to be four times larger.

[0095] The 1h size of each line 4 corresponding to the fourth frame is four times the 1h size of each of lines 1, 2, and 3 corresponding to frames 1 to 3. In other words, by delaying the 1h size of each line 4 corresponding to the fourth frame by four times the data of the previous frame, only the line data required for reading out each line is read out in the last frame, stored in the line buffer, and then output immediately, so a frame buffer corresponding to the fourth frame is not required.

[0096] The data corresponding to frames 1 through 3 are stored in a frame buffer. Each line 4 corresponding to frame 4 is stored in a line buffer instead of a frame buffer. Each line 4 corresponding to frame 4 is delayed by up to four times the horizontal blanking interval Hblank before being stored in the line buffer.

[0097] 8B, the first line data corresponding to the fourth frame is gradually read out until it reaches four times the size of the first line data corresponding to frames 1 to 3 and is stored in the line buffer. Thus, when reading out the first line data corresponding to all frames, the data corresponding to frames 1 to 3 can be output from the frame buffer, and the data corresponding to frame 4 can be output from the data stored in the line buffer, so a frame buffer corresponding to frame 4 is not required.

[0098] The first line data corresponding to the first frame, the first line data corresponding to the second frame, the first line data corresponding to the third frame, and the first line data corresponding to the fourth frame need to be read out sequentially, and the first line data corresponding to all frames may be read out sequentially and transmitted to the ISP.

[0099] In the present invention, it has been explained that when processing four frames, the output speed of line data 4 corresponding to the fourth frame is reduced by up to four times, but when processing N frames, it is also possible to output the line data corresponding to the Nth frame at a speed reduced by up to N times.

[0100] In the present invention, decreasing the output speed is expressed in the same sense as outputting such that the output period is increased or the 1h time is lengthened.

[0101] According to the present invention, when reading out a plurality of frame data, a line size corresponding to 1h can be variably applied instead of arranging the line sizes corresponding to 1h so that they are the same. As a result, the first to (N-1)th frame data can be read out relatively quickly, and the Nth frame data can be read out relatively slowly, so that the Nth frame data can be read out without storing it in a frame buffer. According to the present invention, the 1h size of the frame data corresponding to the last frame among the ADC outputs of the image sensor can be controlled to be increased to at least a multiple of the number corresponding to the total number of frames and output.

[0102] FIG. 9 is a diagram illustrating signals corresponding to ADC output data and signals for reading out the same according to an embodiment of the present invention.

[0103] Referring to FIG. 9, there are shown signals for ADC output in an image sensor that processes four frames and signals for finally merging and reading out the signals.

[0104] Referring to FIG. 9, the first transform section 1 corresponding to the first frame st A / D and first frame lead-out section (1 st frame ADC readout), the second conversion section 2 corresponding to the second frame nd A / D and second frame lead-out section (2 nd frame ADC readout), the third conversion section 3 corresponding to the third frame rd A / D and third frame lead-out section (3 rd frame ADC readout), the fourth conversion area 4 corresponding to the fourth frame th A / D and 4th frame lead-out section (4 th frame ADC readout) is shown.

[0105] Referring to FIG. 9, the first, second, third, and fourth frames can be merged to output the final frame, so the section that includes all sections corresponding to the first, second, third, and fourth frames is one frame length of the final frame.

[0106] Referring to FIG. 9, the first frame lead-out section (1 st In the second frame ADC readout section (2 nd In the third frame ADC readout section (3 frame ADC readout), the clock signals corresponding to the lines included in the second frame are toggled. rd In the first frame ADC readout, clock signals corresponding to a number of lines included in the third frame are toggled. st frame ADC readout), second frame readout section (2 nd frame ADC readout), 3rd frame readout section (3 rd The period of the clock signal toggled in each frame ADC readout is the same.

[0107] According to one example, the first frame lead-out section (1 st The line data output through the frame ADC readout is stored in the first frame buffer (1 st The second frame lead-out section (2 nd The line data output through the frame ADC readout is stored in the second frame buffer (2 nd The third frame lead-out section (3 rd The line data output through the frame ADC readout is stored in the third frame buffer (3 rdThe image is saved in the frame buffer.

[0108] 4th frame lead-out section (4 th In the fourth frame ADC readout, clock signals corresponding to the lines included in the fourth frame are toggled. According to one example, the cycles of the clock signals corresponding to the lines included in the first to third frames are different from the cycles of the clock signals corresponding to the lines included in the fourth frame. According to one example, the cycles of the clock signals corresponding to the lines included in the first to third frames are shorter than the cycles of the clock signals corresponding to the lines included in the fourth frame, i.e., faster than the cycles of the clock signals corresponding to the lines included in the fourth frame. According to one example, the cycles of the clock signals corresponding to the lines included in the first to third frames are one-fourth the cycles of the clock signals corresponding to the lines included in the fourth frame.

[0109] As a result, when the first line data included in multiple frames are sequentially read out, the first line data included in the first frame is read out from the first frame buffer, the first line data included in the second frame is read out from the second frame buffer, the first line data included in the third frame is read out from the third frame buffer, and the first line data included in the fourth frame is read out from the line buffer. According to the present invention, the clock signal corresponding to each of the multiple lines included in the last frame is output at a cycle that is four times larger than the clock signals corresponding to each of the multiple lines included in the first to third frames, so that after outputting only one line of data, it is stored in the line buffer and then immediately output in accordance with the readout timing of the multiple line data, thereby making it possible to sufficiently read out using only the line buffer.

[0110] According to the present invention, in order to reduce the required buffer size when processing using a frame buffer, image data corresponding to the last frame can be output by having a variable 1h size. According to the present invention, when a total of N frames are used, up to N-1 frames can be stored in a buffer, and the final Nth frame can be gradually read out until it reaches N times the 1h size of the existing frame and transmitted to the image processing device.

[0111] In the comparative example, a system that performs image processing using N frames required N frame buffers, but according to the present invention, a system that performs image processing using N frames requires only N-1 frame buffers, thereby reducing the amount of frame buffer usage.

[0112] FIG. 10 is a diagram showing signals corresponding to ADC output data and signals for reading out the same according to one embodiment of the present invention.

[0113] The signal diagram in Fig. 10 is the same as that in Fig. 9. In the embodiment in Fig. 10, the description that overlaps with the description in Fig. 9 will be omitted.

[0114] The APS in FIG. 10 corresponds to the pixels included in the pixel arrays 100 and 100′ in FIGS. 1 and 2. The timing controller in FIG. 10 corresponds to the timing controllers 400 and 400′ in FIGS. 1 and 2. The frame buffer in FIG. 10 corresponds to the frame buffers 501 and 501′ in FIGS. 1 and 2. The ISP in FIG. 10 is an image signal processor that processes image data that is ultimately output. According to one example, an image device is disclosed that includes an image sensor including a timing controller and a frame buffer, and an image signal processor that can process data output from the image sensor.

[0115] 10, the timing controller may control the signal output period in the lead-out section corresponding to each frame. According to one example, the timing controller may control the signal output period in the lead-out section corresponding to the last frame to be different from the signal output period in the lead-out section corresponding to the remaining frames. According to one example, the timing controller may control the signal output period in the lead-out section corresponding to the last frame to be longer than the signal output period in the lead-out section corresponding to the remaining frames. That is, the timing controller may control the 1h size in the lead-out section corresponding to the last frame to be larger than the 1h size in the lead-out section corresponding to the remaining frames. According to one example, the timing controller may control the output of the Nth frame data to have a horizontal blank section HBlank having a size of (N-1)*1h.

[0116] 10, when outputting readout data from the image signal processor, the frame buffer can sequentially read out each line data included in each frame data. For example, the frame buffer can read out the first line data of the first frame, the first line data of the second frame, the first line data of the third frame, the first line data of the fourth frame, the second line data of the first frame, etc. By increasing the 1H size of the readout section corresponding to the last frame using a timing controller, only a portion of the data of the last frame can be stored in the line buffer and read out, thereby reducing the amount of data used by the frame buffer.

[0117] Referring to FIG. 10, the image signal processor can synthesize line data for each frame and output final image data.

[0118] Although the present invention has been described as an example in which an image sensor performs HDR merging based on data stored in multiple frames, the present invention is not limited thereto. The image sensor according to the present invention can also be applied to cases in which a frame buffer is required in an image processing device that requires time-sequential information.

[0119] As mentioned above, exemplary embodiments have been disclosed in the drawings and specification. Although specific terms have been used to describe the embodiments in this specification, they are used solely for the purpose of explaining the technical idea of ​​the present invention and are not used to limit the meaning or the scope of the present invention as described in the claims. Therefore, a person skilled in the art will understand that various modifications and equivalent embodiments are possible. Therefore, the true technical scope of protection of the present invention should be determined by the technical idea of ​​the claims. [Explanation of symbols]

[0120] 401 Timing Controller 501a First Frame Buffer 501b Second Frame Buffer 501c 3rd frame buffer 501n n-1th frame buffer 502a Line Buffer 503 Frame Controller

Claims

1. 1. An image sensor operating in a global shutter mode, the image sensor comprising: a pixel array including a plurality of pixels; a timing controller configured to control periods of output signals corresponding to N frame data output from the plurality of pixels; N-1 frame buffers configured to store N-1 frame data respectively; Image sensor.

2. The timing controller 2. The image sensor according to claim 1, wherein any one of the periods of the output signals corresponding to the N frame data is controlled to be different from each other.

3. The timing controller 2. The image sensor of claim 1, wherein the period of the output signal corresponding to the last frame data of the N frame data is controlled so as to be different from the periods of the output signals corresponding to the first frame data to the (N-1)th frame data of the N frame data.

4. The timing controller 4. The image sensor of claim 3, configured to control the period of the output signal corresponding to the last frame data of the N frame data so that the period of the output signal corresponding to the last frame data of the N frame data is larger than the periods of the output signals corresponding to the first frame data to the (N-1)th frame data of the N frame data.

5. The timing controller 4. The image sensor of claim 3, configured to control the period of the output signal corresponding to the last frame data of the N frame data so that the period is N times or more larger than the period of the output signals corresponding to the first frame data to the (N-1)th frame data of the N frame data.

6. The N-1 frame buffers are:

2. The image sensor according to claim 1, wherein the image sensor is configured to store N-1 frame data excluding the last frame data among the N frame data.

7. The image sensor of claim 6 , further comprising a line buffer for storing a portion of the last frame data among the N frame data.

8. 8. The image sensor of claim 7, wherein the line buffer is configured to store data corresponding to one line of the last frame data.

9. The image sensor of claim 1 , wherein the plurality of pixels are digital pixels.

10. 1. An imaging device, the imaging device comprising: an image sensor configured to read out N consecutive frames of data; an image signal processor configured to merge and process the N frame data; the image sensor is configured to operate in a global shutter mode; the image sensor is configured to control the readout speed of at least one frame data among the N frame data so that the readout speeds of the frame data are different from each other; Image device.

11. The image sensor includes: N-1 frame buffers for storing N-1 frame data among the N frame data; 11. The image device of claim 10, further comprising: a line buffer for storing data corresponding to a line of any one of the N frame data.

12. The image sensor includes: Controlling the output period of the N-1 frame data stored in the N-1 frame buffers to be a first period; 12. The image device according to claim 11, configured to control an output cycle of the frame data stored in the line buffer to be a second cycle.

13. 13. The imaging device of claim 12, wherein the first period is less than the second period.

14. 13. The imaging device of claim 12, wherein the first period is 1 / Nth of the second period.

15. The frame buffer and the line buffer are 12. The image device according to claim 11, wherein the image device is controlled to output the data stored in the frame buffer and the line buffer sequentially in frame order.

16. the image sensor includes a plurality of pixels; 11. The imaging device of claim 10, wherein the pixels are digital pixels.

17. 1. A method of operating an image sensor, the method comprising: outputting N frame data from the image sensor; storing N-1 frame data of the N frame data in a frame buffer; and storing a portion of the frame data that is not stored in the frame buffer in a line buffer. method.

18. The step of outputting N frame data includes: outputting N-1 pieces of frame data so that an output period of N-1 pieces of frame data out of the N pieces of frame data is a first period; 18. The method of claim 17, further comprising: outputting the last frame data of the N frame data so that the output period of the last frame data is the second period.

19. 20. The method of claim 18, wherein the first period is less than the second period.

20. 20. The method of claim 18, wherein the first period is 1 / Nth of the second period.

Images