Image sensor using SPAD array

Abstract

The image sensor of the present invention includes a SPAD array including one or more SPAD elements, a MUX that receives a signal output from the SPAD array, a controller that transmits a mode setting signal to the MUX, and a first counter and a second counter that output a digital signal based on the signal output from the SPAD array, and the MUX can change the connection relationship between the first counter and the second counter in response to the mode setting signal.

Description

[Technical field]

[0001] The present invention relates to an image sensor using a SPAD array, and more particularly to an image sensor capable of switching modes and improving image quality. [Background technology]

[0002] An image sensor is a device that realizes an image through light and may include various elements capable of sensing light. Among elements capable of sensing light, a SPAD element is an element that can detect a single photon by the avalanche phenomenon, and research on image sensors using SPAD elements has been actively conducted recently.

[0003] Image sensors detect light in various ways depending on the mode. Each mode has its own advantages and disadvantages, so it is necessary to select a mode according to the object to be detected. Also, an image sensor with a new structure that can solve the problems of each mode is required.

[0004] This invention was derived from research conducted as part of the development of a large-area 20 x 20 cm hybrid X-ray video image detector based on Global Shutter by the Ministry of Science and ICT, the Ministry of Trade, Industry and Energy, the Ministry of Health and Welfare, and the Korea Food and Drug Safety Agency (Project unique number: 1711138026, Project number: KMDF_PR_20200901_0048-03, Research project name: Ministry of Health, Labor and Welfare Full-Periodic Medical Device Research and Development Project (R&D), Project management organization: Ministry of Health, Labor and Welfare Full-Periodic Medical Device Research and Development Foundation, Project carrying out organization: Yonsei University Industry-Academia Collaboration Foundation, Research period: March 1, 2021 to February 28, 2022).

[0005] Meanwhile, the Korean government, which is the project provider, has no proprietary interest in any aspect of this invention. Summary of the Invention [Problem to be solved by the invention]

[0006] One object of the present invention is to provide an image sensor that uses two or more counters to compensate for the shortcomings of each mode. Effect of the Invention

[0007] According to one embodiment of the present invention, an image sensor can be provided that uses two or more counters to compensate for the shortcomings of each mode. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing the configuration of a conventional image sensor using a SPAD. [Diagram 2] FIG. 1 is a diagram showing a counter operation and an implementation image in a global shutter mode of a conventional image sensor. [Diagram 3] 1 is a diagram showing a counter operation and an implementation image in a rolling shutter mode of a conventional image sensor; [Figure 4] 1 is a diagram illustrating a configuration of an image sensor according to an embodiment of the present invention. [Diagram 5] FIG. 2 is a diagram illustrating a configuration of an image sensor in a first mode according to an embodiment of the present invention. [Figure 6] FIG. 4 is a diagram for explaining a counter operation in a first mode. [Figure 7] FIG. 4 is a diagram illustrating a configuration of an image sensor in a second mode according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] According to one embodiment, an image sensor includes a SPAD array including one or more SPAD elements, a multiplexer (hereinafter referred to as "MUX") that receives a signal output from the SPAD array, a controller that transmits a mode setting signal to the MUX, and a first counter and a second counter that output a digital signal based on the signal output from the SPAD array, and the MUX can change the connection relationship between the first counter and the second counter in response to the mode setting signal.

[0010] Here, the mode setting signal may include a first signal corresponding to a first mode, which is a mode in which a photon integration period of the first counter and a photon integration period of the second counter do not overlap, and a second signal corresponding to a second mode, which is a mode in which the operation of the second counter is performed after the operation of the first counter.

[0011] Here, the first mode may be a mode in which when one of the first counter and the second counter is in a photon accumulation period, the other is in a read out period, and the second mode may be a mode in which a photon accumulation period corresponding to a first SPAD element included in the SPAD array of the first counter overlaps with a photon accumulation period corresponding to a second SPAD element included in the SPAD array.

[0012] Here, when the MUX receives the first signal from the controller, it transmits the signal output from the SPAD array to the first counter and the second counter, and when the MUX receives the second signal from the controller, it transmits the signal output from the SPAD array directly only to the first counter.

[0013] Here, when the MUX acquires the first signal, the first counter and the second counter are connected in parallel, and the second counter can acquire the signal output from the SPAD array from the MUX.

[0014] Here, when the MUX acquires the second signal, the first counter and the second counter are connected in series, and the second counter can acquire the signal output from the SPAD array from the first counter.

[0015] Here, the first SPAD element and the second SPAD element may be located in different rows on the SPAD array.

[0016] Here, the SPAD array may be disposed on a second wafer different from a first wafer on which the first counters and the second counters are disposed.

[0017] Here, the controller can generate an image of an object by reflecting light from the SPAD array based on a first result signal output from the first counter and a second result signal output from the second counter.

[0018] Here, the controller may change the mode setting signal based on a first image of the object at a first time point and a second image of the object at a second time point.

[0019] The embodiments described in this specification are intended to clearly explain the concept of the present invention to those having ordinary skill in the art to which the present invention pertains. Therefore, the present invention is not limited to the embodiments described in this specification, and the scope of the present invention should be construed as including modifications or variations that do not depart from the concept of the present invention.

[0020] The terms used in this specification are selected as general terms that are currently widely used as much as possible, taking into consideration the functions in the present invention, but this may vary depending on the intentions of those with ordinary skill in the art to which the present invention belongs, precedents, the emergence of new technologies, etc. However, if a specific term is used with a definition of an arbitrary meaning, the meaning of the term will be described separately. Therefore, the terms used in this specification should be interpreted based on the substantial meaning of the term and the overall content of this specification, rather than simply the name of the term.

[0021] The drawings attached to this specification are intended to easily explain the present invention, and the shapes shown in the drawings may be exaggerated as necessary to facilitate understanding of the present invention, and therefore the present invention is not limited by the drawings.

[0022] In this specification, if it is determined that a specific description of a known structure or function according to the present invention may obscure the gist of the present invention, the detailed description thereof will be omitted as necessary.

[0023] FIG. 1 is a diagram showing the configuration of a conventional image sensor using a SPAD.

[0024] Referring to FIG. 1, a conventional image sensor using a single photon avalanche diode (SPAD) may include a SPAD array including one or more SPAD elements, and a counter that receives a result signal of the SPAD array and counts photons.

[0025] Conventional image sensors have various problems because they are fixed to one of the global shutter mode and the rolling shutter mode and use one counter. Hereinafter, the problems of the conventional image sensor will be described with reference to FIGS. 2 and 3.

[0026] FIG. 2 is a diagram showing counter operation and implementation concept in the global shutter mode of a conventional image sensor.

[0027] Referring to FIG. 2, the conventional image sensor uses one counter, and therefore has a problem in that photons cannot be accumulated when the counter is read out in the global shutter mode.

[0028] Specifically, referring to FIG. 2(a), in the counter, a photon integration period (T_INT) in which photons are accumulated and a read out period (T_READ) in which the accumulated photons are read are alternately repeated.

[0029] For example, in FIG. 2(a), in the case of 30 fps, 0 s to 16.5 ms is a photon accumulation period (T_INT), and 16.5 ms to 33 ms after the photon accumulation period (T_INT) can be a lead-out period (33 ms). After the lead-out period from 16.5 ms to 33 ms, the photon accumulation period (T_INT) starts again from 33 ms to 49.5 ms. In this way, in the global mode, the photon accumulation period and the lead-out period of the counter can be alternately repeated.

[0030] Fig. 2(b) is a diagram showing an image generated based on the counting result of the counter in the global shutter mode. Fig. 2(b) will be described below in comparison with Fig. 3(b) in the explanation of Fig. 3.

[0031] Because photons cannot be accumulated during the counter readout period, the image sensor takes a long time to generate one image frame. In other words, when the image sensor operates in global shutter mode, the frame rate is low. To solve this problem of global shutter mode, the rolling shutter mode was introduced.

[0032] FIG. 3 is a diagram showing counter operation and implementation concept in the rolling shutter mode of a conventional image sensor.

[0033] Referring to FIG. 3, the counter operation in rolling shutter mode, which performs photon accumulation and readout for each row of the SPAD array to address the low frame rate in global shutter mode, and the generated image can be seen.

[0034] Specifically, referring to FIG. 3(a), the counter can distinguish between the SPAD elements in the SPAD array by row and perform photon accumulation and readout for each one.

[0035] For example, in the case of 30 fps, the counter can assign, to the first row, a first period that is a part of 33 ms as a photon accumulation period (T_INT), and a second period that is the remaining period of 33 ms excluding the first period as a lead-out period (T_READ).The counter can also assign, to the second row, a third period that is a part of 33 ms as a photon accumulation period (T_INT), and a fourth period that is the remaining period of 33 ms excluding the third period as a lead-out period (T_READ).

[0036] In this case, the read-out periods for each row can be set so as not to overlap. In other words, when the counter performs read-out for the first row, it cannot accumulate photons for the first row, but can accumulate photons for the second row, which partially solves the problem of not being able to accumulate photons during the read-out period that occurs with the global shutter.

[0037] In the rolling shutter mode, the counter accumulates and reads out photons for each row, resulting in less time loss than in the global shutter mode. However, in the rolling shutter mode, the image generated by the image sensor may have a problem in that the sensing time differs for each row. That is, motion artifacts and jello effects may occur. Therefore, when the object to be imaged moves, the image may be distorted.

[0038] Specifically, comparing Figure 2(b) and Figure 3(b), both of these objects are moving fans. When an image sensor operates in global shutter mode for a moving object, it can be seen that no image distortion occurs, even when the fan moves, taking into account the size of the fan blades. This is because the global shutter mode generates an image based on the results of sensing by all SPAD elements included in the SPAD array at the same point in time.

[0039] In contrast, the image sensor operated in rolling shutter mode shows image distortion when considering the size of the fan, as shown in Figure 3(b). This is because the rolling shutter mode divides the SPAD elements in the SPAD array into rows, and the time at which light is sensed varies for each row.

[0040] As described above, since the global shutter mode and the rolling shutter mode each have problems, an image sensor that operates in one mode may have difficulty in achieving a high quality image. In addition, the use of one counter in the image sensor in the global shutter mode may cause a low frame rate. Therefore, the image sensor of the present invention proposes a solution to solve the above problems.

[0041] FIG. 4 is a diagram showing a configuration of an image sensor according to an embodiment of the present invention.

[0042] Referring to FIG. 4, an image sensor according to an embodiment of the present invention may include a SPAD array 100, a MUX (Multiplexer), a first counter 300 and a second counter 400.

[0043] The SPAD array 100 may include one or more SPAD elements. The SPAD array 100 may include a plurality of SPAD elements arranged in a matrix. For example, the SPAD array 100 may be a SPAD matrix arranged in a form such as, but not limited to, 8×8, 64×64, 100×100, 1024×1024, 4000×4000, etc., and may be realized in various forms such as a circle, an ellipse, a honeycomb structure, etc.

[0044] SPAD array 100 may include one or more subarrays of N SPAD elements. For example, SPAD array 100 may include multiple subarrays of SPAD elements arranged in a 4×4 matrix.

[0045] When light is incident on the SPAD array 100, photons can be detected by an avalanche phenomenon. The SPAD array 100 can output a result signal based on the photon detection. The result signal can be an analog pulse. Therefore, a counter capable of counting photons based on the analog pulse is required.

[0046] The MUX 200 is connected to the SPAD array 100 and can receive a signal output from the SPAD array 100. At this time, the signal output from the SPAD array 100 is a result signal by photon detection, and the result signal may be an analog pulse as described above.

[0047] The MUX 200 may receive a mode setting signal 500 by a controller (not shown), which may provide overall control of the image sensor, and may be an MCU that generates an image based on the resulting signals of the SPAD array 100.

[0048] The MUX 200 can transmit signals received from the SPAD array 100 to the counters 300, 400. At this time, the transmission of signals to the counters 300, 400 can be changed according to a mode setting signal 500. The mode setting signal will be described later together with the description of the controller.

[0049] The MUX 200 can change the connection relationship between the counters 300 and 400 in response to the mode setting signal 500. Specifically, the MUX 200 can change the connection relationship between the first counter 300 and the second counter 400 to either serial or parallel in response to the mode setting signal 500. Alternatively, the MUX 200 can change the signal applied to the second counter 400. Alternatively, the MUX 200 can change the entity that inputs a signal to the second counter 400.

[0050] For example, when the mode setting signal is the first signal, the MUX 200 may set a connection relationship of the counters such that the first counter 300 and the second counter 400 are connected in parallel to each other. In this case, the second counter 400 may receive a signal output from the SPAD array 100 via the MUX 200. That is, the entity that inputs a signal to the second counter 400 may be the MUX 200.

[0051] Specifically, MUX 200 can set the input (IN) as a terminal connected to the SPAD array 100, and the output (OUT) as two terminals connected to the first counter 300 and the second counter 400, thereby setting the connection relationship so that the first counter 300 and the second counter 400 are connected in parallel.

[0052] Furthermore, for example, when the mode setting signal is the second signal, the MUX 200 can set the connection relationship of the counters such that the first counter 300 and the second counter 400 are connected in series. At this time, the second counter 400 is connected in series with the first counter 300, and can acquire a signal output from the SPAD array 100 via the first counter 300. In other words, the first counter 300 can be the entity that inputs a signal to the second counter 400.

[0053] Specifically, MUX 200 sets the input (IN) as a terminal connected to SPAD array 100 and sets the output (OUT) as a terminal connected to first counter 300, and can transmit the signal output from SPAD array 100 directly only to first counter 300. At this time, the output terminal of first counter 300 is connected to the input terminal of second counter 400, so that second counter 400 can indirectly obtain the signal output from SPAD array 100.

[0054] The operation of MUX 200 with mode setting signal 500 is described below with reference to FIGS.

[0055] The first counter 300 and the second counter 400 can count photons based on the signal output from the SPAD array 100 and output a corresponding digital signal. The first counter 300 and the second counter 400 can be an N-bit counter (N is an integer equal to or greater than 1) that can count up to the Nth power of 2. When the first counter 300 and the second counter 400 are connected in series, they can operate as a 2N-bit counter.

[0056] In the present specification, the image sensor is described as including two counters, but is not limited thereto, and the image sensor may include a plurality of counters. In this case, the two counters may operate as a pair. That is, the two counters may be connected in series or parallel according to a mode setting signal.

[0057] The image sensor may include a controller that outputs a mode setting signal 500. The mode setting signal 500 may include a first signal corresponding to a global shutter mode and a second signal corresponding to a rolling shutter mode.

[0058] The global shutter mode may be a mode in which photon integration and readout are performed at the same time for all SPAD elements included in the SPAD array 100 to generate one frame. That is, the global shutter mode may be a mode in which a photon integration period and a readout period of a counter are alternately and repeatedly performed.

[0059] The rolling shutter mode may be a mode in which, to generate one frame, photon accumulation and readout are performed for each row of the SPAD elements included in the SPAD array 100. That is, the rolling shutter mode may be a mode in which the photon accumulation period of the SPAD elements in the first row included in the SPAD array 100 partially overlaps with the photon accumulation period of the SPAD elements in the second row.

[0060] The controller can generate an image of the object to be sensed based on the output signal of the counter, and can determine and / or set the mode setting signal 500 according to the movement of the object.

[0061] For example, if the object is dynamic, the controller may set the mode setting signal 500 as a first signal corresponding to a global shutter mode to minimize motion artifacts and jello for dynamic motion.

[0062] Also, for example, if the object is static, the controller may set the mode setting signal 500 as a second signal corresponding to a rolling shutter mode to generate images at a higher frame rate.

[0063] In this way, the controller can determine the movement of the object and set or change the mode optimized for the object. There can be various ways in which the controller determines the movement of the object.

[0064] For example, the controller may sense the distance of the object using light, and if the change in the distance of the object exceeds a certain level, determine that the movement of the object is dynamic. Specifically, the image sensor may include a separate SPAD element for distance sensing, and may sense the distance of the object through the separate SPAD element. However, the present invention is not limited thereto, and one or more SPAD elements included in the SPAD array 100 may be used for sensing the distance of the object without disposing a separate SPAD element.

[0065] For example, the controller may determine the movement of the target object based on a difference in pixel values ​​between a first image generated at a first time point and a second image generated at a second time point. Specifically, the controller may compare a pixel value of a first position included in a first image generated at a first time point with a pixel value of the first position included in a second image generated at a second time point, and determine that the movement of the target object is dynamic if a change in the pixel value exceeds a certain level.

[0066] The image sensor can be realized over one or more wafers, depending on the placement of the SPAD array 100 and the counters 300, 400.

[0067] According to one embodiment, the SPAD array 100 and the counters 300, 400 may be disposed on one wafer. In this case, two counters may be connected to one SPAD element included in the SPAD array 100. However, this is not limited thereto, and two counters may be connected to one subarray included in the SPAD array 100.

[0068] According to another embodiment, the SPAD array 100 and the counters 300, 400 may be arranged on different wafers, where the first wafer on which the SPAD array 100 is arranged may be located above the second wafer on which the counters 300, 400 are arranged.

[0069] The reason why the SPAD array 100 and the counters 300, 400 are arranged on different wafers may be to secure the light receiving area of ​​the SPAD elements. That is, if the counters 300, 400 are arranged between the SPAD elements included in the SPAD array 100, light loss may occur because the counters cannot sense light incident on the area where they are arranged. Therefore, the area where the SPAD array 100 can sense light and the area where the counters can perform circuit operations may be separated by using different wafers.

[0070] FIG. 5 is a diagram illustrating a configuration of an image sensor in a first mode according to an embodiment of the present invention.

[0071] 5, the controller may set the mode setting signal 500 as a first signal 510 to set the image sensor to a first mode, where the first mode may be a mode corresponding to a global shutter mode.

[0072] The MUX 200 can receive a first signal 510 from the controller. The MUX 200 can change the connection relationship between the first counter 300 and the second counter 400 based on the first signal 510. The MUX 200 can connect the first counter 300 and the second counter 400 in parallel and transmit the signal output from the SPAD array 100 directly to the first counter 300 and the second counter 400.

[0073] In the first mode, the first counter 300 and the second counter 400 can alternately perform photon accumulation and readout processes on the output signal of the SPAD array 100. Using two counters can solve the problem of the global shutter mode in which photon accumulation is not possible during the readout period.

[0074] Next, the operation of the first counter 300 and the second counter 400 will be described with reference to FIG.

[0075] FIG. 6 is a diagram for explaining the operation of the counter in the first mode.

[0076] 6, the first counter 300 and the second counter 400 can alternately perform photon accumulation and readout. That is, while the first counter 300 performs photon accumulation, the second counter 400 can perform readout, and while the first counter 300 performs readout, the second counter 400 can accumulate photons. Therefore, the photon accumulation period of the first counter 300 and the photon accumulation period of the second counter 400 can not overlap each other.

[0077] For example, when the frame rate is 30 fps as shown in FIG. 6, the first counter 300 can accumulate photons during the first section from 0 seconds to 16.5 ms as a photon accumulation period (T_INT), and the second counter 400 can read out the results of the photon accumulation performed before the first section as a readout period (T_READ).

[0078] In addition, the first counter 300 uses the second section from 16.5 ms to 33 ms as a readout period (T_INT) to read out the results of the photon accumulation performed in the first section, and the second counter 400 uses the second section as a photon accumulation period (T_INT) to accumulate photons.

[0079] The image sensor of the present invention uses two counters, and therefore can alternate the readout of the counters in the first mode corresponding to the global shutter mode, thus solving the problem of the low frame rate of the conventional image sensor, i.e., the time required to generate an image can be reduced compared to the case of using one counter.

[0080] FIG. 7 is a diagram illustrating a configuration of an image sensor in a second mode according to an embodiment of the present invention.

[0081] 7, the controller may set the mode setting signal 500 as a second signal 520 to set the image sensor to a second mode, where the second mode may be a mode corresponding to a rolling shutter mode.

[0082] The MUX 200 can receive a second signal 520 from the controller. The MUX 200 can change the connection relationship between the first counter 300 and the second counter 400 based on the second signal 520. The MUX 200 can connect the first counter 300 and the second counter 400 in series, and transmit a signal output from the SPAD array 100 directly to the first counter 300 and indirectly to the second counter 400 via the first counter 300.

[0083] Since the second counter 400 receives an input signal from the first counter 300, the counting operation of the second counter 400 can be performed after the counting operation of the first counter 300. That is, the operations of the first counter 300 and the second counter 400 are not performed simultaneously, but the operation of the second counter 300 can be performed after the operation of the first counter 300.

[0084] In the second mode, the first counter 300 and the second counter 400 are connected in series to operate as a 2N-bit counter. Since two counters can be used to perform 2^2N counts, the dynamic range of the image sensor can be increased. Therefore, the image sensor can maintain a high frame rate in the rolling shutter mode and minimize motion artifacts to improve image quality.

[0085] As shown in FIG. 3, the first counter 300 and the second counter 400 can perform photon accumulation and readout for each row.

[0086] The method according to the embodiment may be realized in the form of program instructions that may be executed by various computer means and may be recorded on a computer readable medium. The computer readable medium may include, alone or in combination, program instructions, data files, data structures, and the like. The program instructions recorded on the medium may be those specially designed and constructed for the embodiment, or may be those known and available to those skilled in the art of computer software. Examples of computer readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include not only machine code, such as produced by a compiler, but also high-level language code that may be executed by a computer using an interpreter, or the like. The hardware devices may be configured to operate as one or more software modules to perform the operations of the embodiment, or vice versa.

[0087] Although the embodiments have been described above by way of limited embodiments and drawings, those skilled in the art may make various modifications and variations from the above description. For example, the described techniques may be performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or joined in a different manner than described, or may be counterbalanced or replaced by other components or equivalents, and still achieve suitable results.

[0088] Accordingly, other implementations, other embodiments, and equivalents of the claims are intended to be within the scope of the following claims.

Claims

1. a SPAD array including one or more SPAD elements; a MUX that receives a signal output from the SPAD array; a controller for transmitting a mode setting signal to the MUX; a first counter and a second counter that output a digital signal based on a signal output from the SPAD array; The MUX changes a connection relationship between the first counter and the second counter in response to the mode setting signal.

2. The mode setting signal is a first signal corresponding to a first mode in which a photon integration period of the first counter and a photon integration period of the second counter do not overlap; and a second signal corresponding to a second mode in which the second counter operates after the first counter operates.

3. The first mode is a mode in which, when one of the first counter and the second counter is in a photon accumulation period, the other is in a read out period; 3. The image sensor of claim 2, wherein the second mode is a mode in which a photon accumulation period corresponding to a first SPAD element included in the SPAD array of the first counter and a photon accumulation period corresponding to a second SPAD element included in the SPAD array overlap.

4. The MUX includes: When the first signal is received from the controller, a signal output from the SPAD array is transmitted to the first counter and the second counter; The image sensor according to claim 2 , wherein when the second signal is obtained from the controller, the signal output from the SPAD array is directly transmitted only to the first counter.

5. When the MUX acquires the first signal, the first counter and the second counter are connected in parallel; The image sensor of claim 4 , wherein the second counter receives a signal output from the SPAD array from the MUX.

6. When the MUX acquires the second signal, the first counter and the second counter are connected in series; The image sensor according to claim 4 , wherein the second counter obtains the signal output from the SPAD array from the first counter.

7. The image sensor of claim 3 , wherein the first SPAD element and the second SPAD element are in different rows on the SPAD array.

8. The image sensor of claim 1 , wherein the SPAD array is disposed on a second wafer different from a first wafer on which the first counters and the second counters are disposed.

9. 2. The image sensor of claim 1, wherein the controller generates an image of an object that reflects light from the SPAD array based on a first result signal output from the first counter and a second result signal output from the second counter.

10. 10. The image sensor of claim 9, wherein the controller modifies the mode setting signal based on a first image of the object at a first time point and a second image of the object at a second time point.

Images