Image sensor using SPAD array

Abstract

The image sensor of the present invention includes a SPAD array including one or more SPAD elements, a MUX that receives a signal output from the SPAD array, a controller that transmits a mode setting signal to the MUX, and a first counter and a second counter that output a digital signal based on the signal output from the SPAD array, and the MUX can change the connection relationship between the first counter and the second counter in response to the mode setting signal.

Description

Technical Field

[0001] The present invention relates to an image sensor using a SPAD array, and more particularly, to an image sensor capable of mode switching and improving image quality.

Background Art

[0002] An image sensor is a device that realizes an image through light and may include various elements capable of sensing light. Among the elements capable of sensing light, a SPAD element is an element capable of detecting a single photon by an avalanche phenomenon, and recently, research on image sensors using SPAD elements has been actively conducted.

[0003] An image sensor senses light in various ways according to the mode. Although advantages and disadvantages coexist for each mode, it is necessary to select each mode according to the object to be sensed. In addition, an image sensor with a new structure capable of solving the problems of each mode is required.

[0004] The present invention is derived from research conducted as part of the development of a 20×20 cm large-area hybrid X-ray fluoroscopic detector based on Global Shutter by the Ministry of Science and ICT, the Ministry of Trade, Industry and Energy, the Ministry of Health and Welfare, and the Food and Drug Administration (task unique number: 1711138026, task number: KMDF_PR_20200901_0048-03, research project name: R&D of medical devices for the entire cycle of the general department, task management agency: R&D consortium for medical devices for the entire cycle of the general department, task execution agency: Yonsei University Industry-University Cooperation Foundation, research period: March 1, 2021 to February 28, 2022).

[0005] On the other hand, in all aspects of the present invention, there is no property interest of the Korean government, which is the task provider.

Summary of the Invention

Problems to be Solved by the Invention

[0006] One objective of the present invention is to provide an image sensor that uses two or more counters to compensate for the shortcomings of each mode. [Effects of the Invention]

[0007] According to one embodiment of the present invention, an image sensor can be provided that uses two or more counters to compensate for the shortcomings of each mode. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram shows the configuration of an image sensor using a conventional SPAD. [Figure 2] This figure shows the counter operation and implementation image in the global shutter mode of a conventional image sensor. [Figure 3] This figure shows the counter operation and implementation image in the rolling shutter mode of a conventional image sensor. [Figure 4] This figure shows the configuration of an image sensor according to one embodiment of the present invention. [Figure 5] This figure shows the configuration of the image sensor in the first mode according to one embodiment of the present invention. [Figure 6] This diagram illustrates the counter operation in the first mode. [Figure 7] This figure shows the configuration of the image sensor in the second mode according to one embodiment of the present invention. [Modes for carrying out the invention]

[0009] An image sensor according to one embodiment includes a SPAD array containing one or more SPAD elements, a multiplexer (hereinafter referred to as "MUX") that receives signals output from the SPAD array, a controller that transmits a mode setting signal to the MUX, and a first counter and a second counter that output digital signals based on signals output from the SPAD array, wherein the MUX can change the connection relationship between the first counter and the second counter according to the mode setting signal.

[0010] Here, the mode setting signal may include a first signal corresponding to a first mode in which the photon accumulation period of the first counter and the photon accumulation (integration) period of the second counter do not overlap, and a second signal corresponding to a second mode in which the operation of the second counter occurs after the operation of the first counter.

[0011] Here, the first mode is a mode in which either the first counter or the second counter is in a photon storage period and the other is in a read-out period, and the second mode may be a mode in which the photon storage period corresponding to the first SPAD element included in the SPAD array of the first counter overlaps with the photon storage period corresponding to the second SPAD element included in the SPAD array.

[0012] Here, when the MUX receives the first signal from the controller, it transmits the signal output from the SPAD array to the first counter and the second counter, and when the MUX receives the second signal from the controller, it can directly transmit the signal output from the SPAD array only to the first counter.

[0013] In this case, when the MUX acquires the first signal, the first counter and the second counter are connected in parallel, and the second counter can acquire the signal output from the SPAD array from the MUX.

[0014] In this case, when the MUX acquires the second signal, the first counter and the second counter are connected in series, and the second counter can acquire the signal output from the SPAD array from the first counter.

[0015] Here, the first SPAD element and the second SPAD element can be located in different rows on the SPAD array.

[0016] Here, the SPAD array can be placed on a second wafer that is different from the first wafer on which the first and second counters are placed.

[0017] Here, the controller can generate an image of the object whose light has been reflected by the SPAD array, based on the first result signal output from the first counter and the second result signal output from the second counter.

[0018] Here, the controller can change the mode setting signal based on a first image of the object at a first time point and a second image of the object at a second time point.

[0019] The embodiments described herein are intended to clearly illustrate the spirit of the invention to those who have ordinary skill in the art to which the invention pertains. Therefore, the invention is not limited to the embodiments described herein, and the scope of the invention should be interpreted as including modifications or variations that do not depart from the spirit of the invention.

[0020] The terms used in this specification are selected as generally as possible in consideration of the functions in the present invention. However, this may vary depending on the intention of those with ordinary knowledge in the technical field to which the present invention pertains, precedents, or the emergence of new technologies. However, in contrast, when a specific term is defined and used in an arbitrary meaning, the meaning of that term will be described separately. Therefore, the terms used in this specification should be interpreted based on the substantial meaning of the term and the overall content of this specification, rather than simply as the name of the term.

[0021] The drawings attached to this specification are for easily explaining the present invention, and the shapes shown in the drawings may be exaggerated as necessary to assist in understanding the present invention. Therefore, the present invention is not limited by the drawings.

[0022] In this specification, when it is determined that a specific description of a known configuration or function related to the present invention may obscure the gist of the present invention, the detailed description thereof will be omitted as necessary.

[0023] Figure 1 is a diagram showing the configuration of a conventional image sensor using a SPAD.

[0024] Referring to Figure 1, a conventional image sensor using a single photon avalanche diode (SPAD) can include a SPAD array including one or more SPAD elements, and a counter that receives the result signal of the SPAD array and counts photons.

[0025] The conventional image sensor is fixed to one of the global shutter mode and the rolling shutter mode, and since it uses one counter, various problems have occurred. Hereinafter, referring to Figures 2 and 3, the problems of the conventional image sensor will be described. <00001​Figure 2 shows the counter operation and realized image in the global shutter mode of a conventional image sensor.

[0027] Referring to Figure 2, conventional image sensors used a single counter, which resulted in a problem where photons could not be accumulated when the counter read out in global shutter mode.

[0028] Specifically, as shown in Figure 2(a), the counter alternates between a photon integration period (T_INT) in which photons are accumulated and a read-out period (T_READ) in which the accumulated photons are read out.

[0029] For example, in Figure 2(a), at 30 fps, the period from 0 s to 16.5 ms is the photon storage period (T_INT), and the period from 16.5 ms to 33 ms after the photon storage period (T_INT) may be the readout period (33 ms). After the readout period from 16.5 ms to 33 ms, the photon storage period (T_INT) begins again from 33 ms to 49.5 ms. In this way, in global mode, the counter's photon storage period and readout period can alternate.

[0030] Figure 2(b) shows an image generated based on the counter count result in global shutter mode. Figure 2(b) is explained below in comparison to Figure 3(b) in the context of Figure 3.

[0031] A problem arose where the image sensor took a long time to generate a single image frame because photons could not be accumulated during the counter's readout period. In other words, when the image sensor operated in global shutter mode, a low frame rate was a problem. To solve this problem in global shutter mode, rolling shutter mode was introduced.

[0032] Figure 3 shows the counter operation and realized image in the rolling shutter mode of a conventional image sensor.

[0033] Referring to Figure 3, we can see the counter operation in rolling shutter mode, where photon accumulation and readout are performed row by row of the SPAD array, and the resulting image, in order to resolve the low frame rate in global shutter mode.

[0034] Specifically, referring to Figure 3(a), the counter can distinguish SPAD elements row by row in the SPAD array and perform photon storage and readout for each row.

[0035] For example, in the case of 30fps, the counter can allocate the first period, which is part of 33ms, as the photon storage period (T_INT) for the first row, and the remaining period of 33ms excluding the first period, which is the second period, as the readout period (T_READ). Similarly, the counter can allocate the third period, which is part of 33ms, as the photon storage period (T_INT) for the second row, and the remaining period of 33ms excluding the third period, which is the fourth period, as the readout period (T_READ).

[0036] In this case, the readout periods for each row can be set so that they do not overlap. That is, when the counter performs a readout for the first row, it is not possible to accumulate photons for the first row, but it is possible to accumulate photons for the second row. This partially solves the problem of not being able to accumulate photons during the readout period, which was a problem with global shutters.

[0037] In rolling shutter mode, the counter performs photon accumulation and readout for each row, resulting in less time loss than in global shutter mode. However, in rolling shutter mode, the image generated by the image sensor may have different sensing times for each row, which can lead to motion artifacts and the Jello effect. Therefore, if the object being imaged is moving, the image may become distorted.

[0038] Specifically, comparing Figure 2(b) and Figure 3(b), both objects are moving fans. When the image sensor operates in global shutter mode for a moving object, it can be seen that no image distortion occurs, even when the fan is moving, considering factors such as the size of the fan blades. This is because global shutter mode generates an image based on the results of sensing by all SPAD elements in the SPAD array at the same time.

[0039] In contrast, the image sensor operating in rolling shutter mode exhibits image distortion, as can be seen in Figure 3(b), when considering factors such as the size of the fan. This is because rolling shutter mode divides the SPAD elements in the SPAD array into rows, and the timing of light sensing differs for each row.

[0040] As described above, both global shutter mode and rolling shutter mode have their own drawbacks, making it difficult for an image sensor that operates in a fixed mode to achieve high-quality images. Furthermore, using a single counter in global shutter mode can lead to a low frame rate. Therefore, the image sensor of the present invention proposes a solution to the above problems.

[0041] Figure 4 shows the configuration of an image sensor according to one embodiment of the present invention.

[0042] Referring to Figure 4, an image sensor according to one embodiment of the present invention may include a SPAD array 100, a MUX (Multiplexer), a first counter 300, and a second counter 400.

[0043] The SPAD array 100 may include one or more SPAD elements. The SPAD array 100 may include multiple SPAD elements arranged in a matrix. For example, the SPAD array 100 may be an SPAD matrix configured in shapes such as 8×8, 64×64, 100×100, 1024×1024, or 4000×4000, but is not limited to these, and can be realized in various shapes such as circular, elliptical, or honeycomb structures.

[0044] The SPAD array 100 may include one or more subarrays, each consisting of N SPAD elements. For example, the SPAD array 100 may include multiple subarrays, each composed of a 4x4 matrix of SPAD elements.

[0045] When light is incident on the SPAD array 100, photons can be detected by the avalanche phenomenon. The SPAD array 100 can output a result signal from the photon detection. This result signal may be in the form of an analog pulse. Therefore, a counter is needed that can count photons based on this analog pulse.

[0046] The MUX200 is connected to the SPAD array 100 and can receive signals output from the SPAD array 100. In this case, the signal output from the SPAD array 100 is a result signal obtained by photon detection, and this result signal may be an analog pulse as described above.

[0047] The MUX200 can receive a mode setting signal 500 by a controller (not shown). The controller can perform overall control of the image sensor. The controller may also be an MCU that generates an image based on the result signals of the SPAD array 100.

[0048] The MUX200 can transmit signals received from the SPAD array 100 to counters 300 and 400. However, the transmission of signals to counters 300 and 400 may vary depending on the mode setting signal 500. The mode setting signal will be described later, along with a description of the controller.

[0049] The MUX200 can change the connection relationship between counters 300 and 400 in response to the mode setting signal 500. Specifically, the MUX200 can change the connection relationship between the first counter 300 and the second counter 400 to series or parallel in response to the mode setting signal 500. Alternatively, the MUX200 can change the signal applied to the second counter 400. Alternatively, the MUX200 can change the entity that inputs the signal to the second counter 400.

[0050] For example, when the mode setting signal is the first signal, the MUX200 can configure the counter connections so that the first counter 300 and the second counter 400 are connected in parallel. In this case, the second counter 400 can receive signals transmitted from the SPAD array 100 via the MUX200. In other words, the entity that inputs signals to the second counter 400 can be the MUX200.

[0051] Specifically, the MUX200 can be configured to set the input (IN) as a terminal connected to the SPAD array 100, and the output (OUT) as two terminals connected to the first counter 300 and the second counter 400, thereby setting the connection relationship so that the first counter 300 and the second counter 400 are connected in parallel.

[0052] Furthermore, for example, if the mode setting signal is the second signal, the MUX200 can configure the counter connections so that the first counter 300 and the second counter 400 are connected in series. In this case, the second counter 400, by being connected in series with the first counter 300, can acquire the signal output from the SPAD array 100 via the first counter 300. In other words, the entity that inputs a signal to the second counter 400 can be the first counter 300.

[0053] Specifically, the MUX200 can set its input (IN) to a terminal connected to the SPAD array 100 and its output (OUT) to a terminal connected to the first counter 300, thereby directly transmitting the signal output from the SPAD array 100 only to the first counter 300. In this case, the output terminal of the first counter 300 is connected to the input terminal of the second counter 400, allowing the second counter 400 to indirectly acquire the signal output from the SPAD array 100.

[0054] The operation of the MUX200 in response to the mode setting signal 500 will be described below with reference to Figures 5 to 7.

[0055] The first counter 300 and the second counter 400 can count photons based on the signal output from the SPAD array 100 and output a digital signal for this count. The first counter 300 and the second counter 400 may be N-bit counters (where N is an integer greater than or equal to 1) capable of counting up to 2 to the power of N. When the first counter 300 and the second counter 400 are connected in series, they can operate as a 2N-bit counter.

[0056] Although this specification illustrates an image sensor including two counters, it is not limited to this, and an image sensor may include multiple counters. In this case, the two counters can operate as a pair. That is, the two counters form a pair and can be connected in series or parallel depending on the mode setting signal.

[0057] The image sensor may include a controller that outputs a mode setting signal 500. The mode setting signal 500 may include a first signal corresponding to global shutter mode and a second signal corresponding to rolling shutter mode.

[0058] The global shutter mode may be a mode in which photon accumulation and readout are performed simultaneously for all SPAD elements included in the SPAD array 100 to generate a single frame. In other words, the global shutter mode may be a mode in which the photon accumulation (integration) period and the readout period of the counter are repeatedly performed alternately.

[0059] The rolling shutter mode may be a mode in which photon accumulation and readout are performed row by row for each SPAD element included in the SPAD array 100 in order to generate a single frame. In other words, the rolling shutter mode may be a mode in which the photon accumulation period of the first row of SPAD elements included in the SPAD array 100 partially overlaps with the photon accumulation period of the second row of SPAD elements.

[0060] The controller can generate an image of the object to be sensed based on the output signal of the counter. At this time, the controller can determine and / or set the mode setting signal 500 according to the movement of the object.

[0061] For example, if the object is moving, the controller can set the mode setting signal 500 as the first signal corresponding to the global shutter mode to minimize motion artifacts and Jero-like phenomena in response to dynamic movement.

[0062] Furthermore, for example, if the target object is static, the controller can set the mode setting signal 500 as a second signal corresponding to the rolling shutter mode, and increase the frame rate to generate an image.

[0063] In this way, the controller can determine the movement of the target object and set or change to a mode optimized for that object. There can be various ways in which the controller determines the movement of the target object.

[0064] For example, the controller can sense the distance of an object using light, and if the distance fluctuation of the object exceeds a certain level, it can determine that the object's movement is dynamic. Specifically, the image sensor can sense the distance of an object via a separate SPAD element, by including a separate SPAD element for distance sensing. However, it is not limited to this, and one or more SPAD elements included in the SPAD array 100 may be used for sensing the distance of an object without arranging a separate SPAD element.

[0065] Furthermore, for example, the controller can determine the movement of an object based on the difference in pixel values ​​between a first image generated at a first time point and a second image generated at a second time point. Specifically, by comparing the pixel value of a first position included in the first image generated at the first time point with the pixel value of the same first position included in the second image generated at the second time point, the controller can determine that the movement of the object is dynamic if the variation in the pixel value exceeds a certain level.

[0066] The image sensor can be implemented via one or more wafers. This can vary depending on the arrangement of the SPAD array 100 and the counters 300 and 400.

[0067] According to one embodiment, the SPAD array 100 and the counters 300 and 400 can be arranged on a single wafer. In this case, two counters can be connected to each SPAD element included in the SPAD array 100. However, it is not limited to this, and two counters may be connected to each sub-array included in the SPAD array 100.

[0068] In another embodiment, the SPAD array 100 and the counters 300 and 400 can be arranged on different wafers. In this case, the first wafer on which the SPAD array 100 is arranged can be located above the second wafer on which the counters 300 and 400 are arranged.

[0069] The reason why the SPAD array 100 and the counters 300 and 400 are placed on different wafers may be to ensure the light-receiving area of ​​the SPAD elements. In other words, if the counters 300 and 400 are placed between the SPAD elements included in the SPAD array 100, the light incident on the area where the counters are placed cannot be sensed, which may result in light loss. Therefore, the area where the SPAD array 100 can sense light and the area where the counters can perform circuit operations can be separated by using different wafers.

[0070] Figure 5 shows the configuration of the image sensor in the first mode according to one embodiment of the present invention.

[0071] Referring to Figure 5, the controller can set the mode setting signal 500 as the first signal 510 in order to set the image sensor to the first mode. In this case, the first mode may be the mode corresponding to the global shutter mode.

[0072] The MUX200 can receive a first signal 510 from the controller. Based on the first signal 510, the MUX200 can change the connection relationship between the first counter 300 and the second counter 400. The MUX200 can connect the first counter 300 and the second counter 400 in parallel and directly transmit signals output from the SPAD array 100 to the first counter 300 and the second counter 400.

[0073] In the first mode, the first counter 300 and the second counter 400 can alternately perform photon storage and readout processes with respect to the output signal of the SPAD array 100. By using two counters, the problem of global shutter mode, where photon storage cannot occur during the readout period, can be solved.

[0074] Next, the operation methods of the first counter 300 and the second counter 400 will be explained based on Figure 6.

[0075] Figure 6 is a diagram illustrating the operation of the counter in the first mode.

[0076] Referring to Figure 6, the first counter 300 and the second counter 400 can alternately perform photon storage and readout. That is, while the first counter 300 is storing photons, the second counter 400 can perform readout, and while the first counter 300 is performing readout, the second counter 400 can store photons. Therefore, the photon storage periods of the first counter 300 and the second counter 400 do not overlap.

[0077] For example, as shown in Figure 6, if the frame rate is 30fps, the first interval from 0 seconds to 16.5ms is set as the photon storage period (T_INT) for the first counter 300 to store photons, and the second counter 400 is set as the readout period (T_READ) to read out the results of the photon storage performed before the first interval.

[0078] Furthermore, in the second interval from 16.5ms to 33ms, the first counter 300 can be used as a readout period (T_INT) to read out the results of the photon accumulation performed in the first interval, while the second counter 400 can be used as a photon accumulation period (T_INT) to accumulate photons.

[0079] The image sensor of the present invention uses two counters, allowing for alternating readout of the counters in the first mode, which corresponds to the global shutter mode, thus solving the conventional problem of low frame rates. In other words, the time required to generate an image can be shortened compared to when only one counter is used.

[0080] Figure 7 shows the configuration of the image sensor in the second mode according to one embodiment of the present invention.

[0081] Referring to Figure 7, the controller can set the mode setting signal 500 as the second signal 520 in order to set the image sensor to the second mode. In this case, the second mode may be the mode corresponding to the rolling shutter mode.

[0082] The MUX200 can receive a second signal 520 from the controller. Based on the second signal 520, the MUX200 can change the connection relationship between the first counter 300 and the second counter 400. The MUX200 can connect the first counter 300 and the second counter 400 in series, and transmit the signal output from the SPAD array 100 directly to the first counter 300, and indirectly to the second counter 400 via the first counter 300.

[0083] Since the second counter 400 receives an input signal from the first counter 300, the second counter 400 can perform its counting operation after the first counter 300 has performed its counting operation. In other words, the operation of the first counter 300 and the operation of the second counter 400 do not occur simultaneously; the operation of the second counter 300 can be performed after the operation of the first counter 300.

[0084] In the second mode, the first counter 300 and the second counter 400 can be connected in series and operate as a 2N-bit counter. Since 2 to the power of 2N counts can be performed using the two counters, the dynamic range of the image sensor can be increased. Therefore, the image sensor can maintain a high frame rate in rolling shutter mode, minimizing motion artifacts and improving image quality.

[0085] As shown in Figure 3, the first counter 300 and the second counter 400 can perform photon storage and readout for each row.

[0086] The methods according to the embodiments are implemented in the form of program instructions that can be performed via various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., individually or in combination. The program instructions recorded on the medium may be specifically designed and configured for the embodiments, or they may be publicly known and available to those skilled in the computer software art. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices such as ROMs, RAMs, and flash memory that are specifically configured to store and execute program instructions. Examples of program instructions include not only machine code that can be produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

[0087] As described above, although embodiments have been described by limited embodiments and drawings, various modifications and variations can be made from the above description by a person with ordinary skill in the art. For example, the described technique may be performed in a different order than described, and / or the components of the described system, structure, apparatus, circuit, etc. may be combined or joined in a different manner than described, or replaced or substituted by other components or equivalents, and the appropriate results may be achieved.

[0088] Therefore, other embodiments, other embodiments, and those equivalent to the claims described below also fall within the scope of the claims.

Claims

1. A SPAD array containing one or more SPAD elements, A MUX that receives signals output from the SPAD array, A controller that transmits a mode setting signal to the aforementioned MUX, Includes a first counter and a second counter that output digital signals based on signals output from the SPAD array, The mode setting signal is, A first signal corresponding to a first mode in which the photon accumulation period of the first counter and the photon accumulation (integration) period of the second counter do not overlap, Includes a second signal corresponding to a second mode, which is a mode in which the operation of the second counter occurs after the operation of the first counter, The MUX changes the connection relationship between the first counter and the second counter to the SPAD array in parallel or in series according to the mode setting signal. The aforementioned MUX is When the first signal is obtained from the controller, the first counter and the second counter are connected in parallel to transmit the signal output from the SPAD array to the first and second counters, and the second counter is configured to obtain the signal output from the SPAD array from the MUX. An image sensor configured such that when the second signal is obtained from the controller, the first counter and the second counter are connected in series, the signal output from the SPAD array is directly transmitted only to the first counter, and the second counter obtains the signal output from the SPAD array from the first counter.

2. The image sensor according to claim 1, wherein the SPAD array is disposed on a second wafer different from the first wafer on which the first counter and the second counter are disposed.

3. The image sensor according to claim 1, wherein the controller generates an image of an object whose light has been reflected by the SPAD array based on a first result signal output from the first counter and a second result signal output from the second counter.

4. The image sensor according to claim 3, wherein the controller changes the mode setting signal based on a first image of the object at a first time point and a second image of the object at a second time point.

Images