Dynamic Vision Sensor Architecture

Abstract

To provide a change detection sensor for processing visual information much faster than a conventional computer vision system.SOLUTION: A dynamic vision sensor (DVS) or a change detection sensor for monitoring how a scene changes in response to a change in light intensity includes one pixel 100 or a two-dimensional or one-dimensional array of pixels, compares changes of registered intensity by the pixel, records a pixel address at which the change is positive or negative, performs processing to correlate a multi-bit pixel value of a color or gray level between continuous frames, and analyzes the frames on the basis of only three values being an increase, a decrease and no change about the pixel.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] [Related Applications] This application claims priority to Swiss Patent Application No. CH20160001764, filed on December 30, 2016, and Swiss Patent Application No. CH20160001765, filed on December 30, 2016, both of which are hereby incorporated by reference in their entirety.

[0002] [Technical Field] The present invention relates to vision sensors, and more particularly to dynamic or change detection sensors. These sensors respond to changes in light intensity and monitor how a scene changes in such a way.

Background Art

[0003] Currently, most machine vision is based on conventional cameras and frame-based image sensors associated with them. For some machine vision tasks, such as object recognition, these conventional frame-based cameras are well-suited. However, for other tasks, such as tracking or the estimation of position and movement, conventional image sensors have drawbacks.

[0004] The main drawback is that conventional cameras produce a large amount of redundant and unnecessary data, which must be captured, communicated, and processed. This high data load slows down the response time by reducing the temporal resolution, increases the power consumption, and further increases the size and cost of the machine vision system. In addition, most image sensors have a limited dynamic range, insufficient low-light performance, and problems with motion blur.

[0005] These drawbacks result from the fact that data is captured as a sequence of still images (frames). In some cases, encoding a dynamic scene as a still image is useful for creating beautiful images and videos, but it is not optimal for data processing for machine vision.

[0006] Conventional computer vision systems using conventional cameras typically compare features between consecutive image frames for object recognition. To estimate the position and orientation of a mobile system and to infer a 3D map of the surrounding world, two consecutive images that are partially overlapping but taken from different poses at different times are compared. To infer the motion that has occurred between two frames, characteristic visual landmarks (key points or other visual features) must be matched across the two images. Finding pairs of these locations that correspond to each other in both images is known as solving the "correspondence problem."

[0007] Solving the correspondence problem requires a significant amount of processing power. To detect landmarks, all pixels in the image must be searched for characteristic features (corners, blobs, edges, etc.). Next, pixels and their neighboring pixels are grouped to characterize so-called feature descriptors, which are then used to match features between frames, thereby establishing pairs of corresponding locations. This is computationally intensive. A direct approach of directly comparing pixel intensities is even more computationally complex.

[0008] On the other hand, a so-called Dynamic Vision Sensor (DVS) is a sensor that overcomes the limitations of frame-based encoding. See Patent Document 1, Lichtsteiner et al., "Photoarray for Detecting Time-Dependent Image Data", which is incorporated herein by reference. By using in-pixel data compression, data redundancy is removed, achieving a high dynamic range with high temporal resolution, low latency, low power consumption, and almost no motion blur. Therefore, DVS is particularly suitable for solar cell or battery-powered compressive sensing applications, or for mobile machine vision applications where the position of the system must be estimated and the processing power is limited due to limited battery capacity.

[0009] The DVS preprocesses visual information locally. Instead of generating a sharp image, the DVS creates smart data for computer applications. Conventional image sensors capture video as a series of still images, while the DVS detects the position of changes in the scene and transmits only that. Since the DVS performs in-pixel data compression, it encodes visual information much more efficiently than conventional cameras. This means that data can be processed with fewer resources, lower net power, and a faster system response time. The high temporal resolution enables continuous tracking of visual features, thereby overcoming correspondence problems. In addition, the architecture of the DVS enables a high dynamic range and good low-light performance.

Prior Art Documents

Patent Documents

[0010]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0011] Typical DVSs may have several drawbacks:

[0012] 1. Pixel circuits can be large in some cases because they contain an amplifier and two comparators.

[0013] 2. Asynchronous readout circuits are prone to timing jitter. If a large area of the scene changes in a short time, the jitter increases, causing data readout from parts of the sensor to be delayed, which may lead to motion artifacts.

[0014] 3. DVS pixel circuits tend to be self-timed, so they cannot impose a sensor time resolution, and very fast changes in a small area may saturate the communication bandwidth.

[0015] 4. DVS sensors have the characteristic that the amount of output data produced mainly depends on the dynamics of the scene. Therefore, the data rate is unpredictable, leading to problems in the processing stage, i.e., the problem of overload (i.e., the processing stage cannot handle the data volume).

[0016] 5. In some DVS sensors, the pixels in the array have their rows reset after each row is read out. This means that small changes from frame to frame cannot be accumulated, so slow movements are not detected. Furthermore, existing pixel circuit designs are prone to motion artifacts caused by rolling shutters.

Means for Solving the Problems

[0017] The present invention relates to the design of DVS (Dynamic Vision Sensor). Different embodiments can be employed to mitigate, remove, or obviate at least some of the disadvantages associated with existing solutions. For example, embodiments of the present invention enable reducing the size of pixels used in a pixel array, and as a result, enable a smaller sensor for the same resolution or a higher resolution for the same die size. Further, in some embodiments, the pixels of the pixel array may be activated simultaneously (thus enabling a sensor that functions simultaneously). Moreover, the time for evaluating the change in light intensity can be controlled, and thus the temporal resolution and event rate can be adapted to the current situation and application.

[0018] The pixel circuit can function simultaneously and / or according to an external timing reference. This has several advantages listed below:

[0019] 1. Event rate control by varying the temporal resolution of the sensor.

[0020] 2. Frame-based readout enables data readout with less temporal jitter from the pixels.

[0021] 3. Having a clocked pixel circuit allows the use of standard digital design tools, making the design of the digital communication circuit interfacing with the pixel array easier. The design of the asynchronous circuit of a typical existing DVS tends to be more difficult because suitable off-the-shelf tools are not available.

[0022] 4. Controlling the timing by a controller enables separating the change detection phase and the readout phase temporally. This reduces the risk of generating false events due to parasitic coupling from one pixel to another.

[0023] Generally, according to one aspect, the present invention features a sensor. The sensor typically comprises an array of pixels. The array can be a one- or two-dimensional array. However, a single-pixel sensor is also possible.

[0024] Each of the pixels of the sensor includes several elements. A photosensor detects incident light. The photoreceptor signal is a function of the amount of light received by the photosensor. A memory capacitor is further provided, where the first plate of the capacitor is charged with the charge from the photoreceptor signal, and the second plate of the capacitor is connected to a comparator node whose voltage varies with the change in the photoreceptor signal. Next, one or more comparators compare the voltage of the comparator node with one or more reference voltages to evaluate the change in the photoreceptor signal against one or more thresholds.

[0025] Generally, according to another aspect, the present invention features a sensor. This sensor typically includes an array of pixels or a single pixel. Each of those pixels includes a photosensor. The photoreceptor signal is a function of the amount of light received by the photoreceptor. Further, the memory capacitor is charged with the charge from the photoreceptor signal. Only one comparator is provided in the pixel. The comparator compares the voltage of the comparator node with one or more reference voltages to evaluate the change in the photoreceptor signal against one or more thresholds.

[0026] The use of only a single comparator in the pixel has the advantage of reducing the size of the pixel. This size reduction can be used to increase the size of the array for the same area of the semiconductor chip. On the other hand, it can be used to reduce the overall size of the sensor, enabling a lower-cost device.

[0027] Generally, according to another aspect, the present invention features a sensor. This sensor also comprises an array of pixels or a single pixel. Each of the pixels includes a photosensor and a memory capacitor. The pixel further includes a comparator that compares the voltage from the memory capacitor with one or more reference voltages to evaluate the change in the photoreceptor signal against one or more thresholds. According to the present invention, the comparator also provides a reset voltage to the memory capacitor.

[0028] It is useful to use a comparator to provide a reset voltage to the memory capacitor. This is because the reset voltage resets the memory capacitor in a way that takes into account the offset in the comparator. This is important because each of the comparators in the pixel array may have a slightly different offset due to manufacturing variations. According to the present invention, any resulting offset is removed by using the comparator to provide a reset voltage to the memory capacitor.

[0029] Generally, according to another aspect, the present invention features a sensor. This sensor also comprises an array of pixels or a single pixel. Each of the pixels includes a photosensor and a memory capacitor. A comparator is further provided in the pixel. The comparator sequentially compares the voltage from the memory capacitor with two reference voltages to evaluate the change on the photoreceptor signal against an on-threshold or an off-threshold.

[0030] The present invention has the advantage of using a single comparator to determine both on-events and off-events by continuously comparing the voltage from the memory capacitor with an on-threshold or an off-threshold.

[0031] Generally, according to another aspect, the present invention features a sensor. This sensor has an array of pixels or a single pixel. Each of those pixels includes a photosensor and a memory capacitor. Further, the pixel has one or more comparators that compare the voltage from the memory capacitor to evaluate the change in the photoreceptor signal against one or more thresholds. Finally, a memory structure is provided in the pixel to store the output of the one or more comparators.

[0032] Providing a memory structure in the pixel enables the pixel to store the result of the threshold processing. This allows, for example, anticipating the timing at which information is read from various pixels in the array.

[0033] Generally, according to another aspect, the present invention features an array of pixels or a single pixel. Each of these pixels includes a photosensor and a memory capacitor. According to the present invention, a switch is also provided in each of the pixels and is controlled by a shutter circuit signal that connects the photosensor to the memory capacitor. In this way, a global shutter signal can be provided to all of the pixels in the array.

[0034] The use of a global shutter is advantageous in that, for example, a single signal can be used to trigger the entire array to avoid problems associated with rolling shutters.

[0035] Generally, according to another aspect, the present invention features a sensor. This sensor comprises an array of pixels or a single pixel. Each of those pixels includes a photosensor. The photoreceptor signal is based on the light received by the photosensor in a memory capacitor that is charged with the charge from the photoreceptor signal. According to the present invention, one or more comparators are provided in the readout circuit of the array. These comparators compare the voltage of the memory capacitor of the array with a reference voltage to evaluate the change in the photoreceptor signal against one or more thresholds.

[0036] In this invention, the comparators are moved to the peripheral circuits surrounding the pixel array. This enables further reduction of their corresponding pixels. On the other hand, in order to provide the threshold processing function required for the entire array, a small number of comparators can be used.

[0037] Generally, according to another aspect, the present invention can also be characterized as a method. Specifically, a method of operating a sensor includes steps of a photosensor of a pixel of an array for detecting incident light, generating a photoreceptor signal that is a function of the amount of light received by the photosensor, using a memory capacitor to store the charge corresponding to the light previously detected by the photosensor, wherein a first plate of the capacitor is charged with the charge from the photoreceptor signal, a second plate of the capacitor is connected to a comparator node and their voltages vary with the change of the photoreceptor signal, and comparing the voltage of the comparator node with one or more reference voltages to evaluate the change of the photoreceptor signal against one or more thresholds.

[0038] The above and other features of the present invention, including various novel details of the configuration and combination of components, as well as other advantages, will be described in more detail below with reference to the accompanying drawings and will be shown in the claims. It will be understood that the specific methods and devices embodying the present invention are shown as illustration and not as limitation of the present invention. The principles and features of the present invention may be employed in various numerous embodiments without departing from the scope of the present invention.

[0039] In the accompanying drawings, reference characters refer to the same parts throughout the different drawings. The drawings are not necessarily to scale and are rather emphasized when showing the principles of the present invention. The drawings are as follows.

Brief Description of the Drawings

[0040]

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Embodiments for Carrying Out the Invention

[0041] Next, the present invention will be described more fully with reference to the accompanying drawings showing embodiments that are examples of the present invention. However, the present invention may be embodied in many different forms and should not be construed as limited to the embodiments presented herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and the scope of the present invention will be fully conveyed to those skilled in the art.

[0042] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. Further, the singular forms and the articles "a", "an" and "the" are intended to also include the plural forms as well, unless otherwise specified. The terms: include, comprise, including and / or comprising, when used herein, specify the presence of the stated features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Further, when an element is referred to and / or shown as being connected or coupled to another element, including a component or subsystem, it will be understood that the element can be directly connected or coupled to the other element, or intervening elements may be present.

[0043] The terms such as "first" and "second" are used herein to describe various elements, but it will be understood that these elements should not be limited by these terms. These terms are only used to distinguish one element from another. Thus, an element discussed below could be referred to as a second element and vice versa, but the second element could also be referred to as the first element without departing from the teachings of the present invention.

[0044] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. Terms defined as in a commonly used dictionary should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant technical field and should not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0045] Throughout this application, many specialized terms are used. For convenience and for an accurate description of the present invention, those specialized terms are defined below.

[0046] Closed switch: A switch that is conducting, i.e., a switch where two terminals are electrically connected.

[0047] Electrical connection: Either a direct (electrical such that current can flow between two nodes) or via a buffer connection between two nodes in a circuit.

[0048] Event: An increase or decrease in light intensity at a given pixel.

[0049] Event frame: One iteration of the application of a threshold voltage, operation of the memory, reading and resetting of the memory content.

[0050] Event rate: The number of events per second generated and transmitted by a pixel circuit. The event rate depends on the pixel circuit configuration, threshold settings and temporal resolution, and further on the movement of the sensor as well as the observed scene.

[0051] Frame rate: The number of times each pixel is read per second. It corresponds to the rate at which one entire event frame is scanned. The frame rate can be fixed or made dependent on the number of events generated.

[0052] Motion artifact: When either the camera or the object is moving fast across a frame, the rolling shutter may show a significant skew of potential vertical lines in the image. If there is fast motion in the frame, the entire image may be distorted.

[0053] OFF event: A discrete decrease in light intensity for a given pixel.

[0054] ON event: A discrete increase in light intensity for a given pixel.

[0055] Open switch: A switch that is not conducting, i.e., an open circuit is provided between two terminals.

[0056] Pixel address: A number or pair of numbers that describes the position of a pixel in an array. Usually, a row number and a column number.

[0057] Power rail: Either ground (Vss) or a power supply voltage (Vdd).

[0058] Generally, FIG. 1 shows the components of a pixel circuit assembled and connected according to the principles of the present invention. Later in this document, reference is made to a two-dimensional pixel array having rows and columns of pixels, each pixel having electronics as shown in this or an alternative embodiment.

[0059] In the present invention, a "pixel" refers to a sensing element that records the intensity of light impinging thereon, and a "pixel circuit" or "pixel electronics" refers to the electronic components and circuit elements of a pixel. In this document, the term "pixel circuit" is used to focus on considerations regarding the electronics of a pixel, and the term "pixel" is used to refer to the sensing element as a whole. Typically, a sensor (see FIG. 3) is composed of a two-dimensional array of pixels 100 and additional peripheral circuit elements. Still, a two-dimensional array is not required for all applications. The sensor can also include a single pixel (an array of one pixel) or a one-dimensional array of pixels (a line sensor).

[0060] The main components of pixel circuit 100 are listed below.

[0061] 1. Photoreceptor module. As shown in the figure, the pixel circuit includes a photodiode PD for measuring incident light 9 and converting the light intensity into a current Iphoto, or other photosensor, a photoreceptor circuit PRC for generating a photoreceptor signal Vpr that depends on the light intensity, and a memory capacitor C1 for storing past photoreceptor signals. The photosensor PD and the photoreceptor circuit PRC constitute the photoreceptor module PR.

[0062] 2. Memory capacitor C1: The memory capacitor C1 receives the photoreceptor signal Vpr such that the first plate of the capacitor becomes charged with the charge responsive to the light received by the photosensor PD, i.e., the photoreceptor signal Vpr. The second plate of the memory capacitor C1 is connected to the comparator node (inverting input) of A1. Accordingly, the voltage Vdiff at the comparator node fluctuates with the change in the photoreceptor signal Vpr.

[0063] 3. Comparator A1: This is a means for comparing the difference between the current photoreceptor signal Vpr and the past photoreceptor signal with a threshold value. This comparator A1 may be in each pixel or shared among a subset of pixels (e.g., columns). In a preferred embodiment, the comparator is integrated with the pixel and each pixel has a dedicated comparator A1.

[0064] 4. Memory: The memory 50 stores the comparator output based on the sample signal from the controller 60. The memory can be a sampling circuit (e.g., a switch and parasitic or explicit capacitor) or a digital memory circuit (latch or flip - flop). In one embodiment, the memory is a sampling circuit and each pixel has two memories.

[0065] 5. Conditional reset circuit R1: The conditions for reset are a combination of the state of the stored comparator output and the reset signal applied by the controller.

[0066] 6. Components of the peripheral circuit: The comparator A1 and the memory 50 can be arranged in the pixel or in the peripheral circuit (outside the pixel circuit).

[0067] The peripheral circuit includes a controller 60, which applies a threshold signal to the comparator A1, sends a control signal to the memory 50, and selects the time when the conditional reset circuit R1 becomes active.

[0068] The peripheral circuit may also include a readout circuit RO, which reads the content of the memory 50, determines whether the light intensity for a given pixel has increased, decreased, or remained unchanged, and sends an output (calculated from the current memory value) to the processor.

[0069] More specifically, the comparator indicates whether the light has increased / decreased. For an Off event: if Vdiff is lower than the threshold Voff (above Vb), the comparator output is high, and this level is stored in the memory. This means that a decrease has been detected. If Vdiff is not lower than the threshold, the comparator output is low: no decrease is detected.

[0070] The only difficulty is that for an On event, a low comparator output means an increase, while a high comparator output means no change, whereas for an Off event, a high comparator output means a decrease, while a low comparator output means no change.

[0071] Therefore, the readout must know the memory content and which threshold was applied. Alternatively, as in the preferred embodiment described later, there is an inverter for On so that the memory polarity is the same for both on and off.

[0072] In one preferred embodiment of the pixel circuit 100 of the present invention, each pixel circuit 100 includes only one comparator, and thus this comparator first functions as a comparator for an On event and then as a comparator for an Off event (and vice versa).

[0073] The pixel circuit 100 and the controller 60 operate as follows.

[0074] The change in the light intensity received by the photosensor PD is converted into a change in the photoreceptor signal Vpr. When the reset circuit R1 is not conducting, the change in Vpr is also reflected in the voltage Vdiff of the comparator node at the inverting input (-) to the comparator A1. This occurs because the voltage across the memory capacitor C1 remains constant.

[0075] At the time selected by the controller 60, the comparator A1 compares the voltage (Vdiff) of the comparator node at the second terminal of the memory capacitor C1 with the threshold voltage Vb (from the controller) applied to the non-inverting input (+) of the comparator A1.

[0076] The controller 60 operates the memory 50 to store the comparator output Vcomp. The memory 50 is typically implemented as part of the pixel circuit 100 as shown. However, in other embodiments, the memory 50 is implemented as part of the column logic circuit (peripheral circuit, one for each column of the pixel array).

[0077] The state of the stored comparator output held in the memory 50 indicates a change in the light intensity. If the AND, GlobalReset signal, which is a global reset signal from the controller 60, is active, the conditional reset circuit R1 conducts. Here, "AND" represents the logical AND operator. When the conditional reset circuit R1 is in the conducting state, the voltage (Vdiff) of the comparator node at the inverting input of the comparator A1 is reset to a known level. Thus, the comparator A1 stores the current photoreceptor signal Vpr on the memory capacitor C1.

[0078] FIG. 2 shows a pixel circuit 100 according to another embodiment that includes an optional sampling circuit SC between the photoreceptor circuit PRC and the memory capacitor C1. The sampling circuit SC enables selective electrical connection or disconnection between the output Vpr of the photoreceptor circuit PRC and the memory capacitor C1.

[0079] The sampling circuit SC is actuated by a sample signal 25 from the controller 60 to globally sample the photoreceptor outputs of all pixels at the same instant in order to avoid motion artifacts. Typically, the sampling circuit SC may comprise a sampling switch 150, a parasitic or explicit sampling capacitor C2, and a buffer amplifier A2. (Note: The buffer amplifier A2 is not used in all implementations.)

[0080] The pixel circuit 100 and the controller 60 operate as follows.

[0081] A change in light intensity is converted into a change in the photoreceptor signal Vpr as in the previous embodiment of FIG. 1.

[0082] Simultaneously, the controller 60 actuates the sampling circuit SC by electrically connecting the photoreceptor output voltage Vpr from the photoreceptor circuit PRC to the sampling capacitor C2. The controller 60 further actuates the sampling circuit SC by opening the switch 150 via the control signal 25 to disconnect the photoreceptor output voltage Vpr from the capacitor C2.

[0083] At the time specified by the controller 60, the comparator A1 compares Vdiff at the comparator node with a threshold value Vb applied on its non-inverting input. Simultaneously, the controller 60 actuates the memory 50 to store the comparator output Vcomp. As before, the memory 50 is disposed either within the pixel circuit 100 or within the column logic circuit 44 of the peripheral RO circuit 42 as described later.

[0084] If the state of the stored comparator output indicates a change in light intensity and the GlobalReset, which is a global reset signal (controlled by the controller), is active, the conditional reset circuit R1 conducts and Vdiff is reset to a known level. Next, the voltage on the sampling C2 is stored on the memory C1.

[0085] As described above, a pixel or pixel array can be used as a sensor for machine vision applications. In machine vision applications, the output of the sensor will (directly or indirectly) go to a data processor, where an algorithm can calculate the position and movement of the object sensed by or through the sensor.

[0086] Conventional sensors output an image that includes pixel values representing the light intensity impinging on the pixels. In contrast, herein, the sensor outputs the address of the pixel(s) at which a change in light intensity has been detected (where the pixel address corresponds to its row and column number). This change in light intensity at a given pixel is referred to as an event. More specifically, the term "event" means that the photoreceptor signal, which represents the light intensity of the pixel and is a function thereof, has changed by an amount greater than a threshold value applied by a controller. To transmit an event, the address of the corresponding pixel is transmitted together with a 1-bit indicating whether the change in light intensity was positive or negative.

[0087] To detect the change in light intensity between the current and previous instances in time, each pixel needs to store a representation of the light intensity at the previous instance in time.

[0088] More specifically, each pixel stores a voltage Vdiff that represents the difference between the photoreceptor signal at the time of the last event emitted by this pixel and the current photoreceptor signal at this pixel.

[0089] In a preferred embodiment, to detect an event, at a comparator node, Vdiff is first compared to one threshold to detect an increase in light intensity (ON event), and the comparator output is sampled on an (explicit or parasitic) capacitor or stored in a flip-flop. Next, at the comparator node, Vdiff is compared to a second threshold to detect a decrease in light intensity (OFF event), and the comparator output is sampled on an (explicit or parasitic) capacitor or stored in a flip-flop. A global reset signal is sent to all pixels, and at each pixel, this global reset signal is logically ANDed with the sampled comparator output to reset only the pixels at which an event has been detected. Next, the sampled comparator output voltage is read out and the corresponding pixel address is sent to a receiver.

[0090] FIG. 3 shows a sensor 8 comprising a two-dimensional array of pixels 100-1 to 100-6. Only two rows and only three columns are shown for the illustrated sensor to avoid confusing the figure. In reality, sensor 8 will consist of m rows (typically much larger than 2) and n columns (typically much larger than 3) of pixels. The pixels in the two-dimensional array can be identified by their address which is the row number and column number of the pixel. For example, pixel 103-6 has row 2 (counting from the top) and column 3 (counting from the left) as its address.

[0091] Controller 60 controls pixel 100 and other components such as row selection circuit 40, readout circuit 42, and the transmission of data from the array to processor 46.

[0092] In the illustrated example, row selection circuit 40 is shown as part of controller 60. This row selection circuit 40 selects one or more subsets of rows. When the row of pixels 100 is selected, the comparator outputs of the pixels in the selected row are transmitted to readout circuit 42.

[0093] The readout circuit 42 reads data (the stored comparator output) from the pixel array. Often, the readout circuit 42 further encodes the data into a more efficient representation before transmitting it to some kind of receiver (usually in the form of a processor), such as processor 46, which may be external to the sensor chip.

[0094] The readout circuit 42, which has n columns divided into several column logic circuits 44-1 to 44-n, reads the comparator output to determine whether the light intensity for the corresponding pixel has increased, decreased, or remained unchanged.

[0095] The controller 60 preferably operates the sensor 8 in a plurality of phases (FIGS. 4A and 4B) such as those listed below:

[0096] 1. Integration of irradiation changes: In this embodiment, the integration of changes continues during other phases and does not require additional dedicated time.

[0097] 2. Comparison phase for ON events (218 in FIGS. 4A and 4B): At each pixel, comparator A1 is used to compare the integrated irradiation change against an ON threshold. The result is stored in the memory 50.

[0098] 3. Comparison phase for OFF events (220 in FIGS. 4A and 4B): At each pixel, comparator A1 is used to compare the integrated irradiation change against an OFF threshold. The result is stored in the memory 50.

[0099] 4. Reset phase (222 in FIGS. 4A and 4B): Each pixel circuit 100 where the stored comparator output state indicates a change in light intensity is reset by conducting its respective reset circuit R1.

[0100] 5. Readout phase (224 in FIGS. 4A and 4B): Under the control of the row selection circuit 40, the comparison results stored in the memory 50 are read from the pixel array.

[0101] Generally, the first phase (integration of irradiation change) does not need to have an explicit duration because the continuous time integration of the change does not stop during other phases. Therefore, it is possible to omit the explicit time allocation for this phase.

[0102] Of course, the order of comparison of ON and OFF events can be reversed. Furthermore, depending on the pixel embodiment, the readout phase can occur before the reset phase.

[0103] The controller 60 preferably controls the relative timing of the phases to generate the signals necessary to control the pixels. As shown in FIGS. 4A and 4B, several pixel operation modes are possible.

[0104] Fixed readout time: FIG. 4A is a timing diagram showing a fixed time mode in which each of the operation phases has a fixed duration. Therefore, the frame rate is fixed. In the illustrated example, each Evt frame is 100 microseconds long.

[0105] Therefore, the readout phase also has a fixed duration, but most likely the number of events to be read will vary from event frame to event frame. To maintain a reasonable limit on the duration of the readout phase, the number of events (214) that can be read in a single event frame must be limited. If the number of events to be read is less than the maximum possible value, there will be an idle phase.

[0106] If the number of events is greater than the number that can be read during the readout phase, there are three options: 1) Notify the processor 46 to discard additional events, 2) Notify the processor 46 to extend the readout phase only for the current frame (after the extended frame, the sensor, i.e., the pixel array, immediately starts the next event frame), or 3) Notify the processor 46 to extend the readout phase for the current frame and, after the extended frame, wait for the "official" frame start time to maintain synchronization with the previous event frame start time.

[0107] FIG. 4A shows a timeline that is an example of operation for a frame rate of 10k event frames per second or 100 microseconds per frame.

[0108] More specifically, in each event frame 212, the comparison 218 of ON events consumes 10 μs, and the comparison 220 of OFF events also consumes 10 μs. Next, the pixels are reset during a 20 μs phase 222. Finally, in the readout phase 224, events are read from the pixel array 210 and accumulated in the readout circuit 42. The readout phase lasts 60 μs. Thus, in this specific example, the total duration of the phases is 10 + 10 + 20 + 60 = 100 microseconds.

[0109] As shown between consecutive event frames 212-1, 212-2, 212-3, 60 μs is allocated to the phase 224, but the actual readout 214 in this phase 224 consumes varying amounts of time. For example, for the event readout 214-1 in the phase 224-1 for the first event frame 212-1, less than half of the 60 μs allocated for 214-1 is consumed. In contrast, for the third event frame 224-3, the readout of the event 214-3 consumes two-thirds of the 60 μs allocated for 224-3.

[0110] Variable readout time: In the variable readout time mode, shown in FIG. 4B, the duration of the readout phase 224 depends on the number of events to be transmitted. As a result, the total length of one frame, and thus the frame rate, is variable and depends on the number of events per frame.

[0111] More specifically, as in the previous example, each of the event frames 212 is divided into a comparison 218 of the ON event, which consumes 10 μs, and a comparison 220 of the OFF event, which also consumes 10 μs. The event reset phase 222 consumes 20 μs within each event frame 212. On the other hand, the event readout phase 224 for each of the event frames 212 is of variable length. As a result, this phase has a length of time determined by the number of events 214 that need to be read out from the pixel array 210.

[0112] FIG. 5 shows a preferred embodiment of a pixel circuit 100 constructed in accordance with the principles of the present invention without sampling.

[0113] A photosensor PD, e.g., a photodiode or a phototransistor or a photoactive region, is used to convert the incident light 9 into an electrical signal (i.e., a current designated as Iphoto, or charge). Iphoto is then converted to a voltage Vpr by a photoreceptor circuit PRC. The relationship of Vpr to the intensity of the light is typically logarithmic, but could also be linear in this and all other pixel embodiments.

[0114] Preferably, the photoreceptor module PR in each of one or more pixels of any sensor embodiment is a logarithmic photoreceptor module. A logarithmic photoreceptor module is a photoreceptor configured to convert an Iphoto current proportional to the intensity of light impinging on the sensing surface of a photosensor PD into a signal that is a logarithmic function of the detected light. It should be noted that Vpr is chosen to be logarithmic with respect to Iphoto, but Vpr could have been chosen to be proportional to Iphoto and other functions.

[0115] The logarithmic conversion of the current (iphoto) generated by the photodiode to the output voltage is very powerful because it allows mapping a wide range of input currents onto a limited voltage range. The comparison of differences in the logarithmic intensity domain also has the advantage that it is mathematically similar to the comparison of normalized differences. Most definitions of contrast are based on normalized differences (e.g., luminance ratio, Weber contrast, or Michelson contrast). The comparison of differences in the logarithmic intensity domain also allows observing differences in the reflectivity of an object independently of the background illumination.

[0116] The memory capacitor C1 receives the photoreceptor signal Vpr such that the first plate of the capacitor is charged with a charge responsive to the photoreceptor signal Vpr. The second plate of the memory capacitor C1 is connected to the comparator node A1. In the illustrated example, the second plate is connected to the inverting input of the event comparator A1. Thus, the voltage Vdiff at the comparator node varies with the photoreceptor signal Vpr and thus with the change in the light received by the photosensor PD.

[0117] During the comparison phase (see, for example, FIGS. 4A and 4B), the reset switch RS of the reset circuit R1 is non-conductive, and thus the voltage Vdiff at the comparator node is floating. Since the voltage across the memory capacitor C1 remains constant since the pixel 100 was reset, the change in the photoreceptor signal Vpr consequently changes the floating voltage Vdiff at the comparator node.

[0118] For comparison of ON events, the threshold voltage Vb is set to the value Von by the controller 60 (see FIG. 6). The event comparator A1 thus compares the threshold voltage Vb with Vdiff. The controller 60 also pulses Onsel to close the ON event sampling switch S2 (for ON selection). As a result, the comparator output is inverted by the inverter I1 and sampled on the capacitance. In the illustrated embodiment, this capacitance is the parasitic gate capacitance of the ON event output transistor NM2 of the output circuit OUT.

[0119] For comparison of OFF events, the bias voltage Vb is set to the value Voff (see FIG. 6). Thus, the event comparator A1 compares the new threshold voltage Vb with Vdiff. The controller 60 also pulses Offsel to close the OFF event sampling switch S1. As a result, the comparator output is sampled on the capacitance. In the illustrated embodiment, this capacitance is the parasitic gate capacitance of the OFF event output transistor NM1 of the output circuit OUT.

[0120] During the reset phase, the threshold voltage Vb is set to the voltage level Vreset (a value between Von and Voff, preferably a value in the middle between Von and Voff), and the GlobalReset signal is activated by the controller 60. As a result, either an ON event or an OFF event is detected (using an OR gate), and if AND, GlabalReset is active, the reset circuit R1 closes the reset switch RS.

[0121] As a result, only the pixels for which an event has been detected are reset. Advantageously, the reset function can accumulate small changes from frame to frame, enabling the detection of slow movements.

[0122] The illustrated circuit example further compensates for any offset in the event comparator A1, thereby helping to make the response of pixel 100 in pixel array 8 (FIG. 3) consistent across the array. Generally, in pixel 100 within pixel array 8 where a high voltage level is stored on either the OFF event output transistor NM1 or the ON event output transistor NM2, the reset switch RS is closed. As a result of the resulting voltage follower configuration, Vdiff at the second terminal of the memory capacitor C1 will settle to Vreset plus any offset of the comparator. As a result, the correction applied to the threshold is now compensated for the offset in event comparator A1.

[0123] During the readout phase, the pixel array is read row by row. Thus, each pixel circuit 100 waits for the controller 60 to activate its RowSelect signal, one row at a time.

[0124] Pixel control signals and their temporal variations (timeline plots) are then considered.

[0125] FIG. 6 shows the timelines of the global pixel control signals and the local pixel signals. While Vb, Onsel, Offsel, and GlobalReset are global signals for all pixels in the pixel array, RowSelect is a (local) signal in the row direction.

[0126] More specifically, two event frames with ON and OFF comparison phases are shown. Specifically, during comparison phase 218-1 and OFF comparison phase 220-1, the threshold voltage Vb is changed between the Von and Voff levels. Since the photoreceptor signal Vpr is constant, no events are detected.

[0127] During the ON comparison phase 218-2, the threshold voltage Vb is changed to Von. Since the photoreceptor signal Vpr is now at a higher level indicating an increase in the amount of light received by the photosensor PD, the voltage (Vdiff) at the second terminal of the memory capacitor C1 also increases (exceeding the level of Von if the change is large enough). As a result, the controller 60 also pulses Onsel, the ON event sampling switch S2 closes, and the ON signal is stored on the capacitance of the ON event output transistor NM2 (Figure 5). When the RowSelect signal is active, the nRxOn line is pulled low.

[0128] Since an event is detected, the reset phase also resets the voltage across the memory capacitor C1. Specifically, in the reset phase 222-2, the threshold voltage Vb is set to the intermediate level Vreset. Because PixReset is high due to the logic in the reset circuit R1, Vdiff is reset to Vreset and a new voltage is stored across the memory capacitor C1.

[0129] Figure 7 shows another pixel circuit 100 (Figure 5) having a sampling circuit SC between the photoreceptor PR and the memory capacitor C1.

[0130] This enables sampling the photoreceptor signal Vpr before comparison. This configuration ensures that the same photoreceptor voltage value is used for comparison for both ON and OFF events, avoiding motion artifacts that could occur due to changes in Vpr during the comparison between ON and OFF events.

[0131] More specifically, prior to each ON comparison phase 218, the sample line from the controller 60 is active for a period of time to close the sampling switch 150. This transfers the photoreceptor signal voltage Vpr to the plate of the memory capacitor C1. Next, the sample signal line from the controller 60 becomes inactive so that the sampling switch 150 opens again. Thus, the charge on the left plate of the memory capacitor C1 is static and does not change with subsequent changes in the photoreceptor signal Vpr. Such changes would typically result from a change in the scene or movement between the sensor and the scene.

[0132] Next, both the ON comparison phase 218 and the OFF comparison phase 220 are performed. Each comparison against a different threshold voltage Vb will then occur for the same voltage sampled from the photoreceptor signal Vpr.

[0133] FIG. 8 shows another pixel circuit 100. This design results in a smaller pixel.

[0134] More specifically, this pixel circuit has only one output line nRX, which replaces the two output lines nRXon and nRXoff employed in the previous examples (FIGS. 5 and 7). This change also allows for the removal of the OR gate (see FIGS. 5 and 7) in the reset circuit R1, as well as the removal of one of the output transistors (see FIGS. 5 and 7) in the NM2, output circuit OUT. For both event polarities, the output signal nRx is active low. The reset circuit R1 uses the GlobalReset signal and the sampled comparator output to determine whether it is conducting.

[0135] In this embodiment, it is preferred to have separate reset phases for each event polarity (i.e., one reset phase for OFF events and a different reset phase for ON events).

[0136] More specifically, similar to the previous embodiment, the change in the photoreceptor output voltage Vpr changes the Vdiff of the floating node.

[0137] For comparison of the ON event, the bias voltage Vb is set to the voltage level Von. The comparator compares Vb with Vdiff. By pulsing OnSel, the comparator output is sampled on the parasitic capacitance of the transistor NM1.

[0138] For reset due to the ON event, Vb is set to the voltage level Vreset (at the center between Von and Voff), and the GlobalReset signal is activated. In the pixel where the high voltage level is stored on the output transistor NM1, the reset switch RS is closed, and as a result, Vdiff will settle to Vreset plus the offset of the comparator.

[0139] For row-by-row readout of the ON event, the RowSelect signal is activated for one row at a time. If the gate capacitance of the output transistor NM1 stores a high voltage level, the output transistor NM1 is conducting, and the corresponding request line nRx is pulled down. This active-low request is latched in the peripheral readout circuit 42.

[0140] For comparison of the OFF event, the bias voltage Vb is set to the level Voff by the controller 60. By pulsing OffSel, this new comparator output is sampled on the parasitic capacitance of the output transistor NM1.

[0141] For the reset OFF event, Vb is set to the voltage level Vreset (at the center between Von and Voff), and the GlobalReset signal is activated. In the pixel where the high voltage level is stored on the gate of the output transistor NM1, the reset switch RS is closed, and as a result, Vdiff will settle to Vreset plus the offset of the comparator.

[0142] For the reading of OFF events, which occurs on a per-row basis, the RowSelect signal is activated for one row at a time. If the gate capacitance of output transistor NM1 holds a high voltage level, it is conducting and the output line nRx is pulled down. This active-low request is latched in the peripheral readout circuit 42.

[0143] FIG. 9 shows the timelines of the global pixel control signal and the local pixel signal. Vb, Onsel, Offsel, and GlobalReset are global signals, and RowSelect is a (local) signal in the row direction.

[0144] In this example, the ON comparison phase is combined with the readout phase 218. During this time, the threshold voltage Vb increases to Von, and RowSelect is active. However, in the illustrated phase example 218-1, the ON event is not detected. Similarly, the OFF comparison phase is combined with the readout phase 220. During the ON reset phase 219 and the OFF reset phase 221, the GlobalReset signal is active.

[0145] Before the ON comparison and readout phase 218-2, Vpr increases due to the increase in light on the photosensor PD, and as a result, Vdiff increases to a value greater than Von. Therefore, the event comparator A1 registers the ON event, and when OnSel goes high and then low again, and thus PixEvt goes high, this is stored in the memory. When the row selection signal RowSelect is active, the ON event is communicated to the peripheral circuit on the output line nRx. During the ON reset phase 219-2, while PixEvt and GlobalReset are both high, PixReset goes high so that the pixel is reset during the reset phase 219-2.

[0146] FIG. 10 shows another pixel circuit 100. This design results in even smaller pixels where the memory function is not placed in the pixel circuit 100 but instead is part of the readout circuit 42.

[0147] More specifically, the reset circuit R1 uses the RowSelect signal and the output acknowledge signal ColAck from the readout circuit 42 to determine whether to close the reset switch RS to reset the memory capacitor C1. The logical AND of ColAck and RowSelect is used to set a latch to store the PixReset signal, and the global signal (ResetPixReset) from the controller is used to reset the latch during the reset phase. The reason for saving the AND of ColAck and RowSelect and not using it directly to control the switch RS is that all the pixels in the array can be reset simultaneously. Without the latch, a reset would have to occur row by row during the readout phase.

[0148] The ColAck signal from the readout circuit 42 is shared among all the pixels in a column. Therefore, the pixel reset has to be made active row by row. During operation, the row selection circuit selects one row of pixels by activating the corresponding RowSelect, and the corresponding comparator output is transmitted to the readout circuit via the output transistor NM1. Next, the controller 60 activates the memory in the readout circuit 42 to save the transmitted comparator output, and the column logic circuit in the readout circuit 42 determines whether there has been an increase or decrease in light intensity. In the columns where the column logic circuit detects a light intensity change, the ColAck signal is made active. The controller then applies the reset voltage Vreset to the positive input of the event comparator A1 of the array 210. Together with the still active RowSelect signal, the active ColAck signal resets the corresponding pixels.

[0149] In addition, in this pixel circuit, the polarity of the comparator output differs between an ON event (increasing light level) and an OFF event (decreasing light level). As a result, the difference in the polarity of the comparator output for the ON and OFF events is taken into account in the readout circuit 42. This means that the output signal nRx is active high for the ON event and active low for the OFF event.

[0150] FIG. 11 shows the timelines of the global pixel control signal and the local pixel signals. While Vb, Onsel, Offsel, and ResetPixReset are global signals, RowSelect is a local signal in the row direction and ColAck is a local signal in the column direction.

[0151] In this example, the ON comparison phase is combined with the readout phase 218. During this time, the threshold voltage Vb increases to Von and RowSelect is active. However, in the illustrated phase example 218-1, the ON event is not detected. Similarly, the OFF comparison phase is combined with the readout phase 220. After the two comparison and readout phases, there is a reset phase 224.

[0152] Before the ON comparison and read phase 218-2, Vpr increases due to the increase in light on the photosensor PD, and as a result, Vdiff increases to a value greater than Von. Therefore, when the event comparator A1 registers an ON event and the row select signal RowSelect is active, this is communicated to the peripheral circuit on the output line nRx. To register the event by the corresponding column logic circuit, this column logic circuit will activate the ColAck signal. In all pixels where ColAck and RowSelect are simultaneously active, the PixReset signal goes high, short-circuiting the input and output of the comparator. During the reset phase 224-2, the controller applies Vreset to Vb, and since PixReset is still high, the pixel is reset. Next, the controller sets ResetPixReset high for a while to return all PixReset to a low value.

[0153] Figure 12 shows another pixel circuit 100. This design allows for faster operation. This design includes two event comparators per pixel that allow for ON event and OFF event comparisons to occur simultaneously. Additionally, the memory is placed within the pixel circuit and is implemented by two sampling circuits at the corresponding comparator outputs. As in FIG. 5, the memory is a combination of switches with parasitic capacitances, here the parasitic gate capacitances of S4 and NM2, as well as the parasitic gate capacitances of S5 and NM1. The comparator output is sampled and stored on the parasitic gate capacitances of the two output transistors NM1 and NM2.

[0154] More specifically, the OFF event comparator A1 receives the Voff threshold voltage provided to the entire pixel array 210. Similarly, the ON event comparator A2 receives the Von threshold voltage provided to the entire pixel array 210.

[0155] When the sample comparison signal SampleComp from the controller 60 is active, the outputs of comparators A1 and A2 are transferred to the gate capacitances of the OFF event output transistor NM1 and the ON event output transistor NM2, respectively. Next, their states are read via the output lines nRxon and nRxoff when RowSelect is active (i.e., NM3 is conducting).

[0156] In response to an ON event or an OFF event, the reset circuit resets the voltage (Vdiff) at the second terminal of the memory capacitor C1, which is supplied to the inverting input of the OFF event comparator A1 and the non-inverting input of the ON event comparator A2. In this case, Vdiff is reset to a common voltage across the array 210.

[0157] FIG. 13 shows two representative pixels 100 along a column from the array 210. The ellipses (dots) along the ResetLevel, ColAck, and Vsf lines indicate the presence of other pixels along the column. They are omitted from the figure to avoid confusion. The pixel circuit embodiment of this figure results in even smaller pixels. The reason is that the comparator function is not placed within the pixel circuit 100 but is instead part of the readout circuit 42 (shown at the bottom of the figure).

[0158] This pixel circuit 100 includes a sampling switch 150 between the photoreceptor circuit PRC, whose output is Vpr as considered for other embodiments, and the memory capacitor C1. This enables a selective electrical connection between the output of the photoreceptor circuit PRC and the memory capacitor C1. The sampling switch 150 is actuated by the sample signal Sample on line 25 from the controller 60. The sample signal is activated by the controller 60 to globally sample the photoreceptor outputs of all the pixels 100 in the array 210 simultaneously. This avoids motion artifacts.

[0159] Specifically, the photoreceptor signal Vpr is transferred as Vprs to the memory capacitor C1. Buffer 27 then holds the voltage (Vdiff) at the second terminal of the memory capacitor C1. Generally, buffer 27 transfers the voltage (Vdiff) at the second terminal of the memory capacitor C1 to the peripheral circuit. The buffer is enabled by the RowSelect signal coming from the row selection circuit 60.

[0160] Pixel circuit 100 uses a source follower as buffer 27. This transfers the voltage on the second terminal of the memory capacitor C1 to the readout circuit 42 for the columns of pixels in array 210. Voltage Vdiff is provided to terminal Vsf of readout circuit 42 on line Vout to the readout circuit. There, event comparator circuit A1 compares Vdiff with both the Von level and the Voff level. This column comparator A1 is arranged in readout circuit 42.

[0161] In buffer 27, transistor M1 functions as a unity gain source follower input transistor (the current source of the source follower is part of readout circuit 42), while M2 is a switch for enabling the source follower. Readout circuit 42 is implemented such that there is a separate ColAck signal for each column of pixels. The ColAck signal is active only in the column where an event is detected (FIG. 37 shows such a readout circuit 42).

[0162] Reset transistor N1 is controlled by the RowSelect signal. Reset transistor N2 is controlled by the output acknowledge signal ColAck. Thus, when both of these signals are active, reset transistors N1 and N2 are conducting, and the voltage (Vdiff) at the second terminal on memory capacitor C1 is reset to the reset voltage supplied to pixel column 210 on the reset line ResetLevel from readout circuit 42. The series connection of transistors N1 and N2 thus forms a logical AND function for signals RowSelect and ColAck.

[0163] It should be understood that this embodiment could also be implemented without the sample switch 150 between the photoreceptor circuit PRC and the memory capacitor C1.

[0164] FIG. 14 shows two representative pixels 100 along a column from the array 210. As in FIG. 13, the ellipses (dots) along the ResetLevel, ColAck, and Vsf lines are omitted to avoid cluttering the figure and indicate the presence of other pixels. This design also focuses on a small pixel area. As in the previous embodiment, the comparator function is not located in the pixel circuit 100 but is instead part of the readout circuit 42. However, unlike the embodiment of FIG. 13, this embodiment uses a capacitive amplifier in the buffer 27 to amplify the voltage Vdiff at the second terminal of the memory capacitor C1, specifically, the change in voltage since the pixel was last reset.

[0165] Vdiff is obtained from Vprs as before (FIG. 13) before amplification. Amplifying the change in Vdiff makes it easier to detect small changes in the photoreceptor signal Vpr. The gain of the buffer 27 is given by the capacitance ratio of C1 divided by C2. It should be noted that capacitor C2 may be explicit or parasitic. Except for this increase by the capacitance ratio, the pixel embodiment and functionality shown in FIG. 14 are the same as those of FIG. 13.

[0166] In buffer 27, transistor M1 is a PFET input transistor, so the buffer is an amplifier, while M2 is a switch to enable the amplifier. The readout circuit 42 is implemented such that there is a separate ColAck signal for each column of pixels. The ColAck signal is active only in the column where an event is detected. FIG. 37 shows such a readout circuit 42.

[0167] The reset of node Vdiff is controlled by a row selection switch RS controlled by the RowSelect signal and a column reset switch CS controlled by the output acknowledge signal ColAck.

[0168] (Together with FIG. 35) In FIG. 13, the reset level is determined by A1 in the source follower and the readout circuit. The actual reset level includes the offset of the source follower and the offset of A1, and thus both of these offsets will be compensated.

[0169] (Together with FIG. 37) In FIG. 14, A1 is implemented as two comparators (one for ON and one for OFF), and thus both of them cannot be included in the reset level. However, since the buffer in FIG. 14 has a gain much larger than 1, the offset of the comparator is not actually much of a problem and thus may remain uncompensated.

[0170] Similarly to FIG. 13, FIG. 14 could also be implemented without the sample switch 150 between the photoreceptor circuit PRC and the memory capacitor C1.

[0171] Next, the discussion moves on to taking up possible examples of aspects / parts of the above pixel circuit embodiments. It should be understood that, generally, any of the above circuits may have any one or more of the features described below.

[0172] FIG. 15 shows a compact embodiment of a sampling circuit SC that can be used in pixel embodiments (FIGS. 2, 7, 13, and 14). The sampling circuit SC uses an nFET transistor T10 as a switch and a source follower implemented by two pFET transistors T11 and T12, where T11 is the current source of the source follower and T12 is the input transistor of the source follower. The gate capacitance of the pFET transistor T12 forms the sampling capacitor.

[0173] Figures 16 and 17 show various embodiments of comparator A1 used to detect small changes in irradiation. It is necessary to detect small voltage changes on the order of several millivolts. This means that the comparators shown in the circuits of FIGS. 1-3, 5, 7, 8, 10, 12-14 will require a fairly large gain (preferably greater than 10 dB or 20 dB, ideally about 40 dB or more).

[0174] Figure 16 shows a two-stage comparator that will provide the required gain. A reset switch RS is also shown, and in this embodiment, the reset switch RS does not connect the input and output of the comparator. Instead, the input is connected to the output of the first stage of the comparator.

[0175] Figure 17 shows another implementation of comparator A1 based on an operational transconductance amplifier. This amplifier uses five transistors along with the output stage of two transistors as a comparator. This amplifier has the advantage that the speed of the comparator does not depend on the reference voltage, and as a result, there is more freedom in the threshold range. Furthermore, the offset compensation is considered to be better than using one of the previous examples with a two-stage comparator of two transistors.

[0176] The reset switch and the reset circuit are part of the pixel circuit in all embodiments. Those embodiments follow below.

[0177] The reset switch can be implemented as an NMOS transistor, a PMOS transistor, or a complete transmission gate including NMOS and PMOS transistors. Depending on the type of switch used, the polarity of the reset signal is active high for an NMOS transistor, active low for a PMOS transistor, and both high and low polarities for a transmission gate. When the reset signal is called PixReset, the polarity is considered active high, and when the reset signal is called nPixReset, the polarity is considered active low.

[0178] Figure 18 shows an implementation of a reset circuit for the pixel circuits in FIGS. 5, 7, and 12. In the circuits shown in FIGS. 5, 7, and 12, an AND-OR combination is used for reset signal generation. The AND-OR combination circuit shown in FIG. 18 can be implemented for the NMOS transistor that functions as a reset switch. ON and OFF are the sampled comparator outputs. For example, if ON is at a high voltage level, switch S1 is conducting while S3 is non-conducting. The voltage on PixReset thus follows the voltage of GlobalReset in this case. When the controller sets GlobalReset to a high voltage level, the reset NMOS transistor is conducting and the comparator is reset.

[0179] If both ON and OFF are low, neither S1 nor S2 is conducting, but both S3 and S4 are conducting. PixReset is thus connected to ground, and as a result, the reset transistor is non-conducting.

[0180] Figure 19 shows an alternative implementation using a PMOS reset transistor. For example, if the voltage level ON is high, NM1 is conducting. When the controller 60 sets GlobalReset to a high voltage level, NM3 is also conducting and the voltage on nPixReset is pulled to ground, and thus the PMOS reset transistor is conducting. When the controller 60 sets GlobalReset to a low voltage level, there is no longer a current path between nPixReset and ground. Next, the bias current in PM1 (controlled by the bias voltage on the gate of PM1) slowly pulls nPixReset to the power supply voltage. If neither ON nor OFF is at a high voltage level, the bias current in PM1 maintains nPixReset at the power supply voltage, and thus the reset transistor is non-conducting.

[0181] Figure 20 shows yet another implementation of the reset circuit. This version incorporates the logic function into the actual reset switch. This allows for a more compact implementation. Either ON or OFF is at a high voltage level, and if GlobalReset (controlled by controller 60) is high, the path between the comparator inputs is conducting.

[0182] Figure 21 shows an implementation of the reset circuit for the pixel embodiment of FIG. 8. Here, an NMOS transistor functions as the reset switch. PixEvt is the sampled comparator output. If PixEvt is at a high voltage level, switch S1 is conducting, while S2 is not. The voltage on PixReset thus follows the voltage of GlobalReset in this case. When the controller sets GlobalReset to a high voltage level, the reset NMOS transistor is conducting and the comparator is reset.

[0183] If PixEvt is low, S1 is not conducting but S2 is conducting. PixReset is thus connected to ground, and as a result, the reset transistor is not conducting.

[0184] Figure 22 shows an implementation of the reset circuit. This circuit uses a PMOS reset transistor. If the voltage level PixEvt is high, NM1 is conducting. When the controller sets GlobalReset to a high voltage level, NM2 is also conducting and the voltage on nPixReset is pulled to ground, and as a result, the PMOS reset transistor is conducting. When the controller sets GlobalReset to a low voltage level, there is no longer a current path between nPixReset and ground. Next, the bias current in PM1 (controlled by the bias voltage on the gate of PM1) slowly pulls nPixReset to the power supply voltage. If PixEvt is at a low voltage level, the bias current in PM1 maintains nPixReset at the power supply voltage, and as a result, the reset transistor is not conducting.

[0185] FIG. 23 shows a reset circuit that incorporates the logic function into an actual reset switch. This enables a more compact implementation. The path between the input and output of the comparator is conductive if PixEvt is at a high voltage level and GlobalReset (controlled by the controller) is high.

[0186] FIG. 24 shows another implementation of a reset circuit that is compatible with the pixel circuit shown in FIG. 10.

[0187] Here, when both RowSelect and ColAck (ColAck is a signal from the column logic circuit to the pixel in FIG. 10) are high, node nPixReset is pulled to ground, and this low voltage level is stored on capacitor CR (explicit or parasitic), as a result, the switch connecting the input and output of the comparator is conductive. After the readout is completed for the entire array, all pixels that generated an event will have grounded nPixReset and thus be reset. Next, the controller will set the signal ResetPixReset to a low voltage level, MP1 is conductive, and pulls nPixReset to the power supply voltage. By controlling the level of ResetPixReset during the reset phase, the rising slope of PixReset can be controlled.

[0188] In the embodiment shown in FIG. 24, a latch for the PixReset signal is implemented using a capacitor (the node is not always driven). Instead, as shown in FIG. 25, two cross-coupled inverters can be used as a latch.

[0189] In a preferred embodiment, the pixel uses a logarithmic front end to enable high dynamic range and sensitivity to temporal contrast rather than temporal difference. By adopting a negative feedback circuit, a fast response to changes in irradiation can be achieved.

[0190] Figure 26 shows a basic logarithmic photoreceptor PR with feedback. This photoreceptor uses a photodiode PD as a photosensor. The photoreceptor circuit PRC includes an inverting amplifier and a circuit element M1 having a logarithmic current-voltage relationship connected between the input and output of the inverting amplifier. The inverting amplifier ensures that the voltage across the photodiode PD remains almost constant.

[0191] Figure 27 shows a preferred embodiment of the photoreceptor PR. An NMOS transistor is used as a feedback element, and a common-source amplifier is used as an inverting amplifier. A source follower may be used between the logarithmic photoreceptor and a capacitor to isolate the front end from voltage transients during pixel reset. The source follower also enables additional low-pass filtering of the input signal, thereby reducing the integration noise.

[0192] Figures 28 and 29 show two further options using two NMOS feedback transistors (Figure 28) or PMOS feedback transistors (Figure 29).

[0193] In the pixel readout circuit RO shown in Figure 5 or Figure 12, the column request lines nRxOn and nRxOff are shared among all pixels in the same column. When RowSelect is active and a high voltage is stored on the parasitic capacitor of NM2, nRxOn is pulled low to signal an ON event to the data readout circuit. When a low voltage is stored on NM2, NM2 is non-conductive, and thus nRxOn remains high.

[0194] For the pixel circuits shown in FIGS. 8 and 10, the column request line nRx of the readout circuit RO is shared among all the pixels in the same column. When RowSelect is active and a high voltage is stored on the parasitic capacitor of NM1, nRx is pulled low to signal the event to the data readout circuit. When a low voltage is stored on NM1, NM1 is non-conductive, and thus, nRx remains high.

[0195] The controller 60 generates the waveforms necessary for the control signals to the pixels and controls the voltage Vb at the input of the comparator to generate the waveforms necessary for controlling the column logic circuit. In many cases, the controller 60 further synchronizes these waveforms to an external timing reference.

[0196] The controller 60 can be integrated on the same sensor integrated circuit (IC) as the pixel circuit or, for example, in a separate IC using a microcontroller or a field programmable gate array (FPGA). The controller can be implemented by a finite state machine or using a microcontroller core.

[0197] A part of the controller 60 is the row selection circuit. The row selection circuit selects and enables the OUT in each pixel in one row by a set of RowSelect signals. The row selection circuit has a clock input that enables it to move from one row to the next. The output of the row selection circuit is a set of RowSelect signals, one for each row of pixels. Selecting a row means that the RowSelect signal for that row is active (high voltage level), while the RowSelect signals for all other rows are inactive (low voltage level). The active RowSelect signal enables the generation of a column request by changing the state of the signal line shared among all the pixels in the column where the pixel with the "high" comparator output is sampled.

[0198] The row selection circuit includes a circuit that encodes the address of the currently selected row and outputs this address to the data readout circuit.

[0199] The row selection circuit may include being configured in a way to skip rows during scanning. This feature is used to perform so-called region-of-interest (ROI) readout.

[0200] The controller 60 is implemented using software or hardware such as a finite state machine that first sets Vb (the first comparator input) to the first threshold voltage (Von), and then, after a short delay, sets the signal OnSel to logic high to electrically connect the inverted comparator output to the ON node. Next, after another delay, the controller (hardware or software) sets OnSel to logic low to disconnect the inverted comparator output from the ON node. Next, the controller sets Vb to the second threshold voltage (Voff). After a short pause, the software sets the signal OffSel to logic high to electrically connect the comparator output to the OFF node. After another short delay, the software sets OffSel to logic low to disconnect the comparator output from the OFF node. Next, the software sets Vb to the reset voltage Vreset. After a certain delay, the software sets the signal GlobalReset to logic high to electrically connect the second terminal of the capacitor to the reset level. After another delay, the software sets GlobalReset to logic low to disconnect the second terminal of the capacitor from the reset level.

[0201] Next, the controller sets the first RowSelect line to logic high to connect the stored comparator output of the pixels in the first row to the readout circuit, and sends a signal to the readout circuit to start transmitting events from this first row. When the readout circuit finishes, the controller sets the first RowSelect line to logic low and sets the second RowSelect line to logic high.

[0202] The process is performed until all lines are exhausted, i.e., until the controller reads the comparator outputs stored for all lines.

[0203] When all the stored comparator outputs have been read, the controller may restart the sequence by resetting Vb to the first threshold voltage again after a short delay. This process is repeated during data acquisition by DVS.

[0204] Instead of restarting the sequence directly, the controller may wait for an external timing reference signal. This external timing reference signal may be derived from a processor.

[0205] The readout circuit is described below.

[0206] In its most basic form, the readout circuit reads the comparator outputs for all pixels 100 in the array 210 and sends a three - valued (increasing, decreasing, or no change) image to the receiver once per frame.

[0207] FIG. 30 shows a readout circuit for the pixel array 210.

[0208] To read the comparator outputs for the entire pixel array 210, the array 210 is scanned row by row. This means that the row selection circuit 40 (part of the controller 60) selects one row of pixels, and the outputs of the comparators in these pixels (or, depending on the pixel embodiment, the stored outputs of the comparators) are connected to the column lines leading to the corresponding column logic circuits 44 of the readout circuit 42. The column logic circuits 44 determine whether there has been a change in the corresponding pixels, and then the outputs of each column logic circuit 44 are scanned using the column scan circuit 48. The column scan circuit 48 continuously connects the outputs of the column logic circuits to the output data lines leading to the processor 46.

[0209] Event - based readout is described next.

[0210] To enable more efficient reading and processing in the processor 46, the data readout circuit can encode the data in a more efficient way. In this type of pixel circuit, the data is expected to be sparse, which means that only a small percentage of the pixels per event frame register a change.

[0211] The sparse digital signals can be easily further compressed. A popular method for compressing multi-dimensional digital data is to encode the coordinates / addresses of the digital signals in this data. This encoding of digital events is also known as event-based reading. A popular encoding scheme in event-based vision sensors is to encode the digital signals as a tuple of the row and column coordinates of the pixels in the array as well as the timestamp, which leads to encoding the location and time of the occurrence of the digital event. In one embodiment, this means that only the addresses of the pixels where a light intensity change was detected (an event occurred) are transmitted.

[0212] FIG. 31 shows a readout circuit that enables reading from the addresses of the pixels where an event occurred. For this purpose, not only the addresses of all the columns where an event was detected by the column logic circuit, but also the addresses of the corresponding rows are output one after another.

[0213] This implementation employs a shift register 70. Each column of pixels has one corresponding shift register stage 72-1, 72-2, 72-3.

[0214] Each shift register stage 72-1, 72-2, 72-3 can be bypassed. The bypass is controlled by the EventDetect output of the respective column logic circuits 44-1, 44-2, 44-3. If the EventDetect output is low, the corresponding shift register stages 72-1, 72-2, 72-3 are bypassed. If the EventDetect output is high, the shift register stages 72-1, 72-2, 72-3 are not bypassed.

[0215] The controller 60 sets startPulse high and starts reading events by pulsing the clock. This is received by the input multiplexer 74-1 of the first stage 72-1. The input multiplexer provides the d input to the D latch 78-1. The output Q of the D latch 78-1 is provided to the output multiplexer 76-1. The clock input is received at the clock input of the D latch 78-1.

[0216] Next, the controller sets startPulse low again. The first shift register stage that is not bypassed (e.g., the corresponding EventDetect is high) will have saved a high voltage level at its output. This connects the Address+EventPolarity of the corresponding column logic circuit to the communication bus. The receiver can now read this address. On the next pulse of the clock, the high level moves to the next shift register stage 72 that is not bypassed, and the corresponding Address+EventPolarity is connected to the communication bus. This continues until the high level moves to the last shift register stage 72 that is not bypassed. The output of this stage uses line 80 to tell the controller 60 that the read for this row is complete. The controller 60 then activates the RowSelect signal for the next row, connects the row address encoder 40 to the communication bus, and restarts the shift register.

[0217] The column logic circuit is then described as follows.

[0218] Referring to FIG. 32, for pixel outputs nRxOn and nRxOff (see FIGS. 5, 7, or 12) or nRX only (see FIG. 10), depending on the embodiment of the pixel circuit, the column logic circuit 44 checks whether the light intensity in the corresponding pixel has remained unchanged, increased, or decreased by checking whether the state of the pixel output (logic low or logic high) corresponds to a value representing an increase or decrease. If the column logic circuit 44 detects an increase or decrease, it notifies the data readout circuit.

[0219] The output of the column logic circuit is a signal (EventDetect) that is active when an event is detected, a signal corresponding to the polarity of the event, and a number encoding the column address of the corresponding column. (High EventPolarity means an increase in light intensity). Since the column address is just a fixed number for each column, its implementation is not shown in the figure.

[0220] Part of the implementation of the column logic circuit is the bias transistors (92, 94 in FIG. 32) for each request line. Those bias transistors keep the request lines (nRxOn and nRxOff) at the logic high level unless the pixel pulls them on.

[0221] Event occurrence is signaled when one of the two request lines (nRxOff or nRxOn) is at a low voltage. The column logic circuit NANDs the two request lines to a flip - flop 96 at the time given by the latch clock signal from the controller 60 and stores the state of nRxOff in a flip - flop 98.

[0222] An implementation of the column logic circuit for the pixel circuit of FIG. 8 is shown in FIG. 33.

[0223] Here, the event is detected when either the request line nRX is low when Von is applied to the pixel comparator or the request line is low when Voff is applied to the pixel comparator. The inverted state of the request line is stored in a flip-flop by a command of the controller. EventDetect is the logical OR of the flip-flop outputs.

[0224] An implementation of the column logic circuit for the pixel circuit of FIG. 10 is shown in FIG. 34.

[0225] In addition to the output signals EventDetect and EventPolarity, the column logic circuit for the pixel embodiment of FIG. 10 must generate signals going to the pixels and reset the pixels after the detection of an event together with the RowSelect signal. In the proposed implementation, this ColAck signal is equivalent to the EventDetect signal.

[0226] The event is detected when the request line nRX is high when Von is applied to the pixel comparator or when the request line is low when Voff is applied to the pixel comparator.

[0227] Since the pixel circuit embodiment of FIG. 10 does not include a memory unit in the pixel, the column logic circuit includes a memory unit.

[0228] An implementation of the column logic circuit for the pixels of the circuit shown in FIG. 13 is shown in FIG. 35.

[0229] Here, the column logic circuit includes a comparator A1. The pixel outputs a signal representing Vdiff, and the comparison for detecting an event is performed in the column logic circuit 44.

[0230] The reset of the pixel is performed using the comparator / amplifier Comp1. When the pixel is to be reset, the EventDetect signal in the matching column is high. The controller 60 sets Vb to Vreset, sets DoReset high, and sets ColAck high. In the row where RowSelect_m is high, since both transistors in the matching R1 are conducting, the voltage Vdiff is then shorted to Vcomp. The feedback circuit formed by the amplifier Comp1 (A1) and the source follower then ensures that Vdiff settles to a voltage that makes Vsf equal to Vreset.

[0231] FIG. 36 shows a signal timeline for the implementation of the pixel circuit shown in FIG. 13 and the column logic circuit of FIG. 35.

[0232] An implementation of the column logic circuit for the pixel circuit of FIG. 14 is shown in FIG. 37.

[0233] The column logic circuit includes two comparators A1-1 and A1-2. The two comparators enable Vout to be compared with both thresholds at once. In a signal latch controlled by the controller 60, the comparator outputs are stored in flip-flops 112, 114. The logical OR of the comparator outputs, together with the signal DoReset from the controller 60, determines whether the pixel is reset by the signal CoAck.

Claims

1. A photoelectric conversion region, a first circuit coupled to the photoelectric conversion region, a first capacitor coupled to the first circuit, a comparison circuit coupled to the first capacitor, the comparison circuit selectively comparing a first signal based on an output from the photoelectric conversion region with both a first reference voltage and a second reference voltage, and including a first terminal for receiving the first signal and a second terminal for selectively receiving the first reference voltage and the second reference voltage, a reset circuit for resetting the first capacitor based on an output from the comparison circuit and a global reset signal, An image device comprising the above.

2. The device according to claim 1, wherein the output from the comparison circuit includes an on-event signal and an off-event signal.

3. The device according to claim 1, wherein the comparison circuit outputs an on-event signal during an on-comparison phase and outputs an off-event signal during an off-comparison phase.

4. The device according to claim 1, wherein the first reference voltage is coupled to the second terminal during the on-comparison phase.

5. The device according to claim 4, wherein the second reference voltage is coupled to the second terminal during the off-comparison phase.

6. The device according to claim 5, wherein the second terminal receives a third reference voltage during a reset phase.

7. The device according to claim 6, further comprising a controller circuit for controlling switching of the first reference voltage, the second reference voltage, and the third reference voltage.

8. The device according to claim 1, wherein the reset circuit includes a logical AND operator.

9. The device according to claim 1, further comprising a memory circuit for storing the output from the comparison circuit.

10. The device according to claim 9, further comprising a read circuit for controlling the memory circuit to read out the output from the comparison circuit.

11. The device according to claim 1, wherein the first circuit includes a logarithmic circuit.

12. An image device including an array of pixels, each of the pixels comprising: a photoelectric conversion region, a first circuit coupled to the photoelectric conversion region, a first capacitor coupled to the first circuit, A comparison circuit coupled to a first capacitor, selectively comparing a first signal based on an output from the photoelectric conversion region with both a first reference voltage and a second reference voltage, including a first terminal for receiving the first signal and a second terminal for selectively receiving the first reference voltage and the second reference voltage. A reset circuit for resetting the first capacitor based on an output from the comparison circuit and a global reset signal. An image device including the above.

13. The device according to claim 12, wherein the output from the comparison circuit includes an on-event signal and an off-event signal.

14. The device according to claim 12, wherein the comparison circuit outputs an on-event signal during an on-comparison phase and an off-event signal during an off-comparison phase.

15. The device according to claim 12, wherein the first reference voltage is coupled to the second terminal during the on-comparison phase, and the second reference voltage is coupled to the second terminal during the off-comparison phase.

16. The device according to claim 15, wherein the second terminal receives a third reference voltage during a reset phase.

17. The device according to claim 16, further comprising a controller circuit for controlling the switching of the first reference voltage, the second reference voltage, and the third reference voltage.

18. The device according to claim 12, wherein the reset circuit includes a logical AND operator.

19. The device according to claim 12, further comprising a memory circuit for storing the output from the comparison circuit and a read circuit for controlling the memory circuit to read out the output from the comparison circuit.

20. The device according to claim 12, wherein the first circuit includes a logarithmic circuit.

Images