Delta image sensor with externally recordable digital pixel storage

Abstract

The present invention relates to a delta image sensor comprising a pixel array and a plurality of acquisition circuits corresponding to at least one pixel and formed as part of an integrated circuit. Each of the acquisition circuits comprises: at least one sensor circuit comprising the photosensor configured to generate a sensor signal VSIG according to a light signal applied to the photosensor of the at least one pixel; at least one analog-to-digital converter (A / D) circuit configured to convert the current VSIG into a digital signal; at least one digital storage circuit configured to store a representation of at least one digital signal corresponding to a previous VSIG; at least one digital comparator circuit configured to compare the level of the stored representation with the current VSIG to detect whether a changed level exists; and at least one digital output circuit configured to generate an event output under the condition that the changed level exists. The digital representation may be externally recorded to the digital storage circuit of the at least one pixel.

Description

Technology Field

[0001] The present invention relates to an image sensor comprising a plurality of pixel circuits. Background Technology

[0002] Image sensors with one ADC per pixel or per small subset of pixels are well known. These are typically used in applications where critical processing is performed at the pixel level (e.g., X-ray particle tracking or satellite imaging).

[0003] Image sensors having a single-slope ADC in which each pixel includes a comparator are known, for example, Reckleben et al. (2015), Suarez et al. (2010), and Chi et al. (2010). In particular, Suarez et al. generate an analog slope and a digital code for the entire pixel array and provide the digital code to the pixel.

[0004] An image sensor is not known in which digital values ​​are stored within pixels and compared between samples to generate event change data. The present invention enables a region-optimized small pixel structure that generates a high-resolution and cost-optimized sensor.

[0005] An image sensor in which a representation of a previous light application level is digitally stored in a pixel and compared between samples to generate event change data is known in WO 2014 / 174498 A1, and Itoh et al., who published “4-Layer 3-D IC with a function of parallel signal processing” in Microelectronic Engineering, vol 15, no. 1-4, 1991, pages 187-190, previously proposed a conceptually related structure with simple single-bit pixel comparison for a fixed level.

[0006] Image sensors that generate events using image pixel-level change detection, storage, and comparison in the analog domain are known, for example, Chi et al. published "CMOS Camera With In-Pixel Temporal Change Detection and ADC" in the IEEE Journal of Solid-State Circuits, vol. 42, no. 10, 2007, pages 2187-2196. Comparison and storage in the analog domain offer advantages (e.g., compact realization) and disadvantages (e.g., gradual storage degradation including "hot pixels," sensitivity to inconsistencies, and reduced flexibility of the opto-electric conversion function) compared to digital approaches.

[0007] Image sensors that share circuits between pixels are well known, for example, KR 2020-0029328 A.

[0008] The use of stroboscopic effects is generally known to suppress harmonics by averaging or synchronizing during a disturbance (unwanted signal) period, for example, in US 2015 / 358570 A1, it is applied to a camera to detect and synchronize a variable light source and then suppress the response to the light source.

[0009] WO 2020 / 080383 A1 discloses an event-based dynamic vision sensor having a filter that ignores an event from one pixel based on the event state (single bit) of an adjacent pixel.

[0010] "Frame-free dynamic digital vision" by T. Delbruck, published in the Proceedings of the International Conference on Secure Life Electronics, Advanced Electronics for Quality Life and Society, 2008, pp. 21–26, discloses filtering events from a dynamic vision sensor based on the timestamp of the last event of a pixel or the timestamp of an adjacent pixel. This processing is performed outside the pixel array, where events must be transmitted out of the array for sequential filtering, in a computationally expensive manner (32-bit timestamp).

[0011] EP 3 313 064 A1 discloses an analog storage dynamic vision sensor, wherein one comparator input is a representation of a previous value of pixel light application and is subsequently used for comparison. This can be considered as analog feedback from a previous level to control a sequential function of an analog comparator that is activated (sampled) when a digital event output is generated.

[0012] EP 2 933 995 A1 addresses the poor inconsistency issue in standard analog storage implementations of dynamic vision sensors using transistors in sub-threshold operation. Since operation in this transistor mode works correctly only within a limited range, full analog feedback is used for the analog portion of the entire array of acquisition circuits to achieve satisfactory operation.

[0013] Background subtraction is a well-known and widely used technique in video image processing, for example, in “Background Subtraction: Theory and Practice” (ISBN 1627054405) by Ahmed Elgammal. This is generally performed outside of an image sensor with a standard (non-event-based) image sensor based on calculated image differences.

[0014] WO 2020 / 229981 A1 discloses an approach that performs static but tolerant background removal outside the pixel array, but performs event-like output (a frame representing pixels outside the normal range) on the image sensor chip.

[0015] A delta image sensor including an improved pixel circuit that digitally stores a representation of the previous light application intensity in the pixel may be required.

[0016] Such requirements may be satisfied as the subject of an independent claim. Advantageous embodiments are defined in dependent claims.

[0017] The idea of ​​the basic embodiment of the present invention can be interpreted as being based particularly on the following observations and perceptions.

[0018] CMOS (Complementary Metal Oxide Silicon) image sensors are mass-produced using specialized processes in optimized foundry processes. CMOS digital logic devices are mass-produced using optimized foundry processes. The proportion of both processes is increasing, particularly the proportion of the logic process. This increased proportion allows for an increase in digital complexity related to the pixel circuits of image sensors.

[0019] Digital storage of previous levels within the pixel circuit has advantages in storage accuracy and sequential processing of stored levels.

[0020] Local processing at the level offers efficiency advantages in terms of power and area because the propagation and processing of redundant data are limited. The problem to be solved

[0021] The present invention discloses a structure and method in which digital conversion and storage are combined in a pixel circuit, along with the local evaluation of differences according to time and location and the preparation and propagation of event information. means of solving the problem

[0022] The present invention is defined in the independent claim. The dependent claim describes the preferred embodiments thereof.

[0023] The present invention relates to a delta image sensor comprising a pixel array and a plurality of acquisition circuits corresponding to at least one pixel and formed as part of an integrated circuit. Each acquisition circuit comprises: at least one sensor circuit comprising a light sensor configured to generate a sensor signal VSIG according to a light signal applied to a photosensor of at least one pixel; and at least one analog-to-digital converter circuit configured to generate a digital representation corresponding to the current VSIG. At least one digital storage circuit is configured to store at least one previous representation of VSIG. At least one digital comparator circuit is configured to detect whether a changed level exists by comparing the level of the stored representation with the current VSIG. At least one digital output circuit is configured to generate an event output under the condition that a changed level exists. The digital representation may additionally be externally written to the digital storage circuit of at least one pixel.

[0024] Various embodiments can preferably implement the following features.

[0025] The digital storage circuit may be configured to input the digital representation from the data stream, preferably as a reference image or a time-varying reference image, to the image sensor.

[0026] The digital storage circuit may be configured to record the digital representation from a data stream, which is an event stream, preferably input to the image sensor as a reference image or a time-varying reference image.

[0027] The above reference image may include reference image data for at least one pixel, preferably at least one group of pixels, more preferably all pixels.

[0028] The above reference image is preferably updated regularly with frame data, and preferably the reference image data can be supplied in a specific order.

[0029] The at least one pixel or the at least one pixel group individually has its own address, and is configured to sequentially write the reference image data to each of the at least one pixel and the at least one pixel group, preferably to each of the at least one pixel and the at least one pixel group in a randomly addressed sequence.

[0030] Each acquisition circuit may be formed as an integrated circuit or as part of an integrated circuit. The acquisition circuit may also be referred to as a pixel circuit throughout the specification. At least one sensor circuit may generate a sensor signal continuously or periodically. An analog-to-digital converter circuit may convert the current VSIG into a digital signal. Accordingly, a digital storage circuit may be configured to store a representation of at least one digital signal corresponding to a previous VSIG. The digital representations of the current VSIG and the previous VSIG may be compared by at least one digital comparator circuit. Each of the previous VSIG or the digital representation of the VSIG may correspond to a signal generated at any previous sampling period and stored in the digital storage circuit. It may also be provided externally and stored as a reference signal. Additionally, a circuit for outputting events from a delta image sensor may include at least one event output line. At least one event output line may also be shared by a plurality of digital output circuits.

[0031] Preferably, each collection circuit is shared between at least two pixels, and preferably, the pixels are adjacent to each other.

[0032] Preferably, the sensor circuit is,

[0033] a) Linear dependence on light intensity

[0034] b) nonlinear dependence on luminous intensity, preferably logarithmic dependence, and

[0035] c) Combination of non-linear and linear dependencies on luminous intensity

[0036] Configured to generate a VSIG based on one of the relationships including,

[0037] Preferably, the above relationship can be changed in response to a control signal.

[0038] Control signals can be provided from outside the pixel array and the acquisition circuit. There may be a single control signal for the entire array or there may be more targets.

[0039] The sensor circuit is configured to change the dependency of VSIG on luminous intensity in response to a control signal, preferably occasionally, during operation, or the sensor circuit is configured to change the dependency of VSIG on luminous intensity in response to a control signal during operation, individually for at least one pixel or a subset of pixels.

[0040] The control signal may change intermittently due to changes in the total light intensity or operating mode. A separate circuit that causes the control signal change intermittently may be provided.

[0041] The image sensor can preferably further implement the following features.

[0042] a) The analog-to-digital converter circuit includes a comparator, and the comparator is,

[0043] aa) swept analog input signal, and

[0044] ab) Sequential swept analog input signals, each of which provides a different difference function

[0045] It is configured to compare VSIG with one of the swept signals including,

[0046] Preferably, the input signal is provided for a plurality of converter circuits and / or,

[0047] b) The analog-to-digital converter circuit is further configured to provide at least one digital code corresponding to a swept analog signal to at least one pixel, wherein

[0048] ba) Preferably, the digital code is a digital signal that is a gray code, and

[0049] bb) Preferably, the relationship between the swept analog signal and the digital code changes between the storage ramp and the comparison ramp.

[0050] also,

[0051] a) The swept signal is a linear ramp, or

[0052] b) The swept signal may be a non-linear ramp, and

[0053] or,

[0054] c) The swept signal is changed during operation and / or,

[0055] d) The period of the swept signal above may be changed during operation, and

[0056] And / or,

[0057] e) The repetition of the above-mentioned swept signal may be interrupted during operation for a short period or an extended period, and / or,

[0058] f) The repetition rate of the swept signal may be selected from one or more repetition rates corresponding to the light source modulation of the light signal applied to the light sensor, preferably to increase or suppress the event output.

[0059] In particular, in item f) above, the repetition rate of the swept signal may preferably be selected from one or more repetition rates corresponding to the light source modulation of the optical signal applied to the optical sensor in order to increase or suppress the response to a target (e.g., a light source of a given frequency) or to suppress the response to a disturbance of a given frequency.

[0060] Preferably, the image sensor further includes at least one reference pixel configured to set a swept ramp range, preferably, at least one reference pixel is located outside the pixel array and / or preferably, at least one reference pixel is biased under reference conditions.

[0061] Preferably, the digital comparator circuit is configured to perform a comparison with a stored level during the sweep of the analog signal, or the digital comparator circuit is configured to perform a comparison with a stored level after the sweep of the analog signal.

[0062] "During the sweep of the analog signal" and "after the sweep of the analog signal" can refer to the same process.

[0063] In particular, if a comparison is performed during an analog signal sweep, only the comparison result can be stored without saving the A / D output value. This reduces memory requirements while allowing for more comparisons. If a comparison is performed after an analog signal sweep, there is no need to store the A / D output value. This increases memory requirements but reduces the number of comparisons needed.

[0064] Preferably,

[0065] a) The digital comparator circuit is configured to perform comparisons using static logic within the pixel circuit, or

[0066] b) The digital comparator circuit is configured to perform a comparison using dynamic logic within the pixel circuit, and

[0067] and / or

[0068] c) The digital comparator circuit is configured to perform hysteresis and comparison, or

[0069] d) The digital comparator circuit is configured to perform comparisons requiring a difference that changes occasionally during operation.

[0070] Preferably, the output circuit is configured to generate an output based on the result of a comparison of a comparator circuit within an adjacent pixel, or the output circuit is configured to generate an output based on a function of an adjacent pixel within a fixed configuration or a configuration that changes during operation—preferably, said function is an average function—or the output circuit is configured to generate an output when the rate of change of the comparison exceeds a specific threshold.

[0071] Preferably,

[0072] a) The output circuit is configured to generate an output based on the output within an adjacent pixel, or

[0073] b) The output circuit is configured to generate an output based on the stored level within adjacent pixels, or

[0074] c) The output circuit is configured to generate an output according to a plurality of stored values.

[0075] Preferably,

[0076] a) The output circuit is configured to generate an event output indicating the direction of the changed level and / or,

[0077] b) The output circuit is configured to generate an event output indicating only a changed level in one direction, and / or,

[0078] c) The output circuit is configured to generate an event output indicating the magnitude of the changed level, and / or,

[0079] d) The output circuit is configured to generate an event output indicating the intensity of the application of light before and / or after the changed level.

[0080] Preferably,

[0081] a) The digital storage circuit is configured to provide a digital representation stored on the output line of the pixel array and / or,

[0082] b) The digital storage circuit is configured to provide, optionally for a pixel having an event output, a digital representation stored on the output line of a pixel array and / or,

[0083] c) The digital storage circuit is configured to provide a digital representation stored on an output line using an event column line.

[0084] An image sensor generally outputs image data. In one embodiment, the image sensor input It can accept image data. A similar concept is not proposed in the literature.

[0085] When reference image data is written to the digital storage of an image sensor, the image sensor can generate an event where the current image is different from the reference image. Due to the physical layout of the image sensor pixels, computation is efficient and existing hardware is reused. Preferably, the embodiment may include one or all of the following.

[0086] (a) Digital tolerance for small differences within pixels;

[0087] (b) digital tolerance for image change timing; or

[0088] (c) Digital tolerance or geographic alignment of the reference and actual images.

[0089] The preferred area of ​​interest may be visual inspection of the manufacturing process. The reference image is as follows.

[0090] (a) Fixed image;

[0091] (b) a fixed image sequence; or

[0092] (c) Video sequence.

[0093] Pixels can individually have their own addresses and sequentially record image information for specific pixels in a randomly addressed sequence.

[0094] Using an image data stream, a complete reference image can be recorded in which data is written to the digital storage of every pixel. Pixel information can be supplied in a specific order. The complete reference image can be periodically updated with frame data.

[0095] Reference image data can be recorded from an event stream, which is a data stream containing only pixel-level changes of the reference image. The event stream can provide a compressed version of the reference image data video frames. The event stream can follow the output data generated by the image sensor. This allows the same event stream generated from the reference video capture to be used for post-recording.

[0096] Dynamic vision sensors can be particularly suitable for high-speed changes in parts of an image. The post-recording of high-speed events can facilitate video differences for video sequences that are difficult or impossible to process for standard frame-based cameras.

[0097] Reference image data may also be predictive data based on optical flow calculations for the image. If it is determined that a section of the image moves across a pixel array at a specific velocity (optical flow), the predicted movement of the image section can be recorded in the pixel array. The pixel array then reports the error between the prediction and the actual image movement as an event. This error can be used to refine the prediction.

[0098] Output lines may be provided as a single line or multiple lines. They may also be shared between output circuits. Output circuits may also be provided.

[0099] Additionally, the digital representation stored in the acquisition circuit can be written from outside the array. Previous outputs can be written back to the stored pixel digital levels to provide a change reference for the circuit. The circuit then reports only the changes related to the reference. Typically, image data represents every pixel. However, since event streams contain only changes, they contain less data. Therefore, writing the event stream back can reduce the data ratio.

[0100] Preferably, the sensor circuit receives feedback from a digital storage circuit corresponding to a stored digital level or a stored digital level of an adjacent pixel, and the sensor circuit is configured to generate a VSIG according to the feedback. The A / D circuit can receive feedback from a stored digital level or a stored digital level of an adjacent pixel and can be configured to adjust an output signal according to the feedback.

[0101] Preferably, the sensor circuit is configured to provide VSIG, and / or the image sensor is composed of multiple semiconductor layers, each level is optimized for the function of the layer.

[0102] In particular, an analog representation or analog level of VSIG is provided so that it can be read directly from the circuit or array.

[0103] The collection circuit group can be connected to one or more common event output lines.

[0104] The acquisition circuit group may share at least one circuit element among (a) a digital output circuit, (b) a digital comparator circuit, (c) an additional digital storage circuit, and (d) an additional analog-to-digital converter circuit. The drawing is representative and other combinations of shared elements are not excluded.

[0105] The present disclosure may also include an implementation of analog-to-digital conversion using a comparator having a swept analog signal and a digital code.

[0106] The present invention provides event generation for a multi-silicon layer device, which is not disclosed by Itoh et al. mentioned above.

[0107] In an embodiment, reference image data is recorded in a digital storage circuit to influence the occurrence of sequential events.

[0108] The inter-pixel sharing circuit of an image sensor (e.g., KR 2020-0029328 A) is very important in image sensor designs that generate a dense pixel array and high pixel resolution. The present invention uses digital storage and event calculation in a time-sequential manner, which is particularly suitable for sharing resources between pixels without performance degradation.

[0109] In addition, the present invention extends the use of the technology disclosed in US 2015 / 358570 A1 to dynamic vision sensors, wherein the use of digital storage and processing with full timing is efficient and the impact on reducing the number of generated events is important for power consumption and sequential processing.

[0110] The present invention, made possible by in-pixel digital storage, can selectively generate events based on digital levels (multiple bits) stored in adjacent pixels. A specific configuration of such selective filtering can be used to achieve the same result as the single-bit filtering disclosed in WO 2020 / 080383 A1.

[0111] In addition, compared to T. Delbruck identified above, for example, the present invention has a unique timebase due to A / D conversion technology for reducing timestamp complexity and enables local, digital, and temporal filtering of events without additional transmission.

[0112] The present invention modulates analog circuit parameters based on stored digital levels or event outputs, because this feedback is efficiently implemented (surfaced) as a switch under digital control, unlike the aforementioned disclosure of EP 3 313 064 A1.

[0113] Compared to the aforementioned EP 2 933 995 A1, the present invention proposes modulation of analog circuit parameters based on locally stored pixel digital levels or event outputs, because this feedback is efficiently implemented (surface) as a switch under digital control. This differs from EP 2 933 995 A1 in function (pixel local), implementation (digital control), and purpose.

[0114] The present invention enables a region-optimized (small) pixel structure that produces high-resolution and cost-optimized sensors by utilizing the advantages of digital in-pixel conversion and storage.

[0115] Static background removal (e.g., WO 2020 / 229981 A1) is an integral part of event-based dynamic vision detection and is implemented within the image sensor. By using digital storage of an image sensor pixel array and providing a write function to said storage, the present invention uses the intrinsic difference function of the dynamic vision sensor to provide static or dynamic background (or reference) removal and actual event output. Brief explanation of the drawing

[0116] The present invention is further described with reference to the drawings. FIGS. 1 to 5 illustrate exemplary embodiments of the present invention. FIG. 1 schematically illustrates a first exemplary configuration of a delta image sensor according to the present invention. FIG. 2 schematically illustrates other exemplary configurations of a delta image sensor according to the present invention, FIG. 2(a) illustrates the sharing of an event generation circuit, FIG. 2(b) illustrates the sharing of an event generation circuit and a digital comparator circuit. FIG. 2(c) illustrates the sharing of an event generation circuit, a digital comparator circuit, and a digital storage circuit, FIG. 2(d) illustrates the sharing of an event generation circuit, a digital comparator circuit, a digital storage circuit, and an A / D converter circuit. FIG. 3 illustrates an exemplary embodiment in which the conversion from analog (VSIG) to a digital representation is performed using a comparator within the pixel circuit and the swept analog signal is applied to the pixel circuit. FIGS. 4a and 4b schematically illustrate another exemplary configuration of a delta image sensor according to the present invention, showing recorded image information for a specific pixel. FIG. 5 schematically illustrates another exemplary configuration of a delta image sensor according to the present invention, and shows prediction data based on light flow calculation for an image. Specific details for implementing the invention

[0117] Identical reference numbers indicate identical or similar components in the drawing. Therefore, redundant descriptions are omitted.

[0118] FIG. 1 schematically illustrates a first exemplary configuration according to the present invention. A delta image sensor, also called an image sensor, comprises a pixel array and at least one acquisition circuit (1). The acquisition circuit (1) may also be called a pixel circuit. The circuit is formed as part of an integrated circuit. The acquisition circuit (1) of each pixel includes at least one sensor circuit (11) comprising a light sensor that generates a sensor signal (VSIG) according to a light signal or light application intensity applied to the light sensor of at least one pixel, and at least one analog-to-digital converter circuit (12, A / D converter). The digital converter circuit (12) may be connected to the output of the sensor circuit (11). At least one digital storage circuit (13) is configured to store a representation of at least one digital signal corresponding to a previous VSIG. The digital storage circuit (13) may be connected to the output of the A / D converter (12). At least one digital comparator circuit (14) is configured to compare the level of the stored representation with the current VSIG to detect whether a changed level exists. The digital comparator circuit (14) can be connected to the output of the digital storage circuit (13) and the output of the A / D converter (12). At least one digital output circuit (15) or digital event generation circuit is configured to generate an event output under conditions where a changed level exists. The output circuit (15) can be connected to the output of the digital comparator circuit (14).

[0119] At least one event output line (16) may be included inside or outside the pixel circuit (1) to output an event output from the delta image sensor. At least one event output line (16) may also be shared by a plurality of digital output circuits (15).

[0120] This first exemplary pixel configuration enables the detection of light application intensity that has changed over an interval. A digital comparator circuit (14) detects the difference between the digital representation of a stored signal and the digital representation after the interval, and if this difference meets a given criterion, an event is generated and the fact of change is reported as event information to the output of the image sensor.

[0121] Digital storage of levels can be preferred in various applications because (i) the stored level does not degrade over time, (ii) the storage circuit (13) can be physically smaller, (iii) values ​​can be easily used from adjacent cells, and (iv) values ​​can be used more flexibly in implementing combination functions with other pixels or previous values.

[0122] In an additional exemplary configuration, as shown in FIG. 2, a portion of the pixel circuit is shared among a subset of pixels. Then, the use of the shared element is performed in a time-sequential manner.

[0123] FIG. 2(a) illustrates the sharing of an event generation circuit (15), FIG. 2(b) illustrates the sharing of an event generation circuit (15) and a digital comparator circuit (14), FIG. 2(c) illustrates the sharing of an event generation circuit (15), a digital comparator circuit (14), and a digital storage circuit (13), and FIG. 2(d) illustrates the sharing of an event generation circuit (15), a digital comparator circuit (14), a digital storage circuit (13), and an A / D converter circuit (13).

[0124] Other shared configurations are possible, and the above list does not exclude such possibilities. It is also possible to share other functions over different ranges, for example, an A / D converter circuit (12) of 4 pixels or more and an event generation circuit (15) of 16 pixels or more are possible. Other configurations may also be advantageous.

[0125] One exemplary embodiment of the present invention converts an optically applied intensity signal into an analog electrical signal having a linear relationship. This embodiment is advantageous at low optical application levels, using a commercial standard cell optimized for image capture applications that integrates charge in photodiode mode.

[0126] A further exemplary embodiment of the present invention converts an optically applied intensity signal into an analog electrical signal having a logarithmic relationship. This embodiment is advantageous for a high dynamic range of optically applied signals because the output analog signal does not saturate over a wide range of optically applied intensities.

[0127] Further exemplary embodiments of the present invention use a combination of transformations using linear and logarithmic functions or a combination of similar non-linear functions as response curves. This embodiment is advantageous for enabling both very low light application performance and a high dynamic range. The first embodiment uses a fixed configuration having combined functions.

[0128] Additional exemplary embodiments allow for a configuration in which the response curve is changed during operation for the entire pixel array. This is preferably done depending on the overall light application conditions or operating mode of the image sensor application.

[0129] Additional embodiments allow for a configuration in which the response curve changes while operating independently for a subset of pixels. This is preferably done according to the focus of interest or local light application conditions of the image sensor application.

[0130] In one exemplary embodiment, according to FIG. 3, the conversion from analog (VSIG) to a digital representation is performed using a comparator of the pixel circuit (1) and a swept analog signal applied to the pixel circuit. The digital representation is copied to a digital storage circuit (13) according to the point where the comparator output is toggled.

[0131] In the derived embodiment, the swept analog signal is a linear ramp; in the additional embodiment, the swept analog signal is a non-linear ramp, e.g., an exponential ramp.

[0132] Since the swept signal can be generated at the center, the implementation of a non-linear ramp is advantageous because the additional complexity for this circuit is required only once on the device, and it provides (i) area saving and (ii) a matching function for all pixels.

[0133] In particular, changing the lamp can be used to compensate for non-ideality of the sensor circuit (11) or to compensate for fluctuations due to temperature or other environmental conditions.

[0134] In particular, changes can be performed in an analog or digital manner.

[0135] In particular, changes can be performed by referencing test cells on the device. This implementation is advantageous because the compensation for non-idealities reflected in the test cells is automatic.

[0136] In one exemplary embodiment, the same swept analog ramp is provided simultaneously to the entire pixel array. In a derived embodiment, different swept analog ramps are applied to a subset of the array.

[0137] The use of different lamps is advantageous when the characteristics of the sensor circuit (11) vary across the pixel array (due to intentional cell variation or process or optical configuration) and this variation is compensated for by the provision of different lamps.

[0138] In an additional embodiment, the swept analog ramp changes occasionally depending on the operating mode of the image sensor during operation.

[0139] In one exemplary embodiment, the changing digital code is provided simultaneously with the analog ramp. The digital code stored in the digital storage circuit (13) is the value of the digital code when the comparator output is toggled. In the first embodiment, the code is a binary code.

[0140] In the derived embodiment, the digital code is a Gray code. The Gray code is advantageous because (i) only one edge is changed at a time and both sides of the edge are valid, so it does not require synchronization with digital storage, and (ii) the Gray code has fewer edges during a cycle and reduces power consumption.

[0141] In the derived embodiment, analog and digital code sweeps are sequentially separated into store and compare sweeps or multiple compare sweeps. The relationship between the analog signal level and the digital code is variable between the reference sweep and the compare sweep. This approach is advantageous because it requires simpler digital comparison logic in the pixel. This reduces the required area.

[0142] In the derived embodiment, different relationships between the swept analog level and the digital code are generated for different subsets of pixels.

[0143] This approach is advantageous because the digital code can be changed either (i) in calculations or (ii) in the timing of the digital circuit. This provides area savings and enhanced flexibility while generating multiple analog ramps, and allows compensating for or changing the function of a subset of the pixel array.

[0144] In one exemplary embodiment, a digital comparison between a stored level and a current level is performed continuously during an analog sweep. The comparison result is related to the toggling timing of the analog comparator output and the digital comparison. In this embodiment, the implementation of hysteresis or different event generation thresholds is performed based on relative timing.

[0145] In one exemplary embodiment, the timing variation is generated based on a clock. In a further exemplary embodiment, the variation is generated together with a timer circuit within the pixel circuit.

[0146] In an additional exemplary embodiment, a digital comparison between the stored level and the current is performed after an analog voltage sweep. This embodiment is advantageous when the logic can repeatedly use sequential elements (e.g., bits) that provide silicon area savings.

[0147] In one exemplary embodiment, sequential logic is performed as static logic.

[0148] In an additional exemplary embodiment, sequential logic is performed as dynamic logic.

[0149] In the derived embodiment, digital comparison is performed with feedback from previous results. This is generally used to implement hysteresis in comparisons and is advantageous for noise suppression and reduction of spurious events.

[0150] In the derived embodiment, digital comparison and event generation are performed by requiring different differences (>1) to generate events. This may be advantageous for (i) reducing the number of events or (ii) improving the event generation resolution.

[0151] The required difference may depend on (i) signal level, (ii) operating mode (e.g., region of interest), (iii) adjacent pixel level, (iv) the pixel's previous level, (v) events from the pixel prior, and (vi) events from adjacent pixels.

[0152] In the derived embodiment, an event generation rate below a certain threshold is suppressed within the pixel circuit. This is advantageous because this information is less useful for sequential image processing and event suppression reduces power and processing requirements.

[0153] In one exemplary embodiment, the digital condition for event generation depends on the event generation of adjacent pixel circuits. For example, this operation can reduce the number of spurious events through event filtering.

[0154] In one exemplary embodiment, the digital condition for event generation depends on the stored level of an adjacent pixel circuit. For example, this operation can provide enhanced sensitivity near the edge through event filtering.

[0155] In one exemplary embodiment, the digital condition for event generation depends on a number of stored levels of the pixel circuit. For example, this operation can reduce the number of spurious events through the filtering of analog noise.

[0156] In one exemplary embodiment, the sampling rate is selected to be synchronized with the flicker frequency of the disturbance light source. This is advantageous because it can suppress events caused by this light source.

[0157] In one exemplary embodiment, the event output is a single event bit, e.g., an up event, or no event. That is, an up event corresponds to increased light intensity.

[0158] In additional embodiments, the event output is 2 bits, e.g., up and down. That is, the event output may indicate whether the luminous intensity or sensor signal (VSIG) is greater or less than the stored luminous intensity level or the stored VSIG.

[0159] In an additional exemplary embodiment, the event output includes a digital representation of the magnitude of the change. This embodiment has the advantage that an image can be completely reconstructed from the event information.

[0160] FIG. 4a illustrates recording information about a complete image in a specific sequence, scanning the first upper row and sequential rows up to the right lower pixel starting from the upper left pixel as the first recorded pixel, and FIG. 4b illustrates random access to any pixel in any sequence.

[0161] Referring to FIG. 4a, an entire reference image can be recorded using an image data stream, in which data is written to the digital storage of all pixels. Pixel information can be provided in a specific order. The entire reference image can be periodically updated with frame data.

[0162] Next, referring to FIG. 4b, pixels can individually have their own addresses, and image information for specific pixels in a randomly addressed sequence can be written sequentially.

[0163] Reference image data can be recorded from an event stream, which is a data stream containing only pixel-level changes of the reference image. The event stream can provide a compressed version of the reference image data video frames. The event stream can follow the output data generated by the image sensor. This allows the same event stream generated from the reference video capture to be used for post-recording.

[0164] Dynamic vision sensors can be particularly suitable for high-speed changes in parts of an image. The post-recording of high-speed events can facilitate video differences for video sequences that are difficult or impossible to process for standard frame-based cameras.

[0165] Referring to FIG. 5, the reference image data may also be prediction data based on optical flow calculations for the image. If it is determined that a section of the image moves across a pixel array at a specific speed (optical flow), the predicted movement of the image section can be recorded in the pixel array. The pixel array then reports the error between the prediction and the actual image movement as an event. This error can be used to refine the prediction.

[0166] In Fig. 5, the rectangle is predicted to move across the image in the direction of the arrow, and pixel information for the pixels in front of and behind the rectangle is recorded as a new predicted value.

[0167] In a further exemplary embodiment, the event output includes a representation of light intensity before and / or after the change. In one embodiment, it is further possible to read the level of the value stored in the pixel array.

[0168] In the derived embodiment, the digital storage value from the target area is selectively read, that is, through random access.

[0169] In an additional exemplary embodiment, the digital level is read through an existing common event output line. In one embodiment, the stored digital level may be written to a pixel array to provide a new or intentionally modified reference for comparison. In a derived embodiment, a data stream is written to the stored digital level to intentionally provide a time-dependent reference for event generation.

[0170] In the derived related embodiments, the data stream is an event stream similar to one normally generated by the sensor.

[0171] In one embodiment, the sensor circuit (11) has feedback from a stored digital level or a stored digital level of an adjacent pixel and adjusts the VSIG level of the corresponding pixel. This embodiment may be advantageous for implementing a hysteresis function, for example, in a minimum surface area.

[0172] In one exemplary embodiment, the comparator circuit has feedback from a stored digital level or a stored digital level of an adjacent pixel to adjust the function of the comparator, typically to add an offset. This embodiment may be advantageous, for example, for implementing a hysteresis function in a minimum surface area.

[0173] In one exemplary embodiment, the sampling rate is electrically configurable or occasionally changed electrically during operation.

[0174] Power consumption and event generation rates may vary depending on the sampling rate. By reducing the sampling rate, a very low power mode can be used in a non-changing state. Additionally, a low sampling rate combined with the operation of the integrated sensor circuit (11) can operate in a low light state.

[0175] In one exemplary embodiment, sampling is completely interrupted during an extended period.

[0176] Power consumption and the event generation rate may vary depending on the sampling rate. Interrupting sampling for a fixed period (e.g., 1 second) enables very low power operation, but maintains the accuracy of the recorded image due to digital storage.

[0177] In one exemplary embodiment, the analog signal level (VSIG) can be read from a pixel array. This embodiment can be implemented using common event column lines.

[0178] In one exemplary embodiment, the image sensor is composed of multiple semiconductor layers, wherein the semiconductor process type of each level is optimized for a function of that layer. This implementation is advantageous because the optimization of the semiconductor process function allows for (i) performance improvement, (ii) reduction of the silicon area and consequently (ii) a) reduction in device size and (ii) b) reduction in device cost. The modified physical structure improves optical performance regarding (i) fill factor and (ii) quantum efficiency, and reduces electrical disturbances between the control signal line and the photosensor circuit.

[0179] Multiple pixels are generally composed of a two-dimensional grid having "rows" and "columns." It should be noted that the definitions of "rows" and "columns" can be interchanged without affecting the present invention. It should also be noted that other geometric configurations mapped to the two-dimensional grid are possible without affecting the applicability of the present invention.

[0180] Finally, it should be noted that the term "comprising" does not exclude other elements or steps, and that the singular expression does not exclude the plural expression. Additionally, components described in relation to other embodiments may be combined. Furthermore, it should be noted that reference symbols in the claims should not be interpreted as limiting the claims.

Claims

Claim 1 A delta image sensor comprising a pixel array and a plurality of acquisition circuits (1) corresponding to at least one pixel and formed as part of an integrated circuit, wherein each of the acquisition circuits (1) comprises: at least one sensor circuit (11) comprising the light sensor configured to generate a sensor signal VSIG according to a light signal applied to the light sensor of the at least one pixel; at least one analog-to-digital converter circuit (12) configured to generate a digital representation corresponding to the current VSIG; at least one digital storage circuit (13) configured to store a representation of at least one digital signal corresponding to the previous VSIG; at least one digital comparator circuit (14) configured to compare the level of the stored representation with the current VSIG to detect whether a changed level exists; and at least one digital output circuit (15) configured to generate an event output under the condition that the changed level exists....including, wherein the digital representation is externally recorded in the digital storage circuit (13) of the at least one pixel, and the image sensor is configured in one or more of the following ways a) and b), where, way a) the analog-to-digital converter circuit (12) includes a comparator, and the comparator is configured to compare the VSIG with one of the swept signals including aa) a swept analog input signal and ab) sequential swept analog input signals, each of which provides a different difference function, wherein the input signal is provided for a plurality of analog-to-digital converter circuits (12); way b) the analog-to-digital converter circuit (12) is further configured to provide at least one digital code corresponding to the swept analog input signal to at least one pixel, wherein ba) where the digital code is a digital signal that is a gray code, and bb) where the relationship between the swept analog input signal and the digital code is stored A delta image sensor that changes between a ramp and a comparison ramp, wherein the image sensor further comprises at least one reference pixel configured to set a swept ramp range, and the at least one reference pixel is configured in one or more of the following two ways, wherein the first way is that the at least one reference pixel is located outside the pixel array, and the second way is that the at least one reference pixel is biased under a reference condition. Claim 2 A delta image sensor according to claim 1, wherein the digital representation can be externally recorded from a data stream to the digital storage circuit (13) and input to the image sensor as a reference image or a time-varying reference image. Claim 3 A delta image sensor according to claim 1, wherein the digital representation can be externally recorded in the digital storage circuit (13) from a data stream which is an event stream, and can be input to the image sensor as a reference image or a time-varying reference image. Claim 4 In paragraph 2, the reference image comprises a delta image sensor including reference image data for at least one pixel, at least one pixel group, or all pixels. Claim 5 A delta image sensor according to claim 4, wherein the at least one pixel or the at least one pixel group individually has its own address and sequentially writes the reference image data to each of the at least one pixel and the at least one pixel group. Claim 6 In any one of claims 1 to 5, each of the collection circuits (1) is shared between at least two pixels, and the pixels are adjacent to each other, a delta image sensor. Claim 7 A delta image sensor according to any one of claims 1 to 5, wherein the sensor circuit (11) is configured to generate a VSIG based on one of a relationship comprising a) a linear dependency on light intensity, b) a non-linear dependency on said light intensity, and c) a combination of said non-linear dependency and linear dependency on said light intensity, and said relationship is changeable in response to a control signal. Claim 8 A delta image sensor according to any one of claims 1 to 5, wherein the sensor circuit (11) is configured to change the dependence of VSIG on light intensity in response to a control signal during operation. Claim 9 delete Claim 10 A delta image sensor according to claim 1, wherein a) the swept signal is a linear ramp, b) the swept signal is a non-linear ramp, or is configured in one or more of c) to f), wherein c) the method is such that the swept signal is changed during operation, d) the method is such that the period of the swept signal is changed during operation, e) the method is such that the repetition of the swept signal can be interrupted during a short period or an extended period during operation, and f) the method is such that the repetition rate of the swept signal is selected from one or more repetition rates corresponding to the light source modulation of the optical signal applied to the optical sensor to increase or suppress the event output. Claim 11 delete Claim 12 A delta image sensor according to claim 1 or 10, wherein the digital comparator circuit (14) is configured to perform a comparison with the stored level during the sweep of the analog input signal, or the digital comparator circuit (14) is configured to perform a comparison with the stored level after the sweep of the analog input signal. Claim 13 A delta image sensor according to any one of claims 1 to 5, wherein a) the digital comparator circuit (14) is configured to perform a comparison using static logic within the pixel circuit, or is configured in one or more of b) to d), wherein method b) is configured such that the digital comparator circuit (14) is configured to perform a comparison using dynamic logic within the pixel circuit, method c) is configured such that the digital comparator circuit (14) is configured to perform a comparison with hysteresis, and method d) is configured such that the digital comparator circuit (14) is configured to perform a comparison requiring a difference that changes during operation. Claim 14 A delta image sensor according to any one of claims 1 to 5, wherein the output circuit (15) is configured to generate an output according to the comparison result of the comparator circuit (14) within an adjacent pixel, or is configured in one or more of the following two ways, the first way being that the output circuit (15) is configured to generate an output according to a function of the adjacent pixel within a fixed configuration or a configuration that changes during operation, and the second way being that the output circuit (15) is configured to generate an output when the rate of change of the comparison exceeds a specific threshold value. Claim 15 A delta image sensor according to any one of claims 1 to 5, wherein a) the output circuit (15) is configured to generate an output according to the output within an adjacent pixel, b) the output circuit (15) is configured to generate an output according to the stored level within an adjacent pixel, or c) the output circuit (15) is configured to generate an output according to a plurality of stored values. Claim 16 A delta image sensor according to any one of claims 1 to 5, wherein the image sensor is configured in one or more of the following ways a) to d), wherein method a) is configured such that the output circuit (15) is configured to generate an event output indicating the direction of the changed level; method b) is configured such that the output circuit (15) is configured to generate an event output indicating only one direction of the changed level; method c) is configured such that the output circuit (15) is configured to generate an event output indicating the magnitude of the changed level; and method d) is configured such that the output circuit (15) is configured to generate an event output indicating the intensity of the application of light to at least one of the time before and after the changed level. Claim 17 A delta image sensor according to any one of claims 1 to 5, wherein the image sensor is configured in one or more of the following ways a) to c), wherein method a) is configured such that the digital storage circuit (13) is configured to provide the stored digital representation to the output line of the pixel array, method b) is configured such that the digital storage circuit (13) is configured to provide the stored digital representation to the output line of the pixel array selectively for a pixel having an event output, and method c) is configured such that the digital storage circuit (13) is configured to provide the stored digital representation to the output line using an event column line. Claim 18 A delta image sensor according to any one of claims 1 to 5, wherein the image sensor is configured in one or more of the following two ways, wherein the first way is configured such that the sensor circuit (11) receives feedback from the digital storage circuit (13) corresponding to the stored digital level or the stored digital level of an adjacent pixel, and the sensor circuit (11) is configured to generate a VSIG according to the feedback, and the second way is configured such that the analog-to-digital converter circuit (12) receives feedback from the stored digital level or the stored digital level of an adjacent pixel, and is configured to adjust an output signal according to the feedback. Claim 19 A delta image sensor according to any one of claims 1 to 5, wherein the image sensor is configured in one or more of the following two ways, wherein the first way is that the sensor circuit (11) is configured to provide an analog representation of VSIG, and the second way is that the image sensor is composed of a plurality of semiconductor layers. Claim 20 In claim 7, the above nonlinear dependency is a logarithmic dependency, a delta image sensor. Claim 21 In claim 8, the sensor circuit (11) is configured to change the dependency of VSIG on the light intensity in response to a control signal during operation, a delta image sensor. Claim 22 In paragraph 14, the above function is an average function, a delta image sensor.

Images