Delta image sensor with externally writable digital pixel storage.

Abstract

The present invention relates to a delta image sensor comprising an array of pixels and a plurality of acquisition circuits formed as part of an integrated circuit, each corresponding to at least one pixel. Each acquisition circuit comprises at least one sensor circuit including a photosensor configured to generate a sensor signal (VSIG) in response to an optical signal illuminating the photosensor of the at least one pixel, at least one analog-to-digital (A / D) circuit configured to convert the current VSIG into a digital signal, at least one digital storage circuit configured to store a representation of at least one digital signal corresponding to a previous VSIG, at least one digital comparison circuit configured to compare the level of the stored representation with the current VSIG to detect whether a changed level exists, and at least one digital output circuit configured to generate an event output under a changed level condition. The digital representation can be externally written to the digital storage circuit of the at least one pixel.

Description

[Technical field]

[0001] The present invention relates to an image sensor comprising a plurality of pixel circuits. [Background technology]

[0002] Image sensors with one ADC per pixel, or per small subset of pixels, are well known and are typically used in applications where significant processing is performed at the pixel (e.g., X-ray particle trajectory tracing or satellite imaging).

[0003] Image sensors are known that have a single-slope ADC in which each pixel includes a comparator, e.g., Reckleben et al., 2015, Suarez et al., 2010, and Chi et al., 2010. In particular, Suarez et al. generate an analog slope and a digital code globally for the pixel array and provide the digital code to the pixel.

[0004] There are no known image sensors in which digital values ​​are stored within the pixel and compared sample-to-sample to create event change data. The present invention enables an area-optimized small pixel structure that produces a high resolution and cost-optimized sensor.

[0005] An image sensor in which a representation of the previous illumination level is digitally stored in the pixels and compared between samples to generate event change data is known from WO2014 / 174498A1, while Itoh et al., "4-Layer 3-D IC with a function of parallel signal processing", Microelectronic engineering, Vol. 15, Nos. 1-4, 1991, pp. 187-190, have previously proposed a conceptually related structure using a simple single-bit pixel comparison against a fixed level.

[0006] Image sensors using event generation using detection, storage and comparison of image pixel level changes in the analog domain are known, for example, Chi et al., "CMOS Camera With In-Pixel Temporal Change Detection and ADC," IEEE Journal of Solid-State Circuits, Vol. 42, No. 10, 2007, pp. 2187-2196. Comparison and storage in the analog domain offers advantages (e.g., compact implementation) and disadvantages (e.g., gradual storage degradation including "hot pixels," sensitivity to mismatches, reduced flexibility of the optical-to-electrical conversion function) over digital approaches.

[0007] Image sensors that share circuitry between pixels are well known, for example KR2020-0029328A.

[0008] The use of stroboscopic effects is commonly known along with suppression of harmonics by averaging or synchronizing over the period of a disturbance (undesired signal), for example in U.S. Patent Application Publication No. 2015 / 358570, where techniques are applied to a camera to detect and synchronize to a variable light source and then suppress the response to the light source.

[0009] WO2020 / 080383A1 discloses an event-based dynamic vision sensor that uses a filter to ignore events from one pixel based on the event status (single bit) of adjacent pixels.

[0010] T. Delbruck, "Frame-free dynamic digital vision," Proceedings of the International Conference on Secure Life Electronics, Advanced Electronics for Quality Life and Society, 2008, pp. 21-26, discloses filtering events from a dynamic vision sensor based on the timestamp of the last event of a pixel or the timestamp of an adjacent pixel. This processing is done outside the pixel array in a computationally intensive manner (32-bit timestamps) and requires sending events out of the array for subsequent filtering.

[0011] EP3 313 064 A1 discloses an analogue stored dynamic vision sensor, where one comparator input is a representation of the previous value of pixel illumination, which is used for subsequent comparison. This can be seen as an analogue feedback from the previous level to control the subsequent function of an analogue comparator that is enabled (sampled) when a digital event output is generated.

[0012] EP2 933 995 A1 addresses the problem of poor consistency in the standard analog storage implementation of dynamic vision sensors using transistors in sub-threshold operation. Since operation in this transistor mode only works correctly over a limited range, a global analog feedback is used to the analog part of the acquisition circuitry of the entire array to obtain satisfactory operation.

[0013] Background subtraction is a well-known and widely used technique in video image processing, see for example Background Subtraction: Theory and Practice, Ahmed Elgammal, ISBN1627054405. It is usually performed outside the image sensor using standard (non-event-based) image sensors based on a computed image difference.

[0014] WO2020 / 229981A1 discloses a technique that uses event-like output (frames showing pixels outside of normal ranges) to perform static but robust background subtraction outside the pixel array but on the image sensor chip.

[0015] There may be a need for a delta circuit sensor with an improved pixel circuit that has digital storage of a representation of the previous illumination intensity within the pixel.

[0016] Such a need is met by the subject matter of the independent claims. Advantageous embodiments are defined in the dependent claims.

[0017] The idea underlying embodiments of the present invention can be interpreted as being based, inter alia, on the following observations and realizations.

[0018] CMOS (Complementary Metal Oxide Silicon) image sensors are manufactured in specialized, high volume optimized foundry processes. CMOS digital logic devices are manufactured in high volume optimized foundry processes. The density of both processes is increasing, especially the density of the logic process. The increased density leads to an increase in the digital complexity associated with the pixel circuitry in the image sensor.

[0019] Digital storage of the previous levels in the pixel circuit has advantages in the fidelity of the storage and in subsequent processing of the stored levels.

[0020] Level-local processing has the advantage of power and area efficiency due to limited propagation and processing of redundant data. Summary of the Invention [Means for solving the problem]

[0021] The present invention discloses structures and methods in which digital conversion and storage are combined in the pixel circuit with local evaluation of differences over time and location, and preparation and propagation of event information.

[0022] The invention is defined in the independent claims. The dependent claims describe preferred embodiments thereof.

[0023] The present invention relates to a delta image sensor comprising an array of pixels and a plurality of acquisition circuits corresponding to at least one pixel and formed as part of an integrated circuit. Each acquisition circuit comprises at least one sensor circuit comprising a photosensor configured to generate a sensor signal (VSIG) in response to an optical signal illuminating a photosensor of the at least one pixel, and at least one analog-to-digital conversion circuit configured to generate a digital representation corresponding to a current VSIG. The at least one digital storage circuit is configured to store a representation of at least one previous VSIG. The at least one digital comparison circuit is configured to compare a level of the stored representation with the current VSIG to detect whether a changed level is present. The at least one digital output circuit is configured to generate an event output under a condition of the changed level. The digital representation can further be written externally to the digital storage circuit of the at least one pixel.

[0024] Various embodiments may preferably implement the following features. The digital storage circuitry may be configured to write a digital representation from the data stream, preferably as a reference image or a reference time-varying image, to be input to the image sensor.

[0025] The digital storage circuitry may be configured to write a digital representation from a data stream that is an event stream, preferably input to the image sensor as a reference image or a reference time-varying image.

[0026] The reference image may include reference image data for at least one pixel, preferably at least one group of pixels, and more preferably, all pixels.

[0027] The reference image may preferably be updated periodically with frame data, and preferably the reference image data may be provided in a particular order.

[0028] At least one pixel or at least one group of pixels may be individually addressable to sequentially write reference image data to the at least one pixel and at least one group of pixels, respectively, preferably in a randomly addressed order.

[0029] Each acquisition circuit may be formed as an integrated circuit or part of an integrated circuit. The acquisition circuit may also be referred to as a pixel circuit throughout this specification. The at least one sensor circuit may generate a sensor signal continuously or periodically. The analog-to-digital conversion circuit may convert the current VSIG to a digital signal. The digital storage circuit may thus be configured to store a representation of at least one digital signal corresponding to a previous VSIG. The digital representations of the current and previous VSIG may be compared by at least one digital comparison circuit. The previous VSIG, or the digital representation of the VSIG, respectively, may correspond to a signal generated in any previous sample period and stored in the digital storage circuit. This may also be provided externally and stored as a reference signal. The circuit may also include at least one event output line for outputting an event output from the delta image sensor. The at least one event output line may also be shared by multiple digital output circuits.

[0030] Preferably, each acquisition circuit is shared between at least two pixels, preferably the pixels being adjacent to each other.

[0031] Preferably, the sensor circuit is configured to generate the VSIG based on one of the following relationships: a) Linear dependence on light intensity, b) a non-linear, preferably logarithmic, dependence on light intensity; and c) a combination of nonlinear and linear dependence on light intensity; Preferably, the relationship is alterable in response to a control signal.

[0032] The control signals may be provided from outside the array of pixels and acquisition circuitry. There may be one control signal for the entire array, or it may be more targeted.

[0033] Preferably, the sensor circuitry is configured to modify the dependence of the VSIG on light intensity in response to a control signal during operation, preferably from time to time, and / or the sensor circuitry is configured to modify the dependence of the VSIG on light intensity in response to a control signal during operation individually for at least one pixel or a subset of pixels.

[0034] The occasional control signal changes are due to changes in global light intensity or operational mode. Separate circuitry may be provided for generating the occasional control signal changes.

[0035] The image sensor is preferably further capable of implementing the following features: a) The analog-to-digital conversion circuit converts VSIG into the following sweep signal: aa) a swept analog input signal; and ab) continuous sweep analog input signals, where each of the swept analog signals provides a different difference function; a comparator configured to compare the first and second inputs to one of the first and second inputs; Preferably, the input signal is provided to a plurality of conversion circuits, and / or b) the analog to digital conversion circuit may be further configured to provide at least one digital code in parallel to the swept analog signal to the at least one pixel; ba) preferably, the digital code is a Gray coded digital signal; bb) Preferably, the relationship between the analog sweep signal and the digital code is altered between a storage ramp and a comparison ramp.

[0036] moreover, a) the sweep signal can be a linear ramp, or b) the sweep signal may be a non-linear ramp, or c) the sweep signal can be changed during operation, and / or d) the duration of the sweep signal can be changed during operation, and / or e) the repetition of the sweep signal may be interrupted for short or long periods during operation; and / or f) The repeat rate of the sweep signal may be selected from one or more repeat rates that preferably match the light source modulation of the optical signal illuminating the photosensor to enhance or suppress the event output.

[0037] In particular, from item f) above, the repeat rate of the sweep signal can be selected from one or more repeat rates that preferably match the light source modulation of the optical signal illuminating the photosensor in order to increase or suppress the response to a target, e.g. a light source having a given frequency, or to suppress the response to an external disturbance having a given frequency.

[0038] Preferably, the image sensor further comprises at least one reference pixel configured to set the swept ramp limit, preferably the at least one reference pixel is positioned outside the array of pixels and / or preferably the at least one reference pixel is biased under reference conditions.

[0039] Preferably, the digital comparison circuit is configured to perform the comparison with the stored levels during the sweep of the analog signal, or the digital comparison circuit is configured to perform the comparison with the stored levels after the sweep of the analog signal.

[0040] "During the sweep of the analog signal" and "while the sweeping analog signal" can refer to the same process.

[0041] In particular, if the comparisons are made during the sweep of the analog signal, the A / D output values ​​do not need to be stored, just the comparison results. This results in fewer memory requirements, but more comparisons. If the comparisons are made after the sweep of the analog signal, the A / D output values ​​do not need to be stored. This increases memory requirements, but reduces the number of comparisons required.

[0042] Preferably, a) the digital comparison circuit is configured to perform the comparison using static logic in the pixel circuit, or b) the digital comparison circuit is configured to perform the comparison using dynamic logic in the pixel circuit; and / or c) the digital comparison circuit is configured to perform a comparison with hysteresis; and / or d) The digital comparison circuit is configured to perform a comparison requiring a difference that is changed from time to time during operation.

[0043] Preferably, the output circuit is configured to generate an output depending on the comparison result of the comparison circuit at adjacent pixels, or the output circuit is configured to generate an output depending on a function of the comparison results of the comparison circuit at adjacent pixels in a fixed configuration or in a configuration that is changed during operation, preferably the function is an average function and / or the output circuit is configured to generate an output when the rate of change of the comparison exceeds a certain threshold.

[0044] Preferably, a) the output circuit is configured to generate an output in response to outputs at adjacent pixels; or b) the output circuit is configured to generate an output in response to the stored levels in adjacent pixels; or c) an output circuit configured to generate an output in response to the plurality of stored values.

[0045] Preferably, a) the output circuit is configured to generate an event output indicative of a direction of the changed level; and / or b) the output circuitry is configured to generate an event output indicative of only the level that has changed in one direction; and / or c) the output circuitry is configured to generate an event output indicative of the magnitude of the changed level; and / or d) The output circuitry is configured to generate an event output indicative of the illumination intensity before and / or after the changed level.

[0046] Preferably, a) the digital storage circuit is configured to provide a stored digital representation at an output line of the array of pixels; and / or b) the digital storage circuit is configured to provide the stored digital representation selectively for pixels having an event output at an output line of the array of pixels; and / or c) the digital storage circuit is configured to provide at an output line a digital representation of the stored event using the event column line;

[0047] An image sensor typically outputs image data. In an embodiment, the image sensor may accept image data as input. No similar concept has been presented in the literature.

[0048] When reference image data is written to the digital storage of the image sensor, the image sensor can generate an event where the current image differs from the reference image. Due to the physical layout of the image sensor pixels, the computation is efficient and reuses existing hardware. Preferably, the embodiment: (a) Digital tolerance to small differences within a pixel, (b) digital immunity to timing of image changes; or (c) digital robustness to geo-alignment of reference and actual images; may include one or all of the following:

[0049] A preferred area of ​​interest may be one for visual inspection during the manufacturing process. The reference image is (a) Fixed image, (b) a sequence of fixed images; or (c) a video sequence; It can be said that:

[0050] The pixels can be individually addressed to sequentially write image information for particular pixels in a random addressed order.

[0051] A complete reference image can be written, where data is written to the digital store of every pixel using an image data stream. The pixel information can be provided in a specific order. The complete reference image can be updated periodically using frame data.

[0052] The reference image data can be written from an event stream, which is a data stream that contains only pixel level changes of the reference image. The event stream can provide a compressed version of the reference image data video frames. The event stream can be matched to the output data generated by the image sensor. This allows the same event stream to be generated from the reference video capture that is used for write-back.

[0053] Dynamic vision sensors may be particularly well suited for fast changes in parts of an image. Writing back fast events can facilitate video differentiation of video sequences that are difficult or impossible to handle with standard frame-based cameras.

[0054] The reference image may also be a prediction based on optical flow calculations of the image. If a section of an image is determined to be moving across the pixel array at a certain rate (optical flow), the predicted movement of the section of the image can be written to the pixel array. The pixel array then reports errors between the prediction and the actual image motion as events. These errors can be used to refine the prediction.

[0055] The output lines may be provided singularly or multiplely. The output lines may be shared between output circuits. Output circuits may also be provided.

[0056] Additionally, the digital representation stored in the acquisition circuitry can also be written from outside the array. Previous outputs can be written back to the stored pixel digital levels to provide a changing reference for the circuitry. The circuitry then reports only the changes relative to the reference. Typically, image data represents every pixel. However, the event stream contains only the changes and therefore less data. Thus, a reduction in data rate is possible by writing back the event stream.

[0057] Preferably, the sensor circuit is configured to receive feedback from the digital storage circuit corresponding to the stored digital levels, or the stored digital levels of adjacent pixels, and the sensor circuit is configured to generate the VSIG in response to the feedback. The A / D circuit may receive feedback from the stored digital levels, or the stored digital levels of adjacent pixels, and may be configured to adjust the output signal in response to the feedback.

[0058] Preferably, the sensor circuitry is configured to provide the VSIG and / or the image sensor is constructed from multiple semiconductor layers, with each level optimized for the function of that layer.

[0059] In particular, it provides an analog representation or level of VSIG that can be read out directly from the circuit or array.

[0060] Groups of acquisition circuits can be connected to one or more common event output lines.

[0061] The groups of acquisition circuits may share at least one of the following circuit elements: (a) digital output circuitry, (b) digital comparison circuitry, (c) additional digital storage circuitry, (d) additional analog-to-digital conversion circuitry. The figures are representative and other combinations of shared elements are not excluded.

[0062] The present disclosure may also encompass implementation of analog-to-digital conversion using a swept analog signal and a comparator using a digital code.

[0063] The present invention provides event generation in multi-silicon layer devices not disclosed by Itoh et al. as identified above.

[0064] In an embodiment, the reference image data is written to a digital storage circuit to affect subsequent event generation.

[0065] Shared circuitry between pixels in an image sensor (e.g., KR2020-0029328A) is critical in image sensor design to create dense pixel arrays and high pixel resolution. The present invention uses digital storage and event calculation in a time-series manner that is particularly suited for resource sharing between pixels without performance degradation.

[0066] Additionally, the present invention extends the use of techniques disclosed by US Patent Application Publication No. 2015 / 358570 to dynamic vision sensors, where the use of digital storage and processing with global timing is effective and the impact in reducing the number of events generated is significant for power consumption and subsequent processing.

[0067] Enabled by in-pixel digital storage, the present invention can selectively generate events based on digital levels (multiple bits) stored in adjacent pixels. A particular configuration of this selective filtering can be used to achieve the same results as the single-bit filtering disclosed by WO2020 / 080383A1.

[0068] In addition, in comparison with e.g. T. Delbruck identified above, the present invention has an inherent time base due to A / D conversion techniques for reduced time stamp complexity, allowing local digital temporal filtering of events without further transmission.

[0069] The present invention modulates an analog circuit parameter based on a stored digital level or event output, since this feedback is effectively realized using a switch under digital control (surface area) which differs from the disclosure of EP 3 313 064 A1 discussed above.

[0070] In comparison with the above-mentioned EP 2 933 995 A1, the present invention proposes the modulation of analog circuit parameters based on locally stored pixel digital levels or event outputs, since this feedback is effectively realized with switches under digital control (surface area), which differs from EP 2 933 995 A1 in function (local to the pixel), implementation (digital control) and objectives.

[0071] The present invention enables area-optimized (small) pixel structures that take advantage of digital in-pixel conversion and storage to produce high resolution, cost-optimized sensors.

[0072] Static background subtraction (e.g., WO2020 / 229981A1) is an inherent part of event-based dynamic vision sensing and is implemented within the image sensor. By using digital storage in the image sensor pixel array and providing write capability to that storage, the present invention uses the inherent difference function of the dynamic vision sensor to provide a static or dynamic background (or reference) difference and true event output.

[0073] The invention will now be further described with reference to the drawings, in which: Figures 1 to 5 show exemplary embodiments of the invention. [Brief description of the drawings]

[0074] [Figure 1] 1 illustrates diagrammatically a first exemplary configuration of a delta image sensor according to the invention; [Diagram 2] 2A and 2B show schematic diagrams of further exemplary configurations of a delta image sensor according to the present invention, where FIG. 2A shows sharing of an event generation circuit, FIG. 2B shows sharing of an event generation circuit and a digital comparison circuit, FIG. 2C shows sharing of an event generation circuit, a digital comparison circuit and a digital storage circuit, and FIG. 2D shows sharing of an event generation circuit, a digital comparison circuit, a digital storage circuit and an A / D conversion circuit. [Diagram 3] An exemplary embodiment is shown in which the conversion from analog (VSIG) to digital representation is performed using a comparator in the pixel circuit, and an analog sweep signal is applied to the pixel circuit. [Figure 4a] 3 shows a schematic diagram of a further exemplary configuration of a delta image sensor according to the present invention, in which writing image information for a particular pixel is shown; [Figure 4b] 3 shows a schematic diagram of a further exemplary configuration of a delta image sensor according to the present invention, in which writing image information for a particular pixel is shown; [Diagram 5] 4 shows a schematic representation of a further exemplary structure of a delta image sensor according to the present invention, where predicted data based on optical flow calculations for an image is shown. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0075] In the drawings, the same reference numerals denote the same or similar components, and therefore redundant descriptions will be omitted.

[0076] FIG. 1 shows a schematic representation of a first exemplary configuration according to the invention. A delta image sensor, also called image sensor, comprises an array of pixels and at least one acquisition circuit (1). The acquisition circuit (1) is also called pixel circuit. The circuit is formed as part of an integrated circuit. The acquisition circuit (1) of each pixel comprises at least one sensor circuit (11) including a photosensor generating a sensor signal (VSIG) depending on an optical signal or illumination intensity illuminating the photosensor of the at least one pixel, and at least one analog / digital conversion circuit (12, A / D converter). The digital conversion circuit (12) can be connected to the output of the sensor circuit (11). At least one digital storage circuit (13) is configured to store a representation of at least one digital signal corresponding to a previous VSIG. The digital storage circuit (13) can be connected to the output of the A / D converter (12). At least one digital comparison circuit (14) is configured to compare the level of the stored representation with the current VSIG to detect whether a changed level is present. The digital comparison circuit (14) can be connected to the output of the digital storage circuit (13) and to the output of the A / D converter (12). At least one digital output circuit (15) or digital event generation circuit is configured to generate an event output under the changed level condition. An output circuit (15) can be connected to the output of the digital comparison circuit (14).

[0077] At least one event output line (16) can be included within the pixel circuit (1) or external to the pixel circuit (1) for outputting an event output from the delta image sensor. The at least one event output line (16) can also be shared by multiple digital output circuits (15).

[0078] This first exemplary pixel configuration allows detection of altered illumination intensity over an interval: a digital comparator circuit (14) detects the difference between the stored digital representation of the signal and the digital representation after an interval, and if this difference meets a given criterion, an event is generated and the fact of the change is reported as event information at the output of the image sensor.

[0079] Digital storage of the levels may be preferred in a wide range of applications because (i) the stored levels do not degrade over time, (ii) the storage circuitry (13) can be physically smaller, (iii) the values ​​can be more easily used in adjacent cells, and (iv) the values ​​can be used more flexibly to realize combination functions with other pixels or previous values.

[0080] In a further exemplary configuration, portions of the pixel circuitry are shared among a subset of the pixels, with use of the shared components being time-sequenced, as shown in FIG.

[0081] FIG. 2a shows the sharing of an event generation circuit (15), FIG. 2b shows the sharing of an event generation circuit (15) and a digital comparison circuit (14), FIG. 2c shows the sharing of an event generation circuit (15), a digital comparison circuit (14) and a digital storage circuit (13), and FIG. 2d shows the sharing of an event generation circuit (15), a digital comparison circuit (14), a digital storage circuit (13) and an A / D conversion circuit (13).

[0082] It should be noted that other sharing configurations are possible and the above list does not exclude these possibilities. It is also possible to share different functions to different degrees, for example the A / D conversion circuit (12) across 4 pixels and the event generation circuit (15) across 16 pixels. Other configurations may also be advantageous.

[0083] One exemplary embodiment of the present invention converts the light illumination intensity signal into an analog electrical signal using a linear relationship. This embodiment integrates charge in photodiode mode and is advantageous at low light illumination levels using commercially available standard cells optimized for image capture applications.

[0084] A further exemplary embodiment of the present invention converts the optical illumination intensity signal to an analog electrical signal using a logarithmic relationship, which is advantageous for high dynamic range optical illumination signals because the output analog signal does not saturate over a wide range of illumination intensities.

[0085] A further exemplary embodiment of the present invention uses a combination of transformations using a combination of linear and logarithmic functions, or similar non-linear functions, as the response curve. This embodiment is advantageous for enabling both good low light illumination performance and high dynamic range. The first embodiment uses a fixed configuration with combined functions.

[0086] Further exemplary embodiments allow for configurations in which the response curve is modified during operation for a complete array of pixels, preferably depending on the global lighting conditions or operational mode of the image sensor application.

[0087] Further embodiments allow for configurations where the response curves are modified independently during operation for a subset of pixels, preferably in response to local lighting conditions or attention focus of the image sensor application.

[0088] In one exemplary embodiment, referring to Figure 3, conversion of the analog (VSIG) to digital representation is performed using a comparator in the pixel circuit (1) and an analog sweep signal applied to the pixel circuit. Depending on the point at which the comparator output switches, the digital representation is copied to a digital storage circuit (13).

[0089] In the embodiment derived, the analog sweep signal is a linear ramp, while in further embodiments the analog sweep signal is a non-linear ramp, for example an exponential ramp.

[0090] Because the sweep signal can be generated in a centralized manner, a nonlinear ramp implementation is advantageous because the additional complexity of this circuitry is only needed once in the device, resulting in (i) area savings and (ii) consistent functionality for all pixels.

[0091] In particular, lamp modifications can be used to compensate for non-idealities in the sensor circuit (11) or variations over temperature or other environmental conditions.

[0092] In particular, the modification can be performed by analog or digital means. In particular, the modifications can be made with reference to test cells in the device. This implementation is advantageous because the compensation for non-idealities reflected in the test cells is automatic.

[0093] In one exemplary embodiment, the same analog sweep ramp is applied to the entire pixel array simultaneously. In a derived embodiment, different analog sweep ramps are applied to subsets of the array.

[0094] The use of different lamps is advantageous when the characteristics of the sensor circuit (11) vary across the pixel array (either due to deliberate cell variations or due to process or optical configuration) and this variation can be compensated for by providing different lamps.

[0095] In a further embodiment, the analog sweep ramp is changed from time to time during operation according to the operational mode of the image sensor.

[0096] In one exemplary embodiment, a varying digital code is provided simultaneously with the analog ramp. The digital code stored in the digital storage circuit (13) is the value of the digital code when the comparator output switches. In a first embodiment, the code is a binary code.

[0097] In a derived embodiment, the digital code is a Gray code, which is advantageous because (i) only one edge changes at a time and the code on both sides of the edge is valid, so no synchronization with the digital storage is required, and (ii) Gray code has fewer edges over a cycle, reducing power consumption.

[0098] In the embodiment derived, the analog and digital code sweeps are sequentially separated into a store and compare sweep, or into multiple compare sweeps. The relationship between the analog signal level and the digital code varies between the reference and compare sweeps. This approach is advantageous because a simpler digital comparison logic is required in the pixel, which can reduce the area required.

[0099] In a derived embodiment, for different subsets of pixels, different relationships between analog sweep levels and digital codes are generated.

[0100] This approach is advantageous because the digital code can be varied either (i) in the computation or (ii) in the timing with the digital circuitry, resulting in area savings over the generation of multiple analog ramps and improved flexibility, allowing compensation or modification of the function of a subset of the array of pixels.

[0101] In one exemplary embodiment, a digital comparison between the stored level and the current level is performed continuously during the analog sweep. The comparison result is related to the timing of the switch between the analog comparison output and the digital comparison. In this embodiment, hysteresis, or the implementation of different event generation thresholds, is performed based on the relative timing.

[0102] In one exemplary embodiment, the variation in timing is generated based on a clock, hi a further exemplary embodiment, the variation is generated by a timer circuit within the pixel circuit.

[0103] In a further exemplary embodiment, a digital comparison between the stored level and the current level is performed following an analog voltage sweep. This embodiment is advantageous where logic can use successive, e.g. bit-wise, elements repeatedly, resulting in silicon area savings.

[0104] In one exemplary embodiment, the continuous logic is implemented using static logic. In a further exemplary embodiment, the continuous logic is implemented using dynamic logic. In the embodiment to be derived, the digital comparison is performed using feedback from previous results, which is typically used to implement hysteresis in the comparison, which is advantageous in suppressing noise and reducing spurious events.

[0105] In a derived embodiment, digital comparison and event generation is performed requiring a differential (>1) to generate an event, which can be advantageous to (i) reduce the number of events or (ii) improve the resolution of the generation of events.

[0106] The required difference may depend on (i) the signal level, (ii) the operating mode, e.g., region of interest, (iii) the adjacent pixel level, (iv) the previous level of the pixel, (v) the previous event of the pixel, and (vi) the event of the adjacent pixel.

[0107] In a derived embodiment, event generation rates below a certain rate are suppressed in the pixel circuitry, which is advantageous because this information is less useful for subsequent image processing and event suppression reduces power and processing requirements.

[0108] In one exemplary embodiment, the digital condition for event generation depends on the event generation of adjacent pixel circuits. This operation allows, for example, filtering of events to reduce the number of spurious events.

[0109] In one exemplary embodiment, the digital condition for event generation depends on the stored levels of adjacent pixel circuits. This operation allows, for example, filtering of events to provide increased sensitivity near edges.

[0110] In one exemplary embodiment, the digital condition for event generation depends on multiple stored levels of the pixel circuit. This operation allows, for example, filtering of analog noise to reduce the number of spurious events.

[0111] In one exemplary embodiment, the sample rate is selected to be synchronized with the flicker frequency of the disturbing light source, which is advantageous since it allows suppression of events due to this light source.

[0112] In one exemplary embodiment, the event output is a single event bit, e.g., up-event, no event, in other words, an up-event corresponds to an increase in light intensity.

[0113] In a further embodiment, the event output is two bits, e.g., up and down, In other words, the event output can indicate whether the light intensity or sensor signal VSIG is greater or less than the stored light intensity level or stored VSIG.

[0114] In a further exemplary embodiment, the event output includes a digital representation of the magnitude of the change. This embodiment is advantageous because it allows complete reconstruction of the image from the event information.

[0115] FIG. 4a shows writing information for a complete image in a particular order starting with the top left pixel as the first pixel to be written and scanning down to the bottom right pixel, down to the first top row and subsequent rows, while FIG. 4b shows random access to any pixel in any order.

[0116] Referring to Figure 4a, a complete reference image can be written where data is written to the digital storage of every pixel using an image data stream. The pixel information can be provided in a specific order. The complete reference image can be updated periodically using frame data.

[0117] Referring now to FIG. 4b, the pixels can be individually addressed to sequentially write image information for particular pixels in a random addressed order.

[0118] The reference image data can be written from an event stream, which is a data stream that contains only pixel level changes of the reference image. The event stream can provide a compressed version of the reference image data video frames. The event stream can be matched to the output data generated by the image sensor. This allows the same event stream to be generated from the reference video capture that is used for write-back.

[0119] Dynamic vision sensors may be particularly well suited for fast changes in parts of an image. Writing back fast events can facilitate video differentiation of video sequences that are difficult or impossible to handle with standard frame-based cameras.

[0120] Referring to Figure 5, the reference image can also be prediction data based on an optical flow calculation of the image. If it is determined that a section of the image is moving across the pixel array at a certain rate (optical flow), the predicted movement of the section of the image can be written to the pixel array. The pixel array then reports errors between the prediction and the actual image movement as events. These errors can be used to refine the prediction.

[0121] In FIG. 5, a rectangle is predicted to move across the image in the direction of the arrow, and pixel information for pixels in front of the rectangle and behind the rectangle is filled in with new predicted values.

[0122] In a further exemplary embodiment, the event output comprises a representation of the illumination intensity before and / or after the change.In one embodiment, it is further possible to read out the level of the value stored in the pixel array.

[0123] In a derived embodiment, the digitally stored values ​​from the target area are read selectively, ie, by random access.

[0124] In a further embodiment, the digital levels are read out via an existing common event output line. In one embodiment, the stored digital levels can be written to the array of pixels to provide a new or purposefully modified reference for comparison. In a derived embodiment, a stream of data is written to the stored digital levels to purposefully provide a time-dependent reference for event generation.

[0125] In a related derived embodiment, the stream of data is an event stream, similar to those typically generated by a sensor.

[0126] In one embodiment, the sensor circuit (11) has feedback from a stored digital level, or from the stored digital levels of adjacent pixels, to adjust the VSIG level of that pixel. This embodiment can be advantageous for implementing, for example, hysteresis functions in a minimum surface area.

[0127] In one exemplary embodiment, the comparator circuit has feedback from the stored digital level or from the stored digital levels of adjacent pixels to adjust the function of the comparator, usually adding an offset. This embodiment can be advantageous for implementing, for example, a hysteresis function, in a minimum surface area.

[0128] In one exemplary embodiment, the sample rate is electrically configurable or electrically changed from time to time during operation.

[0129] Power consumption and event generation rate can depend on the sample rate. Reducing the sample rate allows for very low power modes under quiet conditions. Furthermore, the low sample rate, combined with the integration of the sensor circuitry (11) behavior, allows operation in low light conditions.

[0130] In one exemplary embodiment, sampling is discontinued completely for an extended period of time.

[0131] Power consumption and event generation rate can depend on the sample rate: Suspending sampling for a period of time, e.g., 1 second, allows for very low power operation, but preserves the fidelity of the recorded image due to digital storage.

[0132] In one exemplary embodiment, the analog signal level (VSIG) can be read out from the pixel array. This embodiment can be realized using a common event column line.

[0133] In one exemplary embodiment, the image sensor is constructed from multiple semiconductor layers, where the semiconductor process type of each level is optimized for the functionality of that layer. This implementation is advantageous because optimizing the functionality of the semiconductor process allows for (i) improved performance, (ii) reduced silicon area, and subsequent reduction in (ii)a) device size and (ii)b) device cost. The modified physical structure improves optical performance in terms of (i) fill factor, (ii) quantum efficiency, and reduces electrical disturbance between the control signal lines and the photosensor circuitry.

[0134] Pixels are typically organized in a two-dimensional grid having "rows" and "columns." It should be noted that the definitions of "row" and "column" are interchangeable without affecting the invention. It should also be noted that other geometric configurations that map onto the two-dimensional grid are possible without affecting the applicability of the invention.

[0135] Finally, it should be noted that the term "comprising" does not exclude other elements or steps, and "a" or "an" do not exclude a plurality. Also, elements described in association with different embodiments may be combined. It should also be noted that reference signs in the claims shall not be construed as limiting the scope of the claims.

Claims

1. A delta image sensor comprising an array of pixels and a number of acquisition circuits (1) corresponding to at least one pixel and formed as part of an integrated circuit, each acquisition circuit (1) comprising: at least one sensor circuit (11) comprising a photosensor of the at least one pixel configured to generate a sensor signal (VSIG) in response to a light signal illuminating the photosensor; at least one analog-to-digital conversion (A / D) circuit (12) configured to generate a digital representation corresponding to the current VSIG; at least one digital storage circuit (13) configured to store a representation of at least one digital signal corresponding to a previous VSIG; at least one digital comparator circuit (14) configured to compare the level of said stored representation with said current VSIG to detect whether a changed level is present; at least one digital output circuit (15) configured to generate an event output under said altered level conditions; Equipped with The digital storage circuit (13) of the at least one pixel is configured to store an externally provided digital representation, said digital representation can be written externally to said digital storage circuit (13) from a data stream which is an event stream input to said image sensor as a reference image; The reference image is predicted data based on an optical flow calculation of the image.

2. said reference image comprising reference image data for at least one pixel, preferably at least one group of pixels, more preferably all pixels; and / or 2. The image sensor of claim 1, wherein the reference image is preferably updated periodically with frame data, and wherein the reference image data is preferably provided in a specific order.

3. 3. The image sensor of claim 2, wherein the at least one pixel or the at least one group of pixels are individually addressable to sequentially write reference image data to the at least one pixel and the at least one group of pixels, respectively, preferably in a randomly addressed order.

4. The image sensor according to any one of claims 1 to 3, wherein each acquisition circuit (1) is shared between at least two pixels, preferably said pixels being adjacent to each other.

5. The sensor circuit (11) satisfies the following relationship: a) Linear dependence on light intensity; b) a non-linear, preferably logarithmic, dependence on said light intensity; and c) a combination of non-linear and linear dependencies on said light intensity; and configured to generate VSIG based on one of: Preferably, the relationship is alterable in response to a control signal.

6. The sensor circuit (11) is configured to modify the dependence of VSIG on light intensity in response to a control signal during operation, preferably from time to time; and / or 6. The image sensor of claim 5, wherein the sensor circuit (11) is configured to modify, during operation, the dependence of the VSIG on the light intensity in response to a control signal for at least one pixel or a subset of pixels individually.

7. a) The analog to digital conversion circuit (12) converts the VSIG into the following sweep signal: aa) a swept analog input signal; and ab) continuous sweep analog input signals, where each of said swept analog input signals provides a different difference function; a comparator configured to compare the first and second inputs to one of the first and second inputs; Preferably, the swept analog input signal is provided to a plurality of conversion circuits (12), and / or b) the analog to digital conversion circuit (12) is further configured to provide at least one digital code in parallel to the swept analog input signal to at least one pixel; a) preferably, said digital code is a Gray coded digital signal; bb) An image sensor according to any one of claims 1 to 6, wherein preferably the relationship between the swept analog input signal and the digital code is changed between a storage ramp and a comparison ramp.

8. a) the swept analog input signal is a linear ramp, or b) the swept analog input signal is a non-linear ramp; or c) the swept analog input signal is varied during operation; and / or d) the period of the swept analog input signal is varied during operation; and / or e) the repetition of said swept analog input signal may be interrupted for short or long periods during operation; and / or 8. The image sensor of claim 7, wherein f) a repeat rate of the swept analog input signal is selected from one or more repeat rates that correspond to a light source modulation of the optical signal illuminating the photosensor, preferably to enhance or suppress event output.

9. the image sensor further includes at least one reference pixel configured to set a swept ramp limit; Preferably, said at least one reference pixel is positioned outside said array of pixels, and / or 9. An image sensor according to claim 7 or 8, wherein preferably said at least one reference pixel is biased under reference conditions.

10. the digital comparison circuit (14) is configured to perform a comparison with the stored level during a sweep of the swept analog input signal; or The image sensor of any one of claims 7 to 9, wherein the digital comparison circuit (14) is configured to perform a comparison with the stored level after the sweep of the swept analog input signal.

11. a) the digital comparison circuit (14) is configured to perform the comparison using static logic in the acquisition circuit; or b) the digital comparison circuit (14) is configured to perform the comparison using dynamic logic in the acquisition circuit; and / or c) the digital comparison circuit (14) is configured to perform a comparison with hysteresis; and / or d) the digital comparison circuit (14) is configured to perform comparisons requiring a difference that is changed from time to time during operation.

12. the digital output circuit (15) is configured to generate an output depending on the comparison results of the digital comparison circuit (14) for adjacent pixels; or the digital output circuit (15) is configured to generate an output depending on a function of adjacent pixels in a fixed configuration or in a configuration that is changed during operation, preferably the function being an average function; and / or An image sensor according to any preceding claim, wherein the digital output circuit (15) is configured to generate an output if the rate of change of the comparison exceeds a certain threshold.

13. a) the digital output circuit (15) is configured to generate an output dependent on the output at adjacent pixels; or b) the digital output circuit (15) is configured to generate an output in response to the stored levels at adjacent pixels; or An image sensor according to any preceding claim, wherein c) the digital output circuit (15) is configured to generate an output in dependence on a plurality of stored values.

14. a) the digital output circuit (15) is configured to generate an event output indicative of the direction of the changed level; and / or b) the digital output circuit (15) is configured to generate an event output indicative of only the level changed in one direction; and / or c) the digital output circuit (15) is configured to generate an event output indicative of the magnitude of the altered level; and / or d) the digital output circuit (15) is configured to generate an event output indicative of illumination intensity before and / or after the changed level.

15. a) the digital storage circuit (13) is configured to provide the stored digital representation at an output line of the array of pixels; and / or b) the digital storage circuit (13) is configured to provide the stored digital representation at the output lines of the array of pixels selectively for pixels having an event output; and / or An image sensor according to any one of claims 1 to 14, wherein c) the digital storage circuit (13) is arranged to provide the stored digital representation at the output line by means of an event column line.

16. the sensor circuit (11) is configured to receive feedback from the digital storage circuit (13) corresponding to the stored digital level or to the stored digital levels of adjacent pixels, the sensor circuit (11) being configured to generate V SIG in response to the feedback; and / or 16. The image sensor of any one of claims 1 to 15, wherein the A / D circuit (12) is configured to receive feedback from the stored digital level or the stored digital levels of adjacent pixels and to adjust the event output in response to the feedback.

17. the sensor circuit (11) is configured to provide an analog representation of VSIG; and / or The image sensor of any one of claims 1 to 16, wherein the image sensor is built from multiple semiconductor layers, each level being optimized for the function of the semiconductor layer.

Images