Event sensor and method for generating data stream of event data

Abstract

The present invention relates to an event sensor and a method of generating a data stream of event data in response to light incident on a pixel array (10). The event sensor comprises: for each pixel, at least one photodetector (1) for generating a detector signal; a signal converter (2) for repeatedly generating and storing sampling values on the basis of the detector signals sampled at sampling intervals; and a read-out processor (4) for reading out the sampled values and generating event data. According to the invention, the pixels of the pixel array (10) are divided into pixel blocks, each pixel block comprising two or more pixels of the pixel array (10), and the readout processor (4) is configured to process the sampled values in block units.

Description

Technical Field

[0001] The present invention relates to an event sensor and a method of generating a data stream of event data in response to light incident on a pixel array. Background Art

[0002] In some machine vision tasks, such as tracking, localizing, and detecting moving objects, machine vision algorithms require input with high temporal resolution and extremely low latency. These tasks do not require the processing of large amounts of redundant or irrelevant information. Therefore, these tasks require intelligent image sensors with high temporal resolution, low latency, and minimal data redundancy.

[0003] US 7,728,269 B2 proposes an unconventional sensor design that encodes the temporal contrast of the scene captured by its light sensor. By encoding the temporal contrast, the temporal redundancy in the output data of the image sensor is almost eliminated, thereby generating activity-driven sparse data in the form of polarity events. This means that each polarity event consists of a polarity, i.e. the sign of the temporal contrast, and a pixel coordinate. This event-based temporal contrast sensor concept, or event sensor concept, provides several unique advantages that traditional image sensors cannot provide: high dynamic range, high temporal resolution, low latency, and low data redundancy. Therefore, the event sensor concept has been applied to many alternative designs or improved designs, and is also the basis of the present invention.

[0004] The polarity event may be encoded per single pixel, wherein the polarity event includes the polarity and coordinates of the pixel. Alternatively, to improve event throughput under limited output speed capabilities, the polarity event may be encoded per a group of adjacent pixels, wherein the polarity event includes the polarity and coordinates of the group of pixels.

[0005] The design proposed in US 7,728,269 B2 exploits the subthreshold characteristics of MOSFETs to achieve logarithmic current-to-voltage conversion for high dynamic range. Because it measures temporal contrast rather than absolute intensity, this design is immune to inter-pixel offset mismatch in the logarithmic current-to-voltage conversion. However, this design still suffers from inter-pixel gain mismatch in both the logarithmic current-to-voltage conversion and the change detection stages. Furthermore, this design uses a storage capacitor to store past light intensity-related signals, which can lead to various leakage events, resulting in errors in polarity events due to leakage.

[0006] EP 4,064,686 A1 describes an improved event sensor, the entire contents of which are incorporated herein by reference. The existing event sensor can generate an event data stream in response to light incident on a pixel array, and the event sensor includes at least one photodetector for each pixel on the pixel array, and the photodetector is configured to generate a detector signal in response to the light incident on the pixel. For each pixel or a group of pixels, the event sensor includes a signal converter connected to the photodetector, and the signal converter is configured to repeatedly generate and store sampled values ​​based on the detector signal sampled at a sampling interval. In addition, the event sensor includes a readout processor connected to the signal converter, and the readout processor is configured to derive a cumulative pixel value based on one or more of the sampled values, wherein the cumulative pixel value corresponds to the accumulation of the detector signal within an interval count sampling interval; and is configured to generate a pixel event of the event data based on the cumulative pixel value and the interval count.

[0007] Furthermore, the event sensor described in EP 4,064,686 A1 is capable of generating not only polarity events but also higher-level events. These higher-level events can include not only the signs of temporal contrast, temporal difference, spatial contrast, spatial difference, and / or absolute intensity, but also their magnitudes. Compared to polarity events, which encode visual input in a lossy manner, higher-level events encode visual input in a lossless manner, thus better supporting more demanding visual tasks such as object recognition and identification.

[0008] However, the additional information carried by advanced events also requires more bits to encode and transmit. Therefore, advanced events require event sensors to have stronger output speed capabilities, which may become a performance bottleneck when event sensors integrate a large number of pixels, that is, have high spatial resolution, and / or operate at high speed, that is, have high temporal resolution, and / or when visual input generates many events in a short period of time. To meet the increasing performance requirements of modern vision systems, there is an urgent need for event sensors that can encode visual input not only in a lossless manner but also in an efficient manner, thereby minimizing the number of bits required to encode the information of each pixel. Summary of the Invention

[0009] The event sensor with the features described in claims 1 and 6 and the method with the features described in claims 16 and 17 provided by the present invention can solve the above-mentioned problems and can encode visual input in a lossless and efficient manner, thereby minimizing the number of bits required to encode the information of each pixel.

[0010] Further advantageous embodiments of the invention are the subject matter of the dependent claims.

[0011] Specifically, the present invention provides an event sensor, comprising: a pixel array (10) composed of pixels, and configured to generate a data stream of event data in response to light incident on the pixel array (10); wherein the pixels of the pixel array (10) are divided into pixel blocks, each pixel block including two or more pixels of the pixel array (10), and the event sensor comprises:

[0012] - for each pixel in the pixel array (10), there is at least one photodetector (1), the photodetector (1) being configured to: generate a detector signal in response to light incident on the pixel;

[0013] - for each pixel or a group of pixels, there is a signal converter (2) connected to the photodetector (1), the signal converter (2) being configured to: repeatedly generate and store sampled values ​​based on the detector signal sampled at sampling intervals;

[0014] - a readout processor (4), said readout processor (4) being connected to said signal converter (2) and configured to: perform the following steps for each pixel block:

[0015] a) deriving, for each pixel in the pixel block, a cumulative pixel value based on one or more of the sampled values, wherein the cumulative pixel value corresponds to the accumulation of the detector signal within a number of sampling intervals;

[0016] b) deriving a cumulative block value based on the cumulative pixel values ​​of the pixels in the pixel block;

[0017] c) Under the condition that the cumulative block value exceeds the cumulative block value threshold and / or the interval count exceeds the count threshold, perform the following steps d), f1), f2), f3), and f4):

[0018] d) determining a cumulative block value threshold and / or a count threshold for subsequent accumulation;

[0019] f1) defining a search area, where the number of pixels included in the search area is equal to or greater than the number of pixels in the pixel block and includes the pixel block, so that the search area contains one or more candidate blocks, wherein each candidate block is geometrically identical to the pixel block, so that the candidate block can be geometrically overlapped with the pixel block only by translation, and the pixel block itself is also a candidate block;

[0020] f2) performing an evaluation procedure for each candidate block with reference to the pixel block, the evaluation procedure comprising the following steps:

[0021] i) for each pixel in the candidate block, reading a previous accumulated pixel value and a previous interval count;

[0022] ii) determining, for each pixel in the pixel block and the corresponding pixel in the candidate block, at least one contrast threshold and / or at least one difference threshold based on the accumulated pixel value of the pixel, the previous accumulated pixel value of the corresponding pixel, the interval count of the pixel, and / or the previous interval count of the corresponding pixel:

[0023] iii) calculating, for each pixel in the pixel block and the corresponding pixel in the candidate block, one, two, or three of the following three values: a spatiotemporal contrast, a spatiotemporal difference, and a transmission cost; wherein the transmission cost is calculated based on an amount of data required to encode the following comparison results: a comparison result of comparing the spatiotemporal contrast with the contrast threshold and / or a comparison result of comparing the spatiotemporal difference with the difference threshold; and

[0024] iv) calculating a cost function result of the candidate block based on the spatiotemporal contrast, the spatiotemporal difference, the transmission cost, and / or the displacement between the pixel block and the candidate block;

[0025] f3) based on the cost function result, finding the best matching block from the candidate blocks, so that the best matching block is the candidate block associated with the minimum cost function result; and

[0026] f4) generating an event in the event data based on the displacement between the best matching block and the pixel block and / or the comparison result associated with the best matching block.

[0027] In addition, the present invention also provides an event sensor, comprising: a pixel array (10) composed of pixels, and configured to generate a data stream of event data in response to light incident on the pixel array (10); wherein the pixels of the pixel array (10) are divided into pixel blocks, each pixel block including two or more pixels of the pixel array (10), and the event sensor comprises:

[0028] - for each pixel in the pixel array (10), there is at least one photodetector (1), the photodetector (1) being configured to: generate a detector signal in response to light incident on the pixel;

[0029] - for each pixel or a group of pixels, there is a signal converter (2) connected to the photodetector (1), the signal converter (2) being configured to: repeatedly generate and store sampled values ​​based on the detector signal sampled at sampling intervals; and

[0030] - a readout processor (4), said readout processor (4) being connected to said signal converter (2) and configured to: perform the following steps for each pixel block:

[0031] a) deriving, for each pixel in the pixel block, a cumulative pixel value based on one or more of the sampled values, wherein the cumulative pixel value corresponds to the accumulation of the detector signal within a number of sampling intervals;

[0032] b) deriving a cumulative block value based on the cumulative pixel values ​​of the pixels in the pixel block;

[0033] c) Under the condition that the cumulative block value exceeds the cumulative block value threshold and / or the interval count exceeds the count threshold, perform the following steps d), g1), g2), g3), and g4):

[0034] d) determining a cumulative block value threshold and / or a count threshold for subsequent accumulation;

[0035] g1) for each pixel in the pixel block, reading a previous accumulated pixel value and a previous interval count;

[0036] g2) determining, for each pixel in the pixel block, at least one contrast threshold, at least one difference threshold, and / or at least one intensity threshold based on the cumulative pixel value, the previous cumulative pixel value, the interval count, and / or the previous interval count of the pixel;

[0037] g3) calculating a spatiotemporal contrast, a spatiotemporal difference, an intensity value, and / or a transmission cost for each pixel in the pixel block; and

[0038] g4) generating an event in the event data based on a comparison result of comparing the spatiotemporal contrast of each pixel in the pixel block with the contrast threshold, a comparison result of comparing the spatiotemporal difference value of each pixel in the pixel block with the difference threshold, and / or a comparison result of comparing the intensity value of each pixel in the pixel block with the intensity value threshold.

[0039] Furthermore, the present invention provides a method for generating a data stream of event data, the method being responsive to light incident on a pixel array (10) composed of pixels, the pixels of the pixel array (10) being divided into pixel blocks, each pixel block comprising two or more pixels of the pixel array (10);

[0040] The method comprises the following steps, performed for each pixel or a group of pixels:

[0041] generating a detector signal in response to light incident on the pixel;

[0042] repeatedly generating and storing sampling values ​​based on the detector signal sampled at sampling intervals;

[0043] a) deriving, for each pixel in the pixel block, a cumulative pixel value based on one or more of the sampled values, wherein the cumulative pixel value corresponds to the accumulation of the detector signal within a number of sampling intervals;

[0044] b) deriving a cumulative block value based on the cumulative pixel values ​​of the pixels in the pixel block;

[0045] c) Under the condition that the cumulative block value exceeds the cumulative block value threshold and / or the interval count exceeds the count threshold, perform the following steps d), f1), f2), f3), and f4):

[0046] d) determining a cumulative block value threshold and / or a count threshold for subsequent accumulation;

[0047] f1) defining a search area, where the number of pixels included in the search area is equal to or greater than the number of pixels in the pixel block and includes the pixel block, so that the search area contains one or more candidate blocks, wherein each candidate block is geometrically identical to the pixel block, so that the candidate block can be geometrically overlapped with the pixel block only by translation, and the pixel block itself is also a candidate block;

[0048] f2) performing an evaluation procedure for each candidate block with reference to the pixel block, the evaluation procedure comprising the following steps:

[0049] i) for each pixel in the candidate block, reading a previous accumulated pixel value and a previous interval count;

[0050] ii) determining, for each pixel in the pixel block and the corresponding pixel in the candidate block, at least one contrast threshold and / or at least one difference threshold based on the accumulated pixel value of the pixel, the previous accumulated pixel value of the corresponding pixel, the interval count of the pixel, and / or the previous interval count of the corresponding pixel:

[0051] iii) calculating, for each pixel in the pixel block and the corresponding pixel in the candidate block, one, two, or three of the following three values: a spatiotemporal contrast, a spatiotemporal difference, and a transmission cost; wherein the transmission cost is calculated based on an amount of data required to encode the following comparison results: a comparison result of comparing the spatiotemporal contrast with the contrast threshold and / or a comparison result of comparing the spatiotemporal difference with the difference threshold; and

[0052] iv) calculating a cost function result of the candidate block based on the spatiotemporal contrast, the spatiotemporal difference, the transmission cost, and / or the displacement between the pixel block and the candidate block;

[0053] f3) based on the cost function result, finding the best matching block from the candidate blocks, so that the best matching block is the candidate block associated with the minimum cost function result; and

[0054] f4) generating an event in the event data based on the displacement between the best matching block and the pixel block and / or the comparison result associated with the best matching block.

[0055] Furthermore, the present invention provides a method for generating a data stream of event data, the method being responsive to light incident on a pixel array (10) composed of pixels, the pixels of the pixel array (10) being divided into pixel blocks, each pixel block comprising two or more pixels of the pixel array (10), the method comprising the following steps performed for each pixel or a group of pixels:

[0056] generating a detector signal in response to light incident on the pixel;

[0057] repeatedly generating and storing sampling values ​​based on the detector signal sampled at sampling intervals;

[0058] a) deriving, for each pixel in the pixel block, a cumulative pixel value based on one or more of the sampled values, wherein the cumulative pixel value corresponds to the accumulation of the detector signal within a number of sampling intervals;

[0059] b) deriving a cumulative block value based on the cumulative pixel values ​​of the pixels in the pixel block;

[0060] c) Under the condition that the cumulative block value exceeds the cumulative block value threshold and / or the interval count exceeds the count threshold, perform the following steps d), g1), g2), g3), and g4):

[0061] d) determining a cumulative block value threshold and / or a count threshold for subsequent accumulation;

[0062] g1) for each pixel in the pixel block, reading a previous accumulated pixel value and a previous interval count;

[0063] g2) determining, for each pixel in the pixel block, at least one contrast threshold, at least one difference threshold, and / or at least one intensity threshold based on the cumulative pixel value, the previous cumulative pixel value, the interval count, and / or the previous interval count of the pixel;

[0064] g3) calculating a spatiotemporal contrast, a spatiotemporal difference, an intensity value, and / or a transmission cost for each pixel in the pixel block; and

[0065] g4) generating an event in the event data based on a comparison result of comparing the spatiotemporal contrast of each pixel in the pixel block with the contrast threshold, a comparison result of comparing the spatiotemporal difference value of each pixel in the pixel block with the difference threshold, and / or a comparison result of comparing the intensity value of each pixel in the pixel block with the intensity value threshold. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 shows a block diagram of an event sensor architecture according to a preferred embodiment;

[0067] Figure 2 A flowchart illustrating a method of generating a data stream of event data in response to light incident on a pixel array according to a preferred embodiment;

[0068] Figure 3 shows a search area with candidate blocks and best matching blocks superimposed thereon to illustrate the process of detecting relative motion at pixel blocks;

[0069] Figure 4 ad show pixel blocks and various results in the process of generating events based on detected relative motion;

[0070] Figure 5 Shown based on Figure 3 The best matching block shown and Figure 4 An example of the resulting event data shown;

[0071] Figure 6 Shown based on Figure 3 The best matching block shown and Figure 4 The result shown updates the example of the previous division result;

[0072] Figure 7 shows a search area with candidate blocks and best matching blocks superimposed thereon to illustrate the process of detecting relative motion and / or illumination changes at pixel blocks;

[0073] Figure 8 ae shows a pixel block and various results in the process of generating events based on detected lighting changes;

[0074] Figure 9 Shown based on Figure 8 An example of the resulting event data shown;

[0075] Figure 10 Shown based on Figure 7 The best matching block shown and Figure 8 The results shown update the previous division result example. DETAILED DESCRIPTION

[0076] Several exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings.

[0077] In a first aspect of the invention, an event sensor having the features of claim 1 is provided.

[0078] The present invention is based on the sensor architecture described in EP 4,064,686 A1. The event sensor comprises a pixel array composed of pixels and is configured to generate a data stream of event data in response to light incident on the pixel array; wherein the pixels of the pixel array are divided into pixel blocks, each pixel block comprising two or more pixels of the pixel array. The event sensor comprises: for each pixel in the pixel array, there is at least one photodetector, the photodetector being configured to generate a detector signal in response to light incident on the pixel; and, for each pixel or a group of pixels, there is a signal converter connected to the photodetector, the signal converter being configured to repeatedly generate and store sampled values ​​based on the detector signal sampled at a sampling interval.

[0079] The term "light" may refer to electromagnetic radiation of any wavelength. In preferred embodiments, "light" may refer to wavelengths ranging from ultraviolet to near-infrared, inclusive. Therefore, the terms "light" and "electromagnetic (EM) radiation" are used interchangeably herein.

[0080] The pixel array may include a two-dimensional array of W×H pixels, where W and H are both positive integers; W is referred to as the width, which refers to the number of pixels along a first dimension; H is referred to as the height, which refers to the number of pixels along a second dimension perpendicular to the first dimension. The pixel array may also be described as having W columns, each having H pixels, H rows, each having W pixels, and / or W columns and H rows of pixels.

[0081] Each pixel in the pixel array includes at least one photodetector that generates a detector signal in response to light incident on the pixel. The photodetector can be any device that converts incident light, i.e., incident photons, into a detector signal. The detector signal can be any form of signal affected by the incident light, such as a thermal signal or a biochemical signal.

[0082] In particular, the detector signal can be an electrical signal. In this example, the photodetector can be referred to as a photoelectric converter. The photodetector can convert photons into electrons. The photodetector can be a photodiode, in particular, a partially or fully pinned photodiode (PPD). In particular, the photodetector can be configured to generate charge at a certain rate, where the charge generation rate, i.e., the amount of charge generated per unit time, is the detector signal. Therefore, the detector signal can be linearly related to or proportional to the light intensity incident on the photodetector. In particular, the detector signal can be sampled per unit time, i.e., per sampling interval, where each unit time or sampling interval comprises an exposure of the photodetector. In other words, within each sampling interval, the photodetector can first be exposed for a predetermined exposure time, and then the detector signal can be sampled. Therefore, within each sampling interval, the photodetector can first be exposed once, and then the detector signal can be sampled once. In particular, the exposure time can be regular. Because the light intensity incident on the photodetector can vary over time, the detector signal can also vary over time. In other words, a detector signal sampled at one point in time may be different from a detector signal sampled at another point in time. However, just because a detector signal is sampled multiple times at different points in time does not mean that the signal has been replicated into multiple identical detector signals. The detector signal of a single photodetector described in this article is a time-varying variable that can be repeatedly sampled over time.

[0083] In one embodiment, for each pixel, the signal converter is connected to the photodetector of the pixel. In other words, there is a one-to-one correspondence between each pixel and a signal converter. Alternatively, a group of multiple pixels can share a common signal converter. In other words, multiple photodetectors belonging to multiple pixels can be connected to the same signal converter.

[0084] The signal converter is configured to repeatedly sample the detector signal at a sampling interval and repeatedly generate and store sampled values ​​based on the detector signal sampled at the sampling interval, wherein each sampled value depends on (particularly, is linearly related to or proportional to) the detector signal sampled at the corresponding sampling interval.

[0085] Furthermore, the present invention is based on a readout processor configured to operate on pixel blocks defined in the pixel array. Thus, the pixels in the pixel array are divided into pixel blocks, each pixel block comprising two or more pixels, preferably comprising at least 4, 9 or 16 pixels of the pixel array. The pixel blocks are proper subsets of the pixel array, so that the pixel array is divided into a plurality of said pixel blocks. Furthermore, advantageously, all pixel blocks are geometrically identical and are arranged in such a way that any pixel block can be geometrically overlapped with any other pixel block simply by translation. For example, the pixel blocks may comprise 4 rows x 4 columns of pixels. Pixels of the same pixel block may be adjacent or connected to each other, or separated by pixels belonging to other pixel blocks. The definition of the pixel blocks may be performed once during initialization and / or the pixel blocks may be redefined during readout. Alternatively, the definition of the pixel blocks may be predetermined, i.e. hard-coded or hard-wired.

[0086] The readout processor is connected to the signal converter and is configured to derive a cumulative pixel value for each pixel of the pixel block based on one or more of the sampled values; wherein the cumulative pixel value corresponds to the accumulation of the detector signal within an interval count of sampling intervals, and the interval count records the number of sampling intervals, or the number of exposures, during which the detector signal is accumulated or integrated during the accumulation process.

[0087] The sample values ​​and the accumulated pixel values ​​are generated, derived, and stored in digital form, i.e., in binary or base-2 form. Furthermore, as described in EP 4,064,686 A1, the accumulation can be performed in the analog domain, the digital domain, or a hybrid domain; it should be noted that all three methods ultimately result in accumulated pixel values ​​in digital form.

[0088] In one advantageous embodiment, when a plurality of sample values, each corresponding to a detector signal generated within a single sampling interval, are combined in the digital domain, for example by summing, to derive a cumulative pixel value, the accumulation can be described as digital-domain accumulation, i.e., accumulation performed purely in the digital domain. During digital-domain accumulation, the interval count is updated by incrementing the interval count for each sampling interval during the accumulation process, and the cumulative pixel value is also updated by adding the sample value for each sampling interval during the accumulation process to the cumulative pixel value. Both the interval count and the cumulative pixel value can be reset or restarted at the start of a new accumulation process.

[0089] While the interval count is defined for each pixel individually, it is the same for all pixels within a pixel block. In other words, the interval count can be defined both for each pixel and for each pixel block. Therefore, all pixels within a pixel block begin and end accumulation synchronously, generating corresponding accumulated pixel values ​​for all pixels within the same pixel block simultaneously and subsequently processing them all simultaneously in a batch manner.

[0090] The readout processor is configured to: for the pixel block, further derive a cumulative block value based on the cumulative pixel values ​​of the pixels in the pixel block. Specifically, various methods can be applied to process the cumulative pixel values ​​of the pixels in the pixel block to derive the cumulative block value. For example, the cumulative block value can be the arithmetic mean (the sum of the values ​​divided by the number of values), the median (the middle value that divides all values ​​into the larger half and the smaller half), the mode (the most frequently occurring value), or the mid-range value (the arithmetic mean of the maximum and minimum values) of the cumulative pixel values ​​of the pixels in the pixel block. Therefore, the cumulative block value can be the average cumulative pixel value, the median cumulative pixel value, the mode cumulative pixel value, or the mid-range cumulative pixel value of the cumulative pixel values ​​of the pixels in the pixel block.

[0091] Subsequently, the readout processor may perform an accumulation end check for the pixel block at each sampling interval during the accumulation period by comparing the accumulated block value with a cumulative block value threshold and / or comparing the interval count with a count threshold. For the pixel block, when the accumulated block value exceeds the accumulated block value threshold and / or the interval count exceeds the count threshold, that is, when the accumulation end condition is met, the readout processor will end accumulation for the pixel block and then perform a processing iteration on the pixels in the pixel block. On the other hand, when the accumulation end condition is not met, the readout processor will allow accumulation for the pixel block to continue and will not perform a processing iteration on the pixels in the pixel block. In other words, the readout processor is configured to perform one processing iteration for each pixel after each accumulation ends. Therefore, for each pixel block, there is a one-to-one correspondence between one accumulation and one processing iteration, that is, each accumulation corresponds to one processing iteration, and each processing iteration corresponds to one accumulation.

[0092] The "and / or" option described in the foregoing text and in the claims can mean that only the cumulative block value is compared with the cumulative block value threshold, or that only the interval count is compared with the count threshold, or that both comparisons are performed simultaneously. In the latter case, the condition can be determined to be satisfied when the cumulative block value exceeds the cumulative block value threshold, when the interval count exceeds the count threshold, or when both thresholds are exceeded. In this example, the "or" in the claim language is a Boolean "or". In addition, the "or" in the claim language can represent the patent language "or", which means that the readout processor can be configured to only check whether the cumulative block value exceeds the cumulative block value threshold, or only check whether the interval count exceeds the count threshold. This can also apply to other usage scenarios of "and / or", that is, the "or" in this statement can be regarded as a Boolean operator, or as a way to distinguish between two different embodiments.

[0093] According to the present invention, each processing iteration includes the following steps d, f1, f2, f3, and f4 performed on the pixels in the pixel block. The step numbers are only for the purpose of ease of understanding and reference, and do not necessarily represent the order in which the steps are performed.

[0094] Step d: Determine, for the pixel block, a cumulative block value threshold and / or a count threshold for subsequent accumulation; the cumulative block value threshold and / or the count threshold are advantageously determined based on the interval count and the cumulative block value, in particular, based on a block division result obtained by dividing the cumulative block value by the interval count.

[0095] Here and below, for the sake of clarity, the processing iteration being executed on the pixel block is referred to as the "current processing iteration", or simply as the "processing iteration". On the same pixel block, the processing iteration executed just before the current processing iteration is referred to as the "previous processing iteration". The accumulation corresponding to the current processing iteration is referred to as the "current accumulation", or simply as the "accumulation". "Subsequent accumulation" refers to a new accumulation that occurs on the same pixel block after the current accumulation ends. This also means that the accumulation block value threshold and / or count threshold used by the readout processor when performing the accumulation end check during the current accumulation period is determined during the previous processing iteration. Therefore, for each pixel block, there is also a one-to-one correspondence between a current accumulation and a current processing iteration, that is, each current accumulation corresponds to a current processing iteration, and each current processing iteration corresponds to a current accumulation.

[0096] Advantageously, the event sensor of the present invention may include a parameter memory connected to a readout processor, the parameter processor being configured to store the cumulative pixel value, the previous cumulative pixel value, and the previous interval count for each pixel in the pixel array, and to store the interval count for each pixel block in the pixel array.

[0097] Here and below, for the sake of clarity, on the same pixel block, the accumulation of events that occurred before the current accumulation and whose corresponding processing iteration has generated the event data constituting the event data is referred to as the "previous accumulation". Therefore, the "previous accumulated pixel value" and the "previous interval count" refer to the accumulated pixel value and the corresponding interval count derived from the previous accumulation, that is, at the end of the previous accumulation, respectively. It is worth mentioning that the previous accumulated pixel value, the previous interval count, and the previous accumulation do not necessarily correspond to the previous processing iteration, because the previous accumulation does not necessarily occur just before the current accumulation. On the other hand, in this article, the accumulated pixel value and the corresponding interval count derived at the end of the current accumulation may be referred to as the "current accumulated pixel value" and the "current interval count", respectively, or simply as the "cumulative pixel value" and the "interval count".

[0098] During the accumulation period, the readout processor may read the sampled value from the signal converter at every sampling interval, read the accumulated pixel value and the interval count from the parameter memory, update the accumulated pixel value and the interval count accordingly, and then write the updated accumulated pixel value and the updated interval count back to the parameter memory.

[0099] Furthermore, the parameter memory may be configured to store the cumulative block value threshold and / or the count threshold for each pixel block in the pixel array.

[0100] During the current processing iteration, the readout processor may determine the cumulative block value threshold and / or count threshold for subsequent accumulation and store it in the parameter memory. In other words, during the current accumulation period, at each sampling interval, in order to perform the accumulation end check, the readout processor may read the cumulative block value threshold and / or count threshold determined and stored during the previous processing iteration from the parameter memory.

[0101] The parameter memory can be physically a single memory device located in one location, or multiple memory devices distributed across multiple locations. While this is a preferred approach, the parameter memory need not be arranged in an array like the pixel array. In particular, the parameter memory can be based on static random-access memory (SRAM) technology.

[0102] According to an advantageous embodiment, the event sensor may further include a threshold memory connected to the readout processor. The threshold memory may be configured to store a first lookup table. The contents of the first lookup table may be pre-calculated and pre-stored in the threshold memory. In particular, the first lookup table may receive the block division result from the readout processor as input and generate the cumulative block value threshold and / or the count threshold as output to the readout processor, thereby enabling the readout processor to determine the cumulative block value threshold and / or count threshold for subsequent accumulation for the pixel block based on the block division result.

[0103] It should be noted that, in this context, the storage of any parameter or value may include not only storing the parameter or value in its original form when derived or generated, but also the possibility of storing the parameter or value in a different encoding form, such as Gray code or compressed code. Furthermore, in this context, the reading of any parameter or value by the readout processor may include an automatic conversion or decoding step, if necessary.

[0104] Step f1: Define a search area, wherein the number of pixels included in the search area is equal to or greater than the number of pixels in the pixel block and includes the pixel block, so that the search area contains one or more candidate blocks, wherein each candidate block includes the same number of pixels as the pixel block and is geometrically identical to the pixel block, so that the candidate block can be geometrically overlapped with the pixel block only by translation, and the pixel block itself is also a candidate block. Preferably, the search area extends beyond the pixel block on at least one side or all sides. The search area can be centered on the pixel block so that the search area extends symmetrically beyond the edges of the pixel block. The candidate block can be composed of actual pixels or of interpolated pixels, that is, virtual pixels, which are geometrically located between the actual pixels and whose number is calculated or interpolated based on the values ​​of the actual pixels.

[0105] Step f2: For each candidate block, an evaluation procedure is performed with reference to the pixel block to calculate a cost function result for each candidate block. In other words, the cost function result depends on the candidate block and the pixel block. The evaluation procedure includes the following steps:

[0106] i) for each pixel in the candidate block, reading a previous accumulated pixel value and a previous interval count from the parameter memory;

[0107] ii) determining, for each pixel in the pixel block and the corresponding pixel in the candidate block, at least one contrast threshold and / or at least one difference threshold based on the accumulated pixel value of the pixel, the previous accumulated pixel value of the corresponding pixel, the interval count of the pixel, and / or the previous interval count of the corresponding pixel:

[0108] iii) calculating, for each pixel in the pixel block and the corresponding pixel in the candidate block, one, two, or three of the following three values: spatiotemporal contrast, spatiotemporal difference, and transmission cost; and

[0109] iv) calculating a cost function result of the candidate block based on the spatiotemporal contrast, the spatiotemporal difference, the transmission cost, and / or the displacement between the pixel block and the candidate block.

[0110] Here and below, each pixel in the pixel block is geometrically aligned with a corresponding pixel in the candidate block. In other words, a pixel in the pixel block and a corresponding pixel in the candidate block have the same relative position relative to the pixel block and the candidate block, respectively. For example, a pixel in the pixel block may be located in the upper left corner of the pixel block, in which case the corresponding pixel in the candidate block is also located in the upper left corner of the candidate block. Each pixel in the pixel block and the corresponding pixel in the candidate block are associated with a spatiotemporal contrast, a spatiotemporal difference, and / or a transmission cost. Each candidate block is associated with a cost function result.

[0111] Advantageously, the threshold memory may be further configured to store a second lookup table. The contents of the second lookup table may be pre-calculated and pre-stored in the threshold memory. In particular, the second lookup table may receive the cumulative pixel value, the previous cumulative pixel value, the interval count, and / or the previous interval count from the readout processor as input, and generate at least one contrast threshold value and / or at least one difference threshold value as output to the readout processor, thereby enabling the readout processor to determine a contrast threshold value and / or a difference threshold value for each pixel in the pixel block and the corresponding pixel in the candidate block based on the cumulative pixel value of the pixel, the previous cumulative pixel value of the corresponding pixel, the interval count of the pixel, and / or the previous interval count of the corresponding pixel.

[0112] The spatiotemporal contrast between each pixel in the pixel block and the corresponding pixel in the candidate block is calculated based on, i.e., linearly related to or equal to, a quotient of a division result of the pixel and a previous division result of the corresponding pixel, wherein the division result of the pixel is calculated by dividing the cumulative pixel value of the pixel by the interval count of the pixel, and the previous division result of the corresponding pixel is calculated by dividing the previous cumulative pixel value of the corresponding pixel by the previous interval count of the corresponding pixel. As an advantageous example, the spatiotemporal contrast can be calculated as the quotient minus 1, i.e., the quotient minus 1.

[0113] The spatiotemporal difference between each pixel in the pixel block and the corresponding pixel in the candidate block is calculated based on, i.e., linearly related to or equal to, the difference between the division result of the pixel and the previous division result of the corresponding pixel. As an advantageous example, the spatiotemporal difference can be calculated as the difference.

[0114] The transmission cost associated with each pixel in the pixel block and the corresponding pixel in the candidate block is calculated based on, i.e., linearly dependent on, or equal to, the amount of data, i.e., the number of bits or packets, required to encode and / or transmit the following comparison results: the comparison result of comparing the spatiotemporal contrast between the pixel and the corresponding pixel with the contrast threshold, and / or the comparison result of comparing the spatiotemporal difference between the pixel and the corresponding pixel with the difference threshold. As an advantageous example, the transmission cost can be calculated as the amount of data. Therefore, each transmission cost corresponds to a comparison result, and both correspond to the spatiotemporal contrast and / or spatiotemporal difference associated with a pixel in the pixel block and the corresponding pixel in the candidate block.

[0115] Step f3: Based on the cost function result, find the best matching block from the candidate blocks, so that the best matching block is the candidate block associated with the minimum cost function result.

[0116] Step f4: generating an event in the event data based on the displacement between the best matching block and the pixel block and / or the comparison result associated with the best matching block.

[0117] Advantageously, if the best match block is identical to the pixel block, i.e., the associated comparison results are all zero, then only the displacement is encoded and transmitted as part of the event data, i.e., no comparison results need to be transmitted. If there is a mismatch or residual between the best match block and the pixel block, i.e., the associated comparison results are not all zero, then the residual, i.e., the non-zero comparison result, can also be encoded and transmitted as part of the event data. Furthermore, if the best match block is located at the same position in the pixel array as the pixel block, i.e., the associated displacement is zero, then only the non-zero comparison result (if any) is encoded and transmitted as part of the event data, i.e., no displacement needs to be transmitted. Finally, if the best match block is identical to the pixel block and is located at the same position in the pixel array as the pixel block, then no event will be generated for that pixel block.

[0118] According to an advantageous embodiment, the readout processor is configured to calculate the cost function result based on: the sum / average of the absolute / squared values ​​of the spatiotemporal contrast / spatial difference between each pixel in the pixel block and the corresponding pixel in the candidate block; the sum / average of the transmission costs associated with each pixel in the pixel block and the corresponding pixel in the candidate block; the weighted sum / average of the absolute / squared values ​​of the spatiotemporal contrast / spatial difference between each pixel in the pixel block and the corresponding pixel in the candidate block, wherein the absolute / squared values ​​of the spatiotemporal contrast / spatial difference are weighted according to the corresponding transmission costs associated with the pixel and the corresponding pixel; and / or whether the displacement between the pixel block and the candidate block is zero, i.e., whether the displacement needs to be encoded and transmitted as part of the event data. The operator " / " here is an abbreviation for "or", so "sum / average" is an abbreviation for "sum or average", "absolute / squared" is an abbreviation for "absolute value or squared value", and "spatial contrast / spatial difference" is an abbreviation for "spatial contrast or spatiotemporal difference".

[0119] Advantageously, the readout processor is configured to terminate the evaluation procedure early when a perfect matching block is found, i.e., before the evaluation process is performed on each candidate block in the search area, wherein the cost function result associated with the perfect matching block is lower than a preset threshold.

[0120] In a second aspect of the present invention, an event sensor is provided having the features of independent claim 13. The event sensor comprises all the features of the aforementioned event sensor, with the modification that steps f1 to f4 are replaced by steps g1 to g4. In the second aspect, the readout processor is configured to perform steps g1 to g4 instead of steps f1 to f4.

[0121] Step g1: for each pixel in the pixel block, reading a previous accumulated pixel value and a previous interval count from the parameter memory;

[0122] Step g2: determining, for each pixel in the pixel block, at least one contrast threshold, at least one difference threshold, and / or at least one intensity threshold based on the cumulative pixel value, the previous cumulative pixel value, the interval count, and / or the previous interval count of the pixel;

[0123] Step g3: For each pixel in the pixel block, calculate the temporal contrast, temporal difference, intensity value, and / or transmission cost.

[0124] As previously described, the readout processor may determine the contrast threshold and / or the difference threshold for each pixel in the pixel block using the cumulative pixel value, the previous cumulative pixel value, the interval count, and / or the previous interval count of the pixel as inputs to a second lookup table. Alternatively, the intensity threshold may advantageously be determined using a third lookup table stored in the threshold memory. The third lookup table may be configured similarly to the second lookup table, but differs in that it generates at least one intensity threshold as an output to the readout processor.

[0125] The temporal contrast of each pixel is calculated based on, i.e., linearly related to, or equal to, a quotient of a division result of the pixel and a previous division result, wherein the division result is calculated by dividing the cumulative pixel value of the pixel by the interval count of the pixel, and the previous division result is calculated by dividing the previous cumulative pixel value of the pixel by the previous interval count of the pixel. As an advantageous example, the temporal contrast can be calculated as the quotient minus 1, i.e., the quotient minus 1.

[0126] The time difference value of each pixel is calculated based on, ie linearly related to or equal to, the difference between the division result of the pixel and the previous division result. As an advantageous example, the time difference value can be calculated as the difference value.

[0127] The intensity value of each pixel is calculated based on, i.e. linearly related to or equal to, the division result and / or the previous division result of the pixel. As an advantageous example, the intensity value can be calculated as the division result, or the average of the division result and the previous division result.

[0128] According to a second aspect of the present invention, a transmission cost for each pixel is calculated based on, i.e., linearly dependent on or equal to, the amount of data (i.e., the number of bits or packets) required to encode the following comparison results: the comparison result of comparing the temporal contrast of the pixel with the contrast threshold, the comparison result of comparing the temporal difference value of the pixel with the difference threshold, and / or the comparison result of comparing the intensity value of the pixel with the intensity threshold. As an advantageous example, the transmission cost can be calculated as the amount of data. Therefore, each transmission cost corresponds to a comparison result associated with a pixel of the pixel block.

[0129] Step g4: Generate an event in the event data based on a comparison result of comparing the time contrast of each pixel in the pixel block with the contrast threshold, a comparison result of comparing the time difference value of each pixel in the pixel block with the difference threshold, and / or a comparison result of comparing the intensity value of each pixel in the pixel block with the intensity value threshold.

[0130] Here and below, unless stated otherwise, all features described apply to both aspects of the invention.

[0131] Preferably, for the two aspects of the present invention, the readout processor is configured to: for each pixel block, find the majority comparison result, that is, the comparison result with the highest frequency, from the comparison results of the pixels in the pixel block; for each pixel block, find the abnormal comparison result that is different from the majority comparison result; and generate the event in the event data based on the majority comparison result and / or the abnormal comparison result.

[0132] Advantageously, particularly in the first aspect of the present invention, during the evaluation procedure, the readout processor may be configured to calculate the cost function result based on the majority comparison result and / or the anomaly comparison result. Specifically, when calculating the sum / average of the transmission costs associated with each pixel in the pixel block and the corresponding pixel in the candidate block, the transmission cost corresponding to the majority comparison result may be taken into account only once, because the majority comparison result is only encoded and transmitted once in the event data.

[0133] In a further preferred embodiment, the readout processor is configured to find the majority comparison result based on: the frequency of occurrence of comparison results with the same value, and / or the weighted frequency of occurrence of comparison results with the same value, wherein the weighted frequency of occurrence is weighted by the corresponding transmission cost required to encode and / or transmit the comparison results with the same value.

[0134] The features described in the above two aspects of the present invention are intended to address the two most common scenarios where the visual input of an event sensor may generate a large number of events in a short period of time.

[0135] In the first scenario, there is relative motion between the event sensor and an object that forms part of its visual input. In this scenario, conventional event sensors perceive all spatial features of the object as temporal contrast / time difference values ​​and generate events triggered by these temporal contrast / time difference values. However, because these temporal contrast / time difference-induced events are all the result of the same relative motion, encoding and transmitting the relative motion itself is more efficient than encoding and transmitting these temporal contrast / time difference-induced events separately. Therefore, by finding the best matching block for each pixel block, a readout processor with the above-specified characteristics can search for the best matching block to estimate the relative motion, encode the estimated relative motion as the displacement between the pixel block and the best matching block, and generate event data accordingly.

[0136] In the second scenario, a uniform illumination variation exists on the object that constitutes part of the visual input for the event sensor. In this scenario, a conventional event sensor would perceive the entire area of ​​the object reflecting the uniform illumination variation as exhibiting a temporal contrast / temporal difference value and generate temporal contrast / temporal difference-induced events based on all illuminated pixels in that area. However, because these temporal contrast / temporal difference-induced events are all the result of the same uniform illumination variation, many of these temporal contrast / temporal difference-induced events could encode the same comparison result associated with the temporal contrast / temporal difference value. Therefore, by finding the mode comparison result for each pixel block, a readout processor having the above-specified characteristics can encode and transmit the mode comparison result shared by multiple pixels as a single comparison result in the event data, rather than encoding and transmitting the same comparison result shared by multiple pixels individually.

[0137] By definition, the temporal contrast described in the second aspect of the present invention can be considered a special case of the spatiotemporal contrast described in the first aspect of the present invention, and the temporal difference described in the second aspect of the present invention can be considered a special case of the spatiotemporal difference described in the first aspect of the present invention. Therefore, for the sake of brevity, hereinafter, "spatial contrast" may refer to spatiotemporal contrast in a specific embodiment of the first aspect of the present invention, or may refer to temporal contrast in a specific embodiment of the second aspect of the present invention. Similarly, "spatial difference" may refer to spatiotemporal difference in a specific embodiment of the first aspect of the present invention, or may refer to temporal difference in a specific embodiment of the second aspect of the present invention.

[0138] Advantageously, the readout processor is configured to designate the comparison result related to the spatiotemporal contrast and / or spatiotemporal difference value as:

[0139] - zero, if the spatiotemporal contrast does not exceed the contrast threshold and / or the spatiotemporal difference does not exceed the difference threshold, thereby minimizing noise in the event data;

[0140] - contains a portion that is linearly related to or equal to the spatiotemporal contrast and / or the spatiotemporal difference, if the spatiotemporal contrast exceeds the contrast threshold and / or the spatiotemporal difference exceeds the difference threshold, and if the spatiotemporal contrast is within a contrast range / limit and / or the spatiotemporal difference is within a difference range / limit;

[0141] - comprising a portion that is linearly related to or equal to a division result of pixels in the pixel block if the spatiotemporal contrast exceeds the contrast range / limit and / or the spatiotemporal difference exceeds the difference range / limit.

[0142] Here, the contrast range / limit and / or the difference range / limit are predetermined based on the encoding capability of the event data, such as the size of a data packet. Preferably, when the comparison result is non-zero, the comparison result may further include an indicator indicating whether the comparison result includes a portion that is linearly correlated with or equal to the spatiotemporal contrast and / or spatiotemporal difference, or a portion that is linearly correlated with or equal to the division result.

[0143] According to a preferred embodiment, when the comparison result includes a portion that is linearly correlated with or equal to the spatiotemporal contrast and / or spatiotemporal difference, the readout processor is configured to quantize this portion of the comparison result, thereby enabling encoding of this portion of the quantized comparison result using fewer bits or a smaller data packet than encoding of a portion of the comparison result that is linearly correlated with or equal to the pixel division result. In this manner, lossless compression of the event data is achieved because the range of variation of the spatiotemporal contrast and / or spatiotemporal difference is generally smaller than the pixel division result.

[0144] According to an advantageous embodiment, when the comparison result contains a portion that is linearly related to or equal to the spatiotemporal difference value or the temporal difference value, the readout processor is configured to quantize this portion of the comparison result with a step size, i.e. a least significant bit value, that is approximately proportional to the previous division result of the corresponding pixel in the best matching block or the previous division result of the pixel.

[0145] Specifically, the comparison result can be quantized as follows: the least significant bit of the quantized comparison result corresponds to, i.e., its bit weight is equal to, the bit in the previous division result that is a fixed distance (i.e., a fixed number of bits) away from the most significant non-zero bit of the previous division result (in the less significant bit direction). For example, the comparison result before quantization may be "0000001010 (binary)", while the previous division result may be "0001101000 (binary)". If the target bit depth of the quantized comparison result is 5 bits, the least significant bit of the quantized comparison result may correspond to the bit in the previous division result that is 3 bits away from the most significant non-zero bit of the previous division result. In this example, the most significant non-zero bit of "0001101000 (binary)" is the fourth bit from the left, so the least significant bit of the quantized comparison result may correspond to the seventh bit from the left. Therefore, the quantized comparison result may become "00001 (binary)". In this way, the value of the least significant bit of the quantized comparison result can be considered to be approximately proportional to the previous division result.

[0146] In a further advantageous embodiment, the readout processor is configured to encode and / or transmit the pixel division result and / or the portion of the comparison result that is linearly related to or equal to the division result in a floating-point format. The pixel division result may contain more significant digits than required by the application, i.e., have a higher precision. Therefore, to achieve a desired balance between range and precision, the division result and / or the portion of the comparison result that is linearly related to or equal to the division result may be encoded in a floating-point format, such as a binary floating-point format, where the significant digits contain exactly the number of significant digits required by the application, i.e., the exact precision. For example, the division result "0110010110 0010110101 (binary)" contains 19 significant digits, while the application may only require 8 bits of precision. In this example, the corresponding portion of the division result and / or the comparison result may be encoded as "11001011 (binary) × 2¹¹," i.e., "11001011 (binary) multiplied by 2 to the 11th power." This also means that the previous division result of the pixel, the previous division result of the corresponding pixel, the spatiotemporal contrast of the comparison result, the spatiotemporal difference value, the intensity value, and / or the related part can also be encoded and / or transmitted in floating-point format. For clarity, all numerical values ​​listed herein should be interpreted in decimal unless otherwise specified, for example, "11001011 (binary)" should be interpreted in binary.

[0147] Preferably, the readout processor is configured to determine the count threshold as the variable power of a constant base, such that the interval count at the end of each accumulation is always equal to the variable power of the constant base. In this manner, in a floating-point format, the significand of the division result can be based on, i.e., linearly related to, or equal to, the accumulated pixel value, and the exponent of the division result can be based on, i.e., linearly related to, or equal to, the inverse of the variable power of the interval count, i.e., the additive inverse. Similarly, in a floating-point format, the significand of the previous division result can be based on, i.e., linearly related to, or equal to, the previous accumulated pixel value, and the exponent of the previous division result can be based on, i.e., linearly related to, or equal to, the inverse of the variable power of the previous interval count, i.e., the additive inverse. In a floating-point format, the base of the division result and the base of the previous division result can both be based on, i.e., linearly related to, or equal to, the constant base of the interval count or the constant base of the previous interval count, i.e., the constant base of the count threshold. This also means that storing the previous accumulated pixel value and the previous interval count is actually equivalent to storing the previous division result in floating point format.

[0148] Advantageously, the readout processor is configured to calculate and normalize a floating-point spatiotemporal difference value or a floating-point temporal difference value with reference to the division result of the pixels in the pixel block. In other words, the spatiotemporal difference value or the temporal difference value can share the exponent of the division result of the pixels through calculation and normalization.

[0149] Advantageously, the readout processor is configured to designate the comparison result associated with the spatiotemporal difference value in floating point format as:

[0150] - zero, if the significant number of spatiotemporal differences does not exceed the difference threshold, thereby minimizing noise in the event data;

[0151] comprising a portion linearly related to or equal to the significand of the spatiotemporal difference and a second portion reflecting, i.e. encoding, a scaling factor, if the significand of the spatiotemporal difference exceeds a difference threshold and does not exceed a difference range / limit, and if the scaling factor does not exceed a scaling factor range / limit, wherein the scaling factor is defined based on, i.e. linearly related to or equal to, a difference between an exponent of a division result of the pixel and an exponent of a previous division result of the corresponding pixel, or is defined based on, i.e. linearly related to or equal to, a difference between an exponent of a division result of the pixel and an exponent of a previous division result of the pixel;

[0152] - comprising a portion that is linearly related to or equal to the division result of the pixels in the pixel block if the significant digit of the spatiotemporal difference exceeds the difference range / limit, or if the scale factor exceeds the scale factor range / limit.

[0153] Here, the difference range / limit and the scale factor range / limit are also predetermined based on the encoding capability of the event data, such as the size of the data packet. Preferably, when the comparison result is non-zero, the comparison result may further include an indicator, the indicator being used to indicate whether the comparison result includes a portion that is linearly related to or equal to the significant number of the spatiotemporal difference and a second portion that reflects the scale factor, or a portion that is linearly related to or equal to the division result. As an advantageous alternative, when the comparison result is non-zero, the comparison result may always include a second portion that reflects the scale factor, wherein the scale factor may further indicate whether the comparison result includes a portion that is linearly related to or equal to the significant number of the spatiotemporal difference, or a portion that is linearly related to or equal to the division result.

[0154] According to a preferred embodiment, in particular in the first aspect of the present invention, the readout processor is configured to generate an event in the event data if the displacement between the best matching block and the pixel block is not zero, or if the comparison results associated with the best matching block are not all zero. The event in the event data may include the following information: a pixel block address, such as pixel block coordinates relative to the pixel array; a displacement; a majority comparison result; a relative pixel address, such as pixel coordinates relative to the pixel block; an abnormal comparison result; and / or a comparison result.

[0155] Furthermore, according to a preferred embodiment, particularly in the second aspect of the present invention, the readout processor is configured to generate an event in the event data if the comparison result of the pixel block is not all zero. The event in the event data may include the following information: a pixel block address, such as pixel block coordinates relative to the pixel array; a majority comparison result; a relative pixel address, such as pixel coordinates relative to the pixel block; an abnormal comparison result; and / or a comparison result.

[0156] Here, preferably, for both aspects of the present invention, if a mode comparison result is found and is non-zero, the relative pixel address may be used to indicate the location of the abnormal comparison result in the pixel block; if the mode comparison result is not found, or the mode comparison result found is zero, the relative pixel address may be used to indicate the location of the non-zero comparison result in the pixel block. Furthermore, if the mode comparison result found is zero, the mode comparison result may not need to be encoded and transmitted as part of the event data.

[0157] According to a cumulative advantageous embodiment, in particular in the first aspect of the present invention, the readout processor is configured to, upon generation of the event, overwrite the previous accumulated pixel value and the previous interval count for each pixel in the pixel block based on the event, i.e. based on the displacement between the best matching block and the pixel block, and the comparison result associated with the best matching block, in particular:

[0158] i) Regardless of whether the displacement of the pixel block is zero, if the comparison result associated with the pixel is not zero, then:

[0159] - overwriting the previous accumulated pixel value of the pixel based on the previous accumulated pixel value of the corresponding pixel in the best matching block and the associated comparison result; and

[0160] - overwriting the previous interval count of the pixel based on the previous interval count of the corresponding pixel in the best matching block and the associated comparison result;

[0161] ii) if the displacement of the pixel block is not zero, and if the comparison result associated with the pixel is zero, then: overwriting a previous accumulated pixel value of the pixel with a previous accumulated pixel value of a corresponding pixel in the best matching block, and overwriting a previous interval count of the pixel with a previous interval count of a corresponding pixel in the best matching block;

[0162] iii) If the displacement of the pixel block is zero, and if the comparison result associated with the pixel is also zero, then the previous accumulated pixel value and the previous interval count of the pixel are not overwritten.

[0163] In addition, according to a favorable embodiment of accumulation, in particular in the second aspect of the present invention, the readout processor is configured to: once the event occurs, for each pixel in the pixel block, based on the event, that is, based on the comparison result of the pixel block, overwrite the previous cumulative pixel value and the previous interval count; specifically: i) if the comparison result associated with the pixel is not zero, then: based on the previous cumulative pixel value itself and the associated comparison result, overwrite the previous cumulative pixel value of the pixel; and based on the previous interval count itself and the associated comparison result, overwrite the previous interval count of the pixel; ii) if the comparison result associated with the pixel is zero, then do not overwrite the previous cumulative pixel value and the previous interval count of the pixel.

[0164] As previously described, the previous cumulative pixel value and the previous interval count may be used together to encode the previous division result in a floating-point format, and the cumulative pixel value and the interval count may be used together to encode the division result in a floating-point format. Therefore, hereinafter, the previous cumulative pixel value and the previous interval count may be collectively referred to as the previous division result, and the cumulative pixel value and the interval count may be collectively referred to as the division result.

[0165] The purpose of the event is to enable the receiver, for each pixel, to derive a receiver-side division result for that pixel based on the event, specifically the associated comparison result and / or the displacement encoded and transmitted by the event, given the previous receiver-side division result for that pixel or the previous receiver-side division result for the corresponding pixel in the best matching block, and to derive the result as close as possible to the division result for that pixel (i.e., the readout processor-side division result for that pixel). However, due to the potential presence of quantization noise in the comparison results, there may be discrepancies between the receiver-side division result and the readout processor-side division result. Because the receiver-side division result is used in turn to derive future receiver-side division results, it is important to prevent this potential discrepancy from accumulating over time. Therefore, upon the occurrence of the event, the readout processor updates the previous division result for that pixel by replicating the receiver's derivation of the receiver-side division result, i.e., updating the previous division result for that pixel based on the associated comparison result and the previous division result for the pixel itself or the corresponding pixel. The updated previous division result will in turn be used by the readout processor as the basis for generating future events. In this way, the readout processor takes into account the potential differences introduced by each event when generating future events, effectively preventing such potential differences from accumulating over time between the event sensor and the receiver.

[0166] According to another aspect of the present invention, a method for generating a data stream of event data in response to light incident on a pixel array composed of pixels is provided. Any features and advantages described herein in relation to event sensors may also be similarly applied to the method.

[0167] Figure 1The event sensor architecture according to a preferred embodiment is shown. The event sensor includes a two-dimensional pixel array 10 consisting of W columns x H rows of pixels. Although various configurations for implementing the pixel array are possible, in one advantageous embodiment, each pixel of the pixel array includes exactly one photodetector, preferably a photodiode that converts photons into electrons, and one photodiode. There are 16 pixels, i.e., 4 rows x 4 columns of pixels, connected to a signal converter, which includes 16 analog converters, preferably charge-to-voltage converters or electron-to-voltage (e / V) converters, and one analog-to-digital converter (ADC). Other advantageous embodiments may configure the signal converter to also implement programmable gain and / or include an analog amplifier.

[0168] exist Figure 1 In the illustrated event sensor, a bias generator 51 provides a bias voltage to the pixel array 10. An ADC controller 52 provides a ramp / reference voltage and a bit pattern in Gray code format to the pixel array 10. A row address encoder 53 and a column address encoder 54 encode pixel block addresses in the pixel array 10.

[0169] Advantageously, the readout processor 4 consists of processing blocks 41 with W / 16 (i.e., one-sixteenth of W) columns and one row. Each processing block 41 is configured to process four columns of pixel blocks in the pixel array 10, one pixel block at a time. All processing blocks 41 are capable of simultaneous, parallel processing, so the readout processor 4 can simultaneously process a quarter row of pixel blocks (i.e., one-quarter of a row) at a time. The readout processor 4 reads sample values ​​and / or sample baselines from the pixel array 10 and reads / writes pixel parameters from / to the parameter memory 3. The parameter memory 3 is preferably SRAM-based. Read and write operations between the pixel array 10, the readout processor 4, and the parameter memory 3 are coordinated by a readout controller 55. The readout processor 4 is connected to a threshold memory 7, which stores two lookup tables: a first lookup table and a second lookup table. The parameter memory 3 and the threshold memory 7 can be loaded from an external non-volatile memory 62, such as flash memory, and / or programmed or updated by an external computer 61 via a memory configurator 56.

[0170] The parameter memory 3 stores pixel parameters, which may include: an accumulated pixel value (hereinafter referred to as APV), a previous APV, an interval count, a previous interval count, and a count threshold.

[0171] The interval count and count threshold are specific to each pixel block, meaning that each pixel block is associated with an interval count and a count threshold, allowing pixels in the same pixel block to start and end accumulation synchronously. The remaining pixel parameters are pixel-specific, meaning that each pixel is associated with an APV, a previous APV, and a previous interval count. Using state-of-the-art 28nm SRAM technology, the silicon area of ​​the entire parameter memory 3 serving a 1-megapixel pixel array 10 is estimated to be less than 8 square millimeters.

[0172] A first lookup table in threshold memory 7 receives as input the block division result (the derivation of which will be described later) from the readout processor and generates as output a count threshold value for the readout processor. A second lookup table in threshold memory 7 receives as input the APV value from the readout processor and generates as output a difference threshold value for the readout processor. These two lookup tables guide the readout processor in dynamically determining the count threshold value and the difference threshold value as it processes different pixel blocks and different pixels at different times. The contents of these two lookup tables can be determined based on a one-time calibration procedure.

[0173] According to a preferred embodiment, a method of generating a data stream of event data in response to light incident on a pixel array 10 is described in Figure 2 The process is illustrated in flowchart form in FIG. In step 101, an APV is derived for each pixel in a pixel block. The APV corresponds to the accumulation of the detector signal during a sampling interval of sample count. The APV is pixel-specific, while the interval count can be pixel block-specific. Then, in another step 102, the average APV of the pixels in the pixel block is derived as the accumulated block value (ABV) for the pixel block. Thus, each pixel block now has two pixel block-specific parameters: the ABV and the interval count.

[0174] During the accumulation period, at each sampling interval, a comparison step 103 is performed in which the interval count is compared with a count threshold previously determined during a previous processing iteration. If the comparison shows that the interval count exceeds the count threshold, then a condition is satisfied (i.e., the condition is positively satisfied), under which condition the accumulation of the pixel block is terminated and one processing iteration is performed on the pixels in the pixel block. If the comparison shows that the interval count does not exceed the count threshold, then the condition is not satisfied, and accumulation of the pixel block is continued without performing a processing iteration on the pixels in the pixel block. Figure 2 In the example, the judgment as to whether the condition is satisfied is represented by a decision symbol "+" or "-" 103, respectively.

[0175] Each processing iteration consists of five steps 110-114. In the threshold determination step 110, a count threshold for subsequent accumulation is determined for the pixel block based on a block division result calculated by dividing the ABV by the interval count by reference to a first lookup table. Furthermore, in the first aspect of the present invention, steps 111-113 are performed to detect relative motion at the pixel block, thereby generating an event in the final step 114. These steps 111-114 correspond to steps f1 to f4 of claim 1, respectively. In the search area definition step 111, a search area is determined, the search area containing a plurality of candidate blocks. In other words, the candidate blocks are defined together with the search area. In the evaluation procedure step 112, the following sub-steps are performed for each candidate block: first, the previous APV and the previous interval count of each pixel in the candidate block are read; second, for each pixel in the pixel block and the corresponding pixel in the candidate block, a difference threshold is determined based on the APV of the pixel by referring to a second lookup table; third, a spatiotemporal difference and a transmission cost are calculated for each pixel in the pixel block and the corresponding pixel in the candidate block; fourth, a cost function result is calculated for each candidate block based on the spatiotemporal difference and transmission cost associated with each pixel in the pixel block and the corresponding pixel in the candidate block. In the subsequent step 113, the cost function results of all candidate blocks are compared to find the best matching block with the minimum cost function result. In the final step 114, an event in the event data can be generated based on the displacement, i.e., relative motion, between the best matching block and the pixel block and / or the comparison result associated with the best matching block.

[0176] Example 1

[0177] In the first example, by Figure 3-6 The above steps 111 to 114 are described. Figure 3 A search area 20 is shown which geometrically covers or spans an area of ​​the pixel array. Figure 3 Each small square in represents a pixel, thus illustrating an area of ​​the pixel array of 37 x 37 pixels. Search area 20 itself consists only of pixels with numbers and therefore consists of 19 x 19 pixels. That is, in this particular case, every two adjacent pixels in the search area are separated by a pixel in the pixel array that is not part of the search area. Each number shown in a pixel of search area 20 represents the previous division result for that pixel—that is, the result of dividing the previous APV by the previous interval count.

[0178] Figure 4 a shows a pixel block 21 being processed. Figure 4Each small square in represents a pixel. The search area consists of 4×4 pixels containing numbers and geometrically covers or spans a 13×13 pixel area in the pixel array. That is, in this particular case, every two adjacent pixels in the pixel block are separated by three pixels in the pixel array that are not part of the pixel block. Each number shown in pixel block 21 represents the result of dividing the APV by the interval count. Figure 3 and Figure 4 The arrowed axes shown in a establish a reference system whose origin (0,0) is located at the center of the pixel block 21.

[0179] Figure 3 The search area 20 shown contains 13×13 candidate blocks. The center of the search area is located at the origin of the pixel block reference system. Figure 3 In the figure, two different candidate blocks 22 are outlined with bold lines. In the pixel block reference frame, the origin of one of them (as indicated by the dashed arrow) is (4, -5), and the origin of the other is (-4, -3). For each candidate block 22 in the search area 20, the spatiotemporal difference between each pixel in pixel block 21 and the corresponding pixel in candidate block 22 is calculated. Figure 4 b shows the calculation result of the candidate block located at (4, −5), which will be subsequently determined as the best matching block 23. Figure 4 The result in b is all zero except for two "1" values ​​and one "2" value. In a further step, for each candidate block, the associated spatiotemporal difference value is compared with a difference threshold, such as "1", and a comparison result is derived, so that each spatiotemporal difference value corresponds to a comparison result. Figure 4 c shows the comparison result of the best matching block. In the current figure, only one value greater than zero is shown, namely the value "2", which exceeds the difference threshold.

[0180] The best matching block 23 is found from the candidate blocks 22 based on a cost function that calculates a weighted sum of the absolute values ​​of the spatiotemporal differences associated with the candidate block 22, where the absolute value of each spatiotemporal difference is weighted according to the transmission cost of transmitting the corresponding comparison result. In this example, the center of the best matching block 23 is located at (4, -5) in the pixel block reference system. Therefore, the displacement between the pixel block and the best matching block is detected as (4, -5) and is encoded as part of the event data to indicate the detected relative motion. In addition, the non-zero comparison result associated with the best matching block 23 is also encoded as part of the event data. In this specific example, the only non-zero comparison result is "2", which is the same as Figure 4 The pixel at the lower left corner of the pixel block 21 outlined by the bold line in d is associated.

[0181] The event data corresponding to pixel block 21 is a string of 8-bit data packets, including Figure 5 The ad part shown in ad. The a part includes one or more data packets, encoding the pixel block address (here Figure 5 Part b includes a data packet that encodes the displacement between pixel block 21 and the best matching block 23 (here Figure 5 The data packet of vector (4, -5) is encoded in b. Part c includes two data packets, encoding the relative pixel address of the non-zero comparison result associated with the best matching block 23 (here Figure 5 c). Part d includes one or more packets that encode the signed magnitude of the non-zero comparison result (here Figure 5 d). Figure 5 In the data packet of part c shown in FIG. 3 , each bit represents a pixel in pixel block 21 , “1” indicates that there is a non-zero comparison result associated with the pixel, and “0” indicates that there is no non-zero comparison result associated with the pixel. The first bit in the first data packet of part c, i.e., the leftmost bit, represents the upper left pixel in pixel block 21 , and the last bit in the second data packet of part c, i.e., the rightmost bit, represents the lower right pixel in pixel block 21 . And, Figure 5 The data packet shown in d encodes the non-zero comparison result "2" associated with the lower left corner pixel of pixel block 21. Figure 4 As outlined by the thick line in d.

[0182] like Figure 6 As shown, once event data corresponding to pixel block 21 is generated, the previous division result of pixel block 21 is updated based on the event data. The lower left pixel in the pixel block is associated with a non-zero comparison result of "2". Therefore, the previous division result of the lower left pixel in the pixel block will be overwritten by the sum of the previous division result of the corresponding lower left pixel in the best matching block and the associated non-zero comparison result "2". The remaining pixels in the pixel block are not associated with a non-zero comparison result, but a displacement of (4, -5) between the pixel block and the best matching block is detected. Therefore, the previous division results of the remaining pixels in the pixel block are respectively overwritten by the previous division results of the corresponding pixels in the best matching block. In other words, the updated previous division result of pixel block 21 is obtained by adding the previous division result of the best matching block 23 to the non-zero comparison result associated with the best matching block on a pixel-by-pixel basis.

[0183] Example 2

[0184] Figure 7-10 Another example is shown where, while searching for the best matching block, no relative motion is detected, but a relatively uniform illumination change is detected for the pixel block. Figure 7-10 Should be explained in Example 1 Figure 3-6The interpretation methods are interpreted separately. Figure 7 The defined search area is shown. Figure 8 a shows a pixel block 21 being processed. Figure 7 and Figure 8 The arrowed axes shown in a establish a reference system whose origin (0,0) is located at the center of pixel block 21. Each number shown in pixel block 21 represents a division result. Each number shown in search area 20 represents a previous division result.

[0185] Figure 7 The candidate block 22 outlined in bold in FIG has its origin at (0,0) in the pixel block reference system. For each candidate block 22 in the search area 20, the spatiotemporal difference between each pixel in the pixel block 21 and the corresponding pixel in the candidate block 22 is calculated. Figure 8 b shows the Figure 7 The calculation result of the candidate block located at (0,0) in the image is then determined as the best matching block 23. In a further step, for each candidate block, the associated spatiotemporal difference value is compared with a difference threshold value, such as "1", and a comparison result is derived, so that each spatiotemporal difference value corresponds to a comparison result. The comparison result of the best matching block is as follows: Figure 8 c. The best matching block 23 is found based on a cost function that calculates a weighted sum of the absolute values ​​of the spatiotemporal differences associated with candidate block 22, where each absolute value of the spatiotemporal difference is weighted by the transmission cost of encoding and transmitting the corresponding comparison result. In this example, the center of best matching block 23 is located at (0,0) in the pixel block reference frame. Therefore, the displacement between pixel block 21 and best matching block 23 is found to be zero, meaning no relative motion is detected.

[0186] like Figure 8 As shown in FIG. 3 , among all the comparison results related to the best matching block 23, the most common comparison result is “3”. The pixels associated with the most common comparison result are all in Figure 8 d is outlined with a thick line. The mode comparison result is encoded as part of the event data. In a further step, abnormal comparison results are found among all comparison results associated with the best matching block 23. The pixels associated with the abnormal comparison results are all in Figure 8 The abnormal comparison result is also encoded as part of the event data.

[0187] The event data corresponding to pixel block 21 is a stream of 8-bit data packets, including Figure 9 The ad part shown in ad. The a part includes one or more data packets, encoding the pixel block address (here Figure 9Since the offset in this example is zero, there is no packet encoding the offset, which is similar to Figure 5 The data packet shown in b is different. Part b includes a data packet that encodes the signed magnitude of the mode comparison result associated with the best matching block (here Figure 9 The data packet in b encodes the value “3”). Furthermore, part c includes two data packets that encode the relative pixel addresses of the abnormal comparison results associated with the best matching block (here Figure 9 c). And part d includes one or more packets encoding the signed magnitude of the abnormal comparison result (here Figure 9 d, encoding the values ​​“2,” “2,” “2,” and “0.”

[0188] like Figure 10 As shown, once event data corresponding to pixel block 21 is generated, the previous division result of pixel block 21 is updated based on the event data. Figure 8 As shown in Figure c, all pixels in the pixel block except the bottom-right corner pixel are associated with non-zero comparison results. Therefore, their previous division results are overwritten by the sum of the previous division results and the corresponding comparison results of the corresponding pixels in the best-matching block. More specifically, for pixels associated with the majority comparison results, their previous division results are overwritten by the sum of the majority comparison result and the previous division results of the corresponding pixels in the best-matching block; while for pixels associated with outlier comparison results, their corresponding previous division results are overwritten by the sum of the corresponding outlier comparison results and the previous division results of the corresponding pixels in the best-matching block. The bottom-right corner pixel in the pixel block is associated with a comparison result of "0," and the displacement between the pixel block and the best-matching block is also zero. Therefore, the previous division result of the bottom-right corner pixel in the pixel block is not updated. In other words, the updated previous division result of pixel block 21 is obtained by adding the previous division result of best-matching block 23 to the non-zero comparison results associated with that best-matching block, pixel by pixel. In this example, when the majority comparison result and the abnormal comparison result are encoded into the event data, the previous division result after the pixel block 21 is updated is obtained by dividing the previous division result of the best matching block 23, Figure 8 The mode comparison results associated with the best matching block shown in d, and Figure 8 The abnormal comparison results associated with the best matching block shown in e are obtained by adding them pixel by pixel.

[0189] Example 3

[0190] An advantageous method of deriving and encoding the comparison result is described below.

[0191] First, the division result is encoded in floating-point format, where the significand is based on the APV, and the base and exponent are based on the interval count. Specifically, by always determining the count threshold as a variable power of a constant base (e.g., base 2), the interval count at the end of each accumulation is always equal to the variable power of the constant base. For clarity, all values ​​listed herein should be interpreted in decimal unless otherwise specified; for example, "0001110010 (binary)" should be interpreted in binary.

[0192] For example, the APV could be "114" and the interval count could be "2¹²." Therefore, the result of dividing the APV by the interval count can be encoded as "114 × 2^(-12)" (i.e., "114 multiplied by 2 raised to the power of -12"). The significand "114" or "0001110010 (binary)" is derived directly from the APV, while the exponent "-12" is derived from the inverse of the variable exponent "12" in the interval count. The base of the division result uses the constant base "2." Furthermore, the exponent of the division result can be encoded as "0011 (binary)" using the exponent code.

[0193] Similarly, the previous APV could be "208" and the previous interval count could be "2¹³." Therefore, the previous division result could be encoded as "208 × 2^(-13)." The significand "208," or "0011010000 (binary)," is derived directly from the previous APV, while the exponent "-13" is derived from the inverse of the variable exponent "13" of the previous interval count. The radix of the previous division result also uses the constant radix "2." Furthermore, the exponent of the previous division result can also be encoded as "0010 (binary)" using the exponent code.

[0194] The index may be derived based on the interval count or the previous interval count and encoded by an index code according to the following table:

[0195] surface:

[0196] Index code index interval count / previous interval count

[0197] (binary)

[0198] 0000 -15 2^15

[0199] 0001 -14 2^14

[0200] 0010 -13 2^13

[0201] 0011 -12 2^12

[0202] 0100 -11 2^11

[0203] 0101 -10 2^10

[0204] 0110 -9 2^9

[0205] 0111 -8 2^8

[0206] 1000 -7 2^7

[0207] 1001 -6 2^6

[0208] 1010 -5 2^5

[0209] 1011 -4 2^4

[0210] 1100 -3 2^3

[0211] 1101 -2 2^2

[0212] 1110 -1 2^1

[0213] 1111 0 2^0

[0214] In a first step, the previous division result is normalized with reference to the exponent of the division result to obtain a normalized previous division result, such that the significand of the normalized previous division result, i.e., "0001101000 (binary)" or "104", together with the exponent of the division result, "-12", still encodes the previous division result, i.e., '104×2^(-12)' is equivalent to '208×2^(-13)'.

[0215] In the second step, the spatiotemporal difference between the division result and the normalized previous division result is calculated. The exponent of the spatiotemporal difference is directly the exponent of the division result, and the significand of the spatiotemporal difference is calculated as the difference between the significand of the division result and the significand of the normalized previous division result. Therefore, the significand of the spatiotemporal difference can be calculated as "0000001010 (binary)", or "10".

[0216] In the third step, the significand of the spatiotemporal difference is compared with a difference threshold (e.g., "6"). Finally, in this particular example, the significand of the spatiotemporal difference exceeds the difference threshold, so the comparison result encodes an 8-bit quantized difference value consisting of a 1-bit sign, a 5-bit magnitude, and a 2-bit scale factor. The sign of the quantized difference is directly the sign of the spatiotemporal difference. The magnitude of the quantized difference is based on the most significant non-zero bit of the significand "0001101000 (binary)" of the normalized previous division result. In this example, the most significant non-zero bit of "0001101000 (binary)" is the fourth bit from the left, while the magnitude of the quantized difference retains 5 bits of information from the significand "0000001010 (binary)" of the spatiotemporal difference, ranging from the bit to the left of the fourth bit (i.e., the third bit from the left) to the seventh bit from the left. Therefore, the magnitude of the quantized difference can become "00001 (binary)." The scale factor for the quantized difference encodes the ratio between the interval count and the previous interval count according to the following table:

[0217] Scale Factor (Binary) Meaning

[0218] 00 Interval count = previous interval count

[0219] 01 interval count = previous interval count / 2

[0220] 10 Interval count = previous interval count * 2

[0221] 11 Out of range

[0222] Therefore, the scale factor in this example is determined to be "01 (binary)".

[0223] Example 4

[0224] Another method of deriving and encoding the comparison result will be described below.

[0225] First, a method similar to that shown in Example 3 is used to encode the division result in floating-point format, where the significand is based on the APV, and the base and exponent are based on the interval count. Again, the count threshold is determined as a variable power of a constant base, for example, 2, so that the interval count at the end of each accumulation is always equal to the variable power of the constant base. For clarity, all values ​​listed herein should be interpreted in decimal unless otherwise noted.

[0226] The exponent code, the correspondence table between exponent and interval count, and the scale factor table in Example 3 can still be used.

[0227] This time, the APV can be "114" and the interval count can be "2^8." Therefore, the division result of the APV divided by the interval count can be encoded as "114 × 2^(-8)." The significand based on the APV is "114" or "0001110010 (binary)," and the exponent based on the interval count is "-8," which is further encoded as "0111 (binary)" using the exponent code. The base of the division result uses the constant base "2."

[0228] Similarly, the corresponding previous APV may be "518" and the corresponding previous interval count may be "2^11." Therefore, the corresponding previous division result may be encoded as "518×2^(-11)," where the significand based on the previous APV is "518" or "1000000110 (binary)" and the exponent based on the previous interval count is "-11," which is further encoded as "0100 (binary)" using the exponent code. The radix of the previous division result also uses the constant radix "2."

[0229] In this example, since the interval count is less than half of the previous interval count, which is out of range for the scale factor, the comparison result no longer encodes the quantized difference as in Example 3.

[0230] Instead, the comparison result directly encodes a 16-bit intensity value, which is composed of the 14-bit division result and a 2-bit scaling factor. The 14-bit division result is further composed of a 10-bit significand based on the APV, namely "0001110010 (binary)", and a 4-bit exponent code based on the interval count, namely "0111 (binary)". The 2-bit scaling factor is determined to be "11 (binary)", which also indicates that the comparison result encodes the intensity value rather than the quantized difference.

[0231] It should be noted that Figure 3-10 The numerical values ​​shown in and described in Examples 1 and 2 above are simplified examples and do not involve floating-point formats. When the division result, previous division result, and comparison result are in floating-point format, they can be processed using the steps described in Examples 1 and 2 in combination with the steps described in Examples 3 and 4.

[0232] Reference numerals

[0233] 10-pixel array

[0234] 3 parameter memories, and

[0235] 4 readout processors, and

[0236] 41 Processing Block

[0237] 51 Bias Generator

[0238] 52 ADC controller

[0239] 53-line address encoder

[0240] 54 column address encoder

[0241] 55 Readout Controller

[0242] 56 Memory Configurator

[0243] 61 External Computer

[0244] 62 External non-volatile memory

[0245] 7 Threshold Memory

[0246] 20 search areas

[0247] 21 pixel blocks

[0248] 22 candidate blocks

[0249] 23 Best Matching Blocks

Claims

1. An event sensor, characterized in that: include: A pixel array (10) composed of pixels and configured to generate a data stream of event data in response to light incident on the pixel array (10); wherein the pixels of the pixel array (10) are divided into pixel blocks, each pixel block including two or more pixels of the pixel array (10), and the event sensor comprises: - for each pixel in the pixel array (10), there is at least one photodetector (1), the photodetector (1) being configured to: generate a detector signal in response to light incident on the pixel; - for each pixel or a group of pixels, there is a signal converter (2) connected to the photodetector (1), the signal converter (2) being configured to: repeatedly generate and store sampled values ​​based on the detector signal sampled at sampling intervals; - a readout processor (4), said readout processor (4) being connected to said signal converter (2) and configured to: perform the following steps for each pixel block: a) deriving, for each pixel in the pixel block, a cumulative pixel value based on one or more of the sampled values, wherein the cumulative pixel value corresponds to the accumulation of the detector signal within a number of sampling intervals; b) deriving a cumulative block value based on the cumulative pixel values ​​of the pixels in the pixel block; c) Under the condition that the cumulative block value exceeds the cumulative block value threshold and / or the interval count exceeds the count threshold, perform the following steps d), f1), f2), f3), and f4): d) determining a cumulative block value threshold and / or a count threshold for subsequent accumulation; f1) defining a search area, where the number of pixels included in the search area is equal to or greater than the number of pixels in the pixel block and includes the pixel block, so that the search area contains one or more candidate blocks, wherein each candidate block is geometrically identical to the pixel block, so that the candidate block can be geometrically overlapped with the pixel block only by translation, and the pixel block itself is also a candidate block; f2) performing an evaluation procedure for each candidate block with reference to the pixel block, the evaluation procedure comprising the following steps: i) for each pixel in the candidate block, reading a previous accumulated pixel value and a previous interval count; ii) determining, for each pixel in the pixel block and the corresponding pixel in the candidate block, at least one contrast threshold and / or at least one difference threshold based on the accumulated pixel value of the pixel, the previous accumulated pixel value of the corresponding pixel, the interval count of the pixel, and / or the previous interval count of the corresponding pixel: iii) calculating, for each pixel in the pixel block and the corresponding pixel in the candidate block, one, two, or three of the following three values: a spatiotemporal contrast, a spatiotemporal difference, and a transmission cost; wherein the transmission cost is calculated based on an amount of data required to encode the following comparison results: a comparison result of comparing the spatiotemporal contrast with the contrast threshold and / or a comparison result of comparing the spatiotemporal difference with the difference threshold; and iv) calculating a cost function result of the candidate block based on the spatiotemporal contrast, the spatiotemporal difference, the transmission cost, and / or the displacement between the pixel block and the candidate block; f3) based on the cost function result, finding the best matching block from the candidate blocks, so that the best matching block is the candidate block associated with the minimum cost function result; and f4) generating an event in the event data based on the displacement between the best matching block and the pixel block and / or the comparison result associated with the best matching block.

2. The event sensor according to claim 1, wherein: The readout processor is configured to calculate the cost function result based on: - the sum / average of the absolute value / square value of the spatiotemporal contrast / spatiotemporal difference between each pixel in the pixel block and the corresponding pixel in the candidate block; - a sum / average of transmission costs associated with each pixel in the pixel block and the corresponding pixel in the candidate block; - a weighted sum / average of the absolute value / square value of the spatiotemporal contrast / spatial difference between each pixel in the pixel block and the corresponding pixel in the candidate block, wherein the absolute value / square value of the spatiotemporal contrast / spatial difference is weighted based on the corresponding transmission cost associated with the pixel and the corresponding pixel; and / or - Whether the displacement between the pixel block and the candidate block is zero.

3. The event sensor according to claim 1 or 2, characterized in that The readout processor (4) is configured to terminate the evaluation procedure early when a perfect matching block is found, wherein the cost function result associated with the perfect matching block is lower than a preset threshold.

4. An event sensor according to any preceding claim, characterized in that The readout processor (4) is configured to generate the event in the event data if the displacement between the best matching block and the pixel block is not zero, or the comparison results associated with the best matching block are not all zero.

5. An event sensor according to any preceding claim, characterized in that The readout processor is configured to: based on the event, overwrite the previous accumulated pixel value and the previous interval count; specifically, for each pixel in the pixel block: i) Regardless of whether the displacement of the pixel block is zero, if the comparison result associated with the pixel is not zero, then: - overwriting the previous accumulated pixel value of the pixel based on the previous accumulated pixel value of the corresponding pixel in the best matching block and the associated comparison result; and - based on the previous interval count of the corresponding pixel in the best matching block and the associated comparison result; overwriting the previous interval count of the pixel; ii) if the displacement of the pixel block is non-zero, and if the comparison result associated with the pixel is zero, then: - overwriting the previous accumulated pixel value of the pixel with the previous accumulated pixel value of the corresponding pixel in the best matching block, and - overwriting the previous interval count of the pixel with the previous interval count of the corresponding pixel in the best matching block; iii) If the displacement of the pixel block is zero, and if the comparison result associated with the pixel is zero, then not overwriting the previous accumulated pixel value and the previous interval count of the pixel.

6. An event sensor, characterized in that: include: A pixel array (10) composed of pixels and configured to generate a data stream of event data in response to light incident on the pixel array (10); wherein the pixels of the pixel array (10) are divided into pixel blocks, each pixel block including two or more pixels of the pixel array (10), and the event sensor comprises: - for each pixel in the pixel array (10), there is at least one photodetector (1), the photodetector (1) being configured to: generate a detector signal in response to light incident on the pixel; - for each pixel or a group of pixels, there is a signal converter (2) connected to the photodetector (1), the signal converter (2) being configured to: repeatedly generate and store sampled values ​​based on the detector signal sampled at sampling intervals; and - a readout processor (4), said readout processor (4) being connected to said signal converter (2) and configured to: perform the following steps for each pixel block: a) deriving, for each pixel in the pixel block, a cumulative pixel value based on one or more of the sampled values, wherein the cumulative pixel value corresponds to the accumulation of the detector signal within a number of sampling intervals; b) deriving a cumulative block value based on the cumulative pixel values ​​of the pixels in the pixel block; c) Under the condition that the cumulative block value exceeds the cumulative block value threshold and / or the interval count exceeds the count threshold, perform the following steps d), g1), g2), g3), and g4): d) determining a cumulative block value threshold and / or a count threshold for subsequent accumulation; g1) for each pixel in the pixel block, reading a previous accumulated pixel value and a previous interval count; g2) determining, for each pixel in the pixel block, at least one contrast threshold, at least one difference threshold, and / or at least one intensity threshold based on the cumulative pixel value, the previous cumulative pixel value, the interval count, and / or the previous interval count of the pixel; g3) calculating, for each pixel in the pixel block, a spatiotemporal contrast, a spatiotemporal difference, an intensity value, and / or a transmission cost; and g4) generating an event in the event data based on a comparison result of comparing the spatiotemporal contrast of each pixel in the pixel block with the contrast threshold, a comparison result of comparing the spatiotemporal difference value of each pixel in the pixel block with the difference threshold, and / or a comparison result of comparing the intensity value of each pixel in the pixel block with the intensity value threshold.

7. The event sensor according to claim 6, characterized in that The readout processor is configured to generate the event in the event data if the comparison result of the pixel block is not all zero.

8. The event sensor according to claim 6 or 7, characterized in that The readout processor is configured to: upon generation of the event, for each pixel in the pixel block, overwrite the previous accumulated pixel value and the previous interval count based on the event, i.e., based on the comparison result of the pixel block; Specifically: i) If the comparison result associated with the pixel is non-zero, then: - overwriting the previous accumulated pixel value of the pixel based on the previous accumulated pixel value itself and the associated comparison result; and - overwriting the previous interval count of the pixel based on the previous interval count itself and the associated comparison result; ii) If the comparison result associated with the pixel is zero, the previous accumulated pixel value and the previous interval count of the pixel are not overwritten.

9. An event sensor according to any preceding claim, characterized in that The readout processor (4) is configured to: - for each pixel block, finding the majority comparison result from the comparison results of the pixels in the pixel block; - for each pixel block, finding an abnormal comparison result that is different from the majority comparison result; and - generating the event in the event data based on the majority comparison result and / or the abnormal comparison result.

10. The event sensor according to claim 9, characterized in that The readout processor (4) is configured to find the mode comparison result based on: - the frequency of occurrence of comparison results with the same value; and / or - a weighted frequency of occurrence of comparison results having the same value, wherein the weighted frequency of occurrence is weighted based on a corresponding transmission cost required to encode the comparison results having the same value.

11. An event sensor according to any preceding claim, characterized in that The readout processor (4) is configured to designate the comparison result associated with the spatiotemporal contrast / spatiotemporal difference value as: - zero, if the spatiotemporal contrast does not exceed the contrast threshold and / or the spatiotemporal difference does not exceed the difference threshold; - contains a portion that is linearly related to or equal to the spatiotemporal contrast and / or the spatiotemporal difference, if the spatiotemporal contrast exceeds the contrast threshold and / or the spatiotemporal difference exceeds the difference threshold, and if the spatiotemporal contrast is within a contrast range / limit and / or the spatiotemporal difference is within a difference range / limit; - comprising a portion that is linearly related to or equal to the division result of the pixels (in the pixel block) if the spatiotemporal contrast exceeds the contrast range / limit and / or the spatiotemporal difference exceeds the difference range / limit.

12. An event sensor according to any preceding claim, characterized in that The readout processor (4) is configured to quantize a portion of the comparison result that is linearly related to or equal to the spatiotemporal contrast and / or the spatiotemporal difference, and encode the portion with a smaller number of bits or a smaller data packet than a portion of the comparison result that is linearly related to or equal to a division result of the pixel.

13. An event sensor according to any preceding claim, characterized in that The readout processor (4) is configured to: when the comparison result contains a portion that is linearly related to or equal to the spatiotemporal difference value, quantize the portion of the comparison result with a step size that is approximately proportional to a previous division result of the pixel or a previous division result of a corresponding pixel in the best matching block.

14. An event sensor according to any preceding claim, characterized in that The readout processor (4) is configured to encode the portion of the division result and / or the comparison result in a floating point format when the comparison result contains a portion that is linearly related to or equal to the division result of the pixel.

15. An event sensor according to any preceding claim, characterized in that The readout processor (4) is configured to: determine the counting threshold as a variable power of a constant base; so that the significand of the division result in a floating point format is based on the accumulated pixel value, and the exponent of the division result is based on the opposite of the variable power.

16. A method of generating a data stream of event data in response to light incident on a pixel array (10) consisting of pixels, characterized in that The pixels of the pixel array (10) are divided into pixel blocks, each pixel block including two or more pixels of the pixel array (10); The method comprises the following steps, performed for each pixel or a group of pixels: generating a detector signal in response to light incident on the pixel; repeatedly generating and storing sampling values ​​based on the detector signal sampled at sampling intervals; a) deriving, for each pixel in the pixel block, a cumulative pixel value based on one or more of the sampled values, wherein the cumulative pixel value corresponds to the accumulation of the detector signal within a number of sampling intervals; b) deriving a cumulative block value based on the cumulative pixel values ​​of the pixels in the pixel block; c) Under the condition that the cumulative block value exceeds the cumulative block value threshold and / or the interval count exceeds the count threshold, perform the following steps d), f1), f2), f3), and f4): d) determining a cumulative block value threshold and / or a count threshold for subsequent accumulation; f1) defining a search area, where the number of pixels included in the search area is equal to or greater than the number of pixels in the pixel block and includes the pixel block, so that the search area contains one or more candidate blocks, wherein each candidate block is geometrically identical to the pixel block, so that the candidate block can be geometrically overlapped with the pixel block only by translation, and the pixel block itself is also a candidate block; f2) performing an evaluation procedure for each candidate block with reference to the pixel block, the evaluation procedure comprising the following steps: i) for each pixel in the candidate block, reading a previous accumulated pixel value and a previous interval count; ii) determining, for each pixel in the pixel block and the corresponding pixel in the candidate block, at least one contrast threshold and / or at least one difference threshold based on the accumulated pixel value of the pixel, the previous accumulated pixel value of the corresponding pixel, the interval count of the pixel, and / or the previous interval count of the corresponding pixel: iii) calculating, for each pixel in the pixel block and the corresponding pixel in the candidate block, one, two, or three of the following three values: a spatiotemporal contrast, a spatiotemporal difference, and a transmission cost; wherein the transmission cost is calculated based on an amount of data required to encode the following comparison results: a comparison result of comparing the spatiotemporal contrast with the contrast threshold and / or a comparison result of comparing the spatiotemporal difference with the difference threshold; and iv) calculating a cost function result of the candidate block based on the spatiotemporal contrast, the spatiotemporal difference, the transmission cost, and / or the displacement between the pixel block and the candidate block; f3) based on the cost function result, finding the best matching block from the candidate blocks, so that the best matching block is the candidate block associated with the minimum cost function result; and f4) generating an event in the event data based on the displacement between the best matching block and the pixel block and / or the comparison result associated with the best matching block.

17. A method of generating a data stream of event data in response to light incident on a pixel array (10) consisting of pixels, characterized in that The pixels of the pixel array (10) are divided into pixel blocks, each pixel block includes two or more pixels of the pixel array (10), and the method includes the following steps performed for each pixel or a group of pixels: generating a detector signal in response to light incident on the pixel; repeatedly generating and storing sampling values ​​based on the detector signal sampled at sampling intervals; a) deriving, for each pixel in the pixel block, a cumulative pixel value based on one or more of the sampled values, wherein the cumulative pixel value corresponds to the accumulation of the detector signal within a number of sampling intervals; b) deriving a cumulative block value based on the cumulative pixel values ​​of the pixels in the pixel block; c) Under the condition that the cumulative block value exceeds the cumulative block value threshold and / or the interval count exceeds the count threshold, perform the following steps d), g1), g2), g3), and g4): d) determining a cumulative block value threshold and / or a count threshold for subsequent accumulation; g1) for each pixel in the pixel block, reading a previous accumulated pixel value and a previous interval count; g2) determining, for each pixel in the pixel block, at least one contrast threshold, at least one difference threshold, and / or at least one intensity threshold based on the cumulative pixel value, the previous cumulative pixel value, the interval count, and / or the previous interval count of the pixel; g3) calculating a spatiotemporal contrast, a spatiotemporal difference, an intensity value, and / or a transmission cost for each pixel in the pixel block; and g4) generating an event in the event data based on a comparison result of comparing the spatiotemporal contrast of each pixel in the pixel block with the contrast threshold, a comparison result of comparing the spatiotemporal difference value of each pixel in the pixel block with the difference threshold, and / or a comparison result of comparing the intensity value of each pixel in the pixel block with the intensity value threshold.

Images