Processing of data from event-based sensors

Abstract

Processing of data from an event-based sensor. Each event includes a pixel position and a timestamp in an array of pixels of an event-based sensor. A grid of cells is defined (201) for the array of pixels, each cell comprising a plurality of pixels. In trajectory determination, incoming events are mapped to a grid of cells during a first time period (202-203), and summary data having representative locations of pixel locations falling within the respective cells during the first time period is generated (204). In-cell evaluation of the summary data generated for the second time period is performed (206) to generate cell trajectories of representative locations within the respective cells. An inter-cell evaluation (208) of a plurality of cell trajectories generated for the plurality of cells over a second time period is performed to generate a grid trajectory.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to data processing related to imaging systems, in particular imaging systems comprising event-based sensors arranged to receive photons reflected or scattered by regions on an object when illuminated by a scanning beam. BACKGROUND

[0002] Image sensors in conventional digital cameras capture two-dimensional (2D) digital images representing incident photons on individual pixels of a 2D pixel array over an exposure period. The number of pixels in the pixel array can be large, resulting in a high computational load for processing the digital images.

[0003] Three-dimensional (3D) image-based localization conventionally uses multiple digital cameras arranged to view a scene from different angles. The digital cameras are operated synchronously to capture respective time sequences of digital images. A time sequence of 3D representations of any objects located in the scene is generated by triangulation processing concurrent digital images from different digital cameras. At least in case of limited computational resources, this image-by-image processing is processing-intensive and difficult to perform in real-time.

[0004] To alleviate these problems, US 10261183 proposes a different type of system for 3D localization. The system comprises an emitter configured to scan a light beam across a scene, and multiple digital cameras arranged to view the scene from different angles. When the light beam hits a voxel on an object in the scene, photons are reflected or scattered from the object. Some of these photons hit a pixel array of a respective digital camera and produce a local signal increase. Based on the location of the local signal increase on each pixel array, the 3D position of the illuminated voxel can be determined by triangulation. When the light beam scans the object, the resulting 3D positions can be compiled into a 3D representation of the object. This technique significantly reduces the amount of image data that needs to be processed for 3D localization, in particular if the digital cameras are so-called event cameras. In case of event cameras, each pixel operates independently and asynchronously to report a luminance change when a luminance change occurs, otherwise it remains silent. Each activation of a pixel forms an event. An event camera outputs a continuous stream of such events, where each event can be represented by an identifier of the activated pixel and a timestamp of its activation.

[0005] Even if the use of event cameras will reduce the amount of data to be processed for 3D localization compared to the processing of 2D images, there is still a need to improve the processing efficiency, e.g. if the 3D localization is to be performed on a resource-constrained system and / or if the 3D localization is time-constrained. The events from the event sensor define a point cloud that moves over time when one or more light beams are scanned in the space observed by the event camera. The moving point cloud can be processed into one or more trajectories for further analysis. However, trajectory estimation is difficult to implement in a processing-efficient way as it involves identifying connections between points that are both spatial and temporal. Thus, the number of processing operations can scale significantly with the number of data points in the point cloud and the temporal resolution at which the point cloud moves.

[0006] The above-mentioned problems apply equally if the event data is to be processed to determine another object property than the 3D position of the object, e.g. surface structure, size, contour, etc. SUMMARY

[0007] It is an object to at least partly overcome one or more limitations of the prior art.

[0008] It is also an object to provide an efficient technique of processing event data from an event-based sensor in a scanning illumination system into one or more trajectories.

[0009] One or more of these objects, as well as further objects that can appear from the following description, are at least partly achieved by a computer-implemented method of processing a stream of events from an event-based sensor, a computer-readable medium, a processing device and a system.

[0010] Aspects of the present disclosure enable efficient trajectory determination based on a stream of events from an event-based sensor. The aspects are based on the insight that it is beneficial to reduce the amount of data early in the processing and to determine trajectories in a step-wise manner, starting with smaller trajectories that are then merged into larger trajectories. This is achieved by defining a computational grid of pixels on the event-based sensor as a grid of cells and by performing a data reduction processing of event data falling within respective cells into summary data relevant for the trajectory determination, including representative positions of events. The data reduction processing can be done quickly and efficiently for the incoming stream of events and will greatly facilitate the downstream processing into trajectories by limiting the amount of data. The data reduction processing can also reduce the impact of noise and other disturbances represented in the stream of events. Furthermore, by spatially dividing into cells and mapping events to cells, the locality of the position data in the stream of events is inherently preserved in the resulting summary data. The preserved locality is a prerequisite for the downstream processing of the summary data into trajectory data.

[0011] The subsequent processing for trajectory determination is divided into intra-cell evaluation for generating cell trajectories based on representative locations from data reduction processing, and inter-cell evaluation for generating grid trajectories based on the cell trajectories. Through inter-cell evaluation, the potentially complex task of trajectory determination is limited to individual spatially constrained sub-regions on the sensor. This significantly reduces the complexity of trajectory determination. Given the intra-cell evaluation, the inter-cell evaluation is deterministic and processing-efficient because it involves processing a finite number of cell trajectories to form a grid trajectory. Typically, aspects of this disclosure are used to limit the number of comparisons required to process a moving point cloud represented by a stream of events into a trajectory.

[0012] Furthermore, intra-cell evaluation is inherently compatible with parallel processing because summary data from different cells can be evaluated in parallel.

[0013] The aspects of this disclosure also provide technical advantages for facilitating trajectory determination when multiple scanning beams move simultaneously within a scene observed by an event-based sensor. If the scanning beams are always spatially separated, a cell grid can be defined such that at any given time, a pixel within any cell of the grid receives only light scattered or reflected by one of the scanning beams. Therefore, cell trajectories can inherently be determined individually for each scanning beam without prior knowledge of the number of scanning beams.

[0014] Other objects and aspects, as well as features, implementation methods and technical effects, will become apparent from the following detailed description, the appended claims and the accompanying drawings. Attached Figure Description

[0015] FIG. 1A It is a block diagram of a system for determining the characteristics of an object using a scanning illumination system and an event-based sensor. FIG. 1B This is a block diagram of an example event stream generated by an event-based sensor.

[0016] FIG. 2 Is FIG. 1A The flowchart shows an example method executed in the system.

[0017] FIG. 3 It is a timing diagram of the light pulse sequence generated by the irradiation system.

[0018] FIG. 4A It is a planar diagram of a pixel array logically divided into unit grids, and FIG. 4B It is a planar diagram of a cell grid with overlapping trajectories of incident light originating from two scanning beams.

[0019] FIG. 5A to FIG. 5E This is a flowchart of a method and process for processing a stream of events from an event-based sensor, based on an example.

[0020] FIG. 6A The active pixels in a cell for three consecutive summary periods are shown, and FIG. 6B Depicting the example in FIG. 6A Summary data generated in the process.

[0021] FIG. 7 It is a plan view of representative locations in the summary data defined for the unit.

[0022] FIG. 8A An example process for stitching cell trajectories between two adjacent cells into a mesh trajectory is shown. FIG. 8B The process for stitching cell trajectories into a grid trajectory is shown. FIG. 8C It shows in FIG. 8B The effect of adjusting the start and end points of the unit trajectory at the boundary between adjacent units. FIG. 8D The time series of light pulses at the edge of the object is shown, and FIG. 8E It shows when a light pulse strikes FIG. 8D The edges of objects in the array are a set of active pixels on the pixel array.

[0023] FIG. 9 The process for stitching together grid trajectories generated for consecutive frames is shown.

[0024] FIG. 10 This is a block diagram of a machine that can implement the methods disclosed in this paper. Detailed Implementation

[0025] Embodiments will now be described more fully below with reference to the accompanying drawings, which illustrate some, but not all, of the embodiments. In fact, the subject matter of this disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided to enable this disclosure to meet applicable legal requirements. The same numerals consistently refer to the same elements.

[0026] Furthermore, it should be understood that, where possible, any advantages, features, functions, apparatus and / or operational aspects of any implementation described and / or contemplated herein may be included in any other implementation described and / or contemplated herein, and / or vice versa. Furthermore, where possible, unless expressly stated otherwise, any term expressed herein in the singular is intended to also include the plural form and / or vice versa. As used herein, “at least one” shall mean “one or more”, and these phrases are intended to be interchangeable. Thus, the terms “a” and / or “an” shall mean “at least one” or “one or more”, even though the phrases “one or more” or “at least one” are also used herein. As used herein, unless the context otherwise requires due to explicit language or necessary meaning, the word “comprising” or variations such as “including” or “consisting of” are used in an inclusive sense, i.e., to specify the presence of the stated feature, but not to exclude the presence or addition of additional features in various implementations.

[0027] Furthermore, it should be understood that although the terms first, second, etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the scope of this disclosure, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. As used herein, the terms “a plurality” and “many” are intended to imply the provision of two or more elements, while the term “group” is intended to imply the provision of at least one element. The term “and / or” includes any and all combinations of one or more of the associated listed elements.

[0028] For the sake of brevity and / or clarity, well-known functions or constructions may not be described in detail. Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0029] As used herein, an "event-based sensor" or "event sensor" refers to a sensor that responds to localized changes in brightness. The sensor includes photosensitive elements ("pixels") that operate independently and asynchronously by reporting brightness changes when they occur, otherwise remaining silent. Thus, the event sensor outputs a stream of asynchronous events triggered by changes in scene illumination. Pixels can be arranged as an array defining a photosensitive surface. The photosensitive surface can be one-dimensional (1D) or two-dimensional (2D). Pixels can be based on any suitable technology, including but not limited to active pixel sensors (APS), charge-coupled devices (CCDs), single-photon avalanche detectors (SPADs), complementary metal-oxide-semiconductors (CMOS), silicon photomultipliers (SiPMs), photovoltaic cells, phototransistors, etc.

[0030] As used herein, a "beam scanner" refers to a device capable of scanning or sweeping a beam of light in a one-dimensional, two-dimensional, or three-dimensional pattern. The beam may or may not be collimated. A beam scanner includes scanning devices configured to redirect the beam. The scanning devices can be based on any suitable technology and may include one or more mirrors, prisms, optical lenses, acousto-optic deflectors, electro-optic deflectors, etc. In another alternative, the scanning devices may implement a scanning phase array. A beam scanner may also include a light source, such as a laser, a light-emitting diode (LED), a light bulb, etc. The light source can provide a continuous or pulsed beam of light at a predefined frequency or frequency range.

[0031] As used herein, “light” refers to electromagnetic radiation in the portion of the electromagnetic spectrum that extends from about 10 nm to about 14 μm, including ultraviolet, visible and infrared radiation.

[0032] The implementation relates to an efficient and rapid technique for processing data streams of events generated by an event-based sensor (hereinafter referred to as an "event sensor" or "event camera"), wherein the events originate from one or more beams of light scanned across a scene viewed by the event sensor. Specifically, the technique aims to generate trajectory data, where events are arranged into one or more trajectories. Each such trajectory represents how the beam of light moves across objects in the scene. For context, reference will be made to... FIG. 1A The device or system shown is used for object characteristic detection to describe the generation of trajectory data.

[0033] FIG. 1AThis is a schematic diagram of a system including three event cameras 30, 40, and 50 and a beam scanner 20. Each event camera 30, 40, and 50 includes a corresponding sensor array or pixel array 31, 41, and 51 defining a two-dimensional (2D) array of photosensitive elements (“pixels”). The event cameras 30, 40, and 50 have corresponding fields of view (FOVs) facing the scene. The FOVs at least partially overlap. Although FIG. 1A Not shown, but each event camera 30, 40, 50 may include one or more optical components defining the field of view (FOV), such as optical lenses. The event camera may also include filters.

[0034] Beam scanner 20 is configured to scan or sweep beam 21 (“scan beam”) across a scene in a predefined random or pseudo-random pattern. The movement path of scan beam 21 can be specific to beam scanner 20, or can be configurable and therefore adjustable. For example, beam scanner 20 can be operated to scan beam 21 independently along two orthogonal axes. It is also conceivable that sweeps along the two axes are interdependent. The period of back-and-forth sweeping (“sweep period,” corresponding to “sweep frequency”) can differ between the axes. In a non-limiting example, the sweep frequency along the horizontal axis can be in the range of 100-300 Hz, and the sweep frequency along the vertical axis can be in the range of 5-20 Hz. In some embodiments, the sweep speed along the respective axis varies with a sine curve. Other functional dependencies of the sweep speed are conceivable, such as trigonometric functions or more square-like functions. In some embodiments, scan beam 21 forms a Lissajous curve.

[0035] In the example shown, the scene includes an object 1 illuminated by a scanning beam 21, which forms a moving spot on the object 1. The moving point thus sequentially illuminates region 2 on the object 1. FIG. 1A These regions 2, represented as dark spots, are also referred to as "scattering regions" (SA) in this document. Therefore, scattering region 2 refers to a sampled surface element of the object. For each SA 2, one or more (e.g., two or three) cameras detect photons reflected or otherwise scattered by that SA. In the example shown, reflected light from the SA is detected by one or more pixels at position (x1, y1) on sensor array 31, position (x2, y2) on sensor array 41, and position (x3, y3) on sensor array 51. Typically, in event cameras, an event is generated only when a pixel is "activated" by receiving light energy exceeding a predefined limit. FIG. 1AIt is understood that reflected light from SA 2 is captured or detected by one or more of cameras 30, 40, and 50, and cameras 30, 40, and 50 output corresponding event streams including identifiers or addresses of the activated pixels. In some implementations, each camera autonomously and asynchronously outputs an event whenever a pixel is detected to be illuminated by a sufficient number of photons.

[0036] FIG. 1B The diagram schematically illustrates a stream of events (“event stream”) that can be output by the event camera 30. The event stream is specified by ES1, and each event is specified by E. In the example presented herein, it is assumed that each event E includes the pixel position on the pixel array (as specified by...). FIG. 1B The sensor coordinates (x, y) and the timestamp of the activated pixel (represented by...) FIG. 1B The 't' in the figure represents the pixel location. Pixel positions are typically given as the index or address of the activated pixel, such as sensor coordinates (as shown in the figure). The timestamp specifies the activation time of the pixel and is set based on the clock output in the event camera. Timestamps can be given in any time unit. FIG. 1B As shown, event E may include additional data a, such as the illumination level (“light intensity”) received by the activated pixel.

[0037] Cameras 30, 40, and 50 are connected, via wired or wireless connection, to a processing system 60 (such as a computer device) configured to receive event streams from the cameras. Conceptually, the processing system 60 includes a first module or subsystem 61 and a second module or subsystem 62. The first module or subsystem 61 is configured to generate trajectory data TD from the event stream, and the second module or subsystem 62 is configured to receive the TD and determine object characteristic data OD based on the TD. The trajectory data TD represents the 2D trajectory of activated pixels on a pixel array, where each 2D trajectory corresponds to the movement of an object across the scene by a scan beam. In other words, the 2D trajectory corresponds to the movement trajectory of SA 2 observed by the event cameras.

[0038] The processing system 60 may include a first module 61 for each event camera 30, 40, 50. Alternatively, the first module 61 may be configured to jointly process events from two or more event cameras.

[0039] The second module 62 can be configured in many different ways, depending on the characteristics of the object to be determined. The structure of the second module 62 is beyond the scope of this disclosure and will not be described in detail thereafter. In a non-limiting example, the second module 62 is configured to computationally combine trajectory data from two or more event cameras into a 3D trajectory in three-dimensional (3D) space, such as in the scene coordinate system associated with the system (see [link to documentation]). FIG. 1A(10 in the original text). The second module 62 can, for example, be configured to calculate a 3D trajectory based on calibration data representing the relative positioning of the event camera, using any known and conventional triangulation techniques. Optionally, the second module 62 can be configured to further process the 3D trajectory to determine refinement data, such as the object's outline, surface structure, 3D shape, orientation, speed, direction of movement, etc. Furthermore, the second module 62 can include a perception system configured to classify objects into one or more predefined object classes. Such a perception system may use artificial intelligence and include one or more trained neural networks. Object characteristic data OD can include any of the 3D trajectory, refinement data of the corresponding object, or classification of the corresponding object.

[0040] It is also conceivable that the second module 62 operates without calculating 3D trajectories. For example, the second module 62 could be configured to determine whether an object exists in the scene, which is also an object characteristic. In its simplest form, this can be achieved by analyzing trajectory data from one or more event cameras to detect 2D trajectories indicating objects, for example, by having a predefined range in one, two, or three dimensions.

[0041] FIG. 1A The system can be stationary or mobile. In some implementations, the system is mounted on a moving land, air, or water vehicle and processes the resulting 3D trajectory to observe a complex, dynamic 3D scene illuminated by a scanning beam. For example, such a 3D scene can be mapped in detail based on the 3D trajectory at a high detection speed. This type of information can be useful, for example, for systems used in autonomous driving and driver assistance.

[0042] FIG. 1A The system described is given only as a non-limiting example. A greater number of event cameras can be used, for example, to improve accuracy and / or introduce redundancy. Alternatively, a smaller number of event cameras can be used, for example, to improve computational speed and / or reduce cost and complexity. Similarly, any number of beam scanners 20 can be used. Therefore, more than one beam 21 can be swept across a scene. It is also conceivable that beam scanners 20 are configured to sweep more than one beam across a scene. In some implementations, the beams are formed by lasers and are therefore coherent and highly collimated. Multiple beams can be generated (whether by a single beam scanner or different beam scanners) such that the spatial separation of the beams at a predefined distance from the beam scanner(s) always exceeds the nominal size of the human eye. This allows for maximizing the energy of the respective beams, taking into account the eye safety requirements of lasers.

[0043] FIG. 2 It can be made by FIG. 1AThe flowchart illustrates an example method 100 executed by the system. In step 101, at least one beam scanner 20 is operated to scan one or more beams 21 within the scene. Step 101 may be performed continuously during the operation of method 100. In step 102, which may or may not be performed in real time, the processing system 60 receives event streams from the corresponding event cameras 30, 40, 50.

[0044] Those skilled in the art will understand that sensor arrays 31, 41, and 51 can also receive and detect light not originating from the scanning beam 21, such as ambient light reflected from object 1. Ambient light may originate from sunlight or lighting in the scene. Furthermore, sensor arrays 31, 41, and 51 can receive and detect light secondary reflections originating from the scanning beam. Additionally, electronic noise on the sensor arrays may also lead to falsely activated pixels. Therefore, method 100 may include an optional step 103, in which processing means 60 operates a filtering process on the corresponding event stream to remove events that may originate from stray light or noise, while retaining other events. Step 103 generates a filtered event stream of the corresponding event stream. The filtering process can be performed in many different ways. Efficient filtering techniques are disclosed in WO2023 / 046392, which are incorporated herein by reference.

[0045] In step 104, the processing device 60 generates the aforementioned trajectory data TD. Step 104 can be generated by... FIG. 1A The first module 61 is executed. Step 104 involves a trajectory determination process, in which a 2D trajectory on a corresponding pixel array is determined based on a (filtered) event stream from the corresponding event camera. As used herein, a 2D trajectory is a sequence of temporally and spatially coherent locations in the coordinate system of the pixel array. In many applications, real-time or near-real-time execution of the processing performed by method 100 may be required or at least expected. Here, step 104 may be particularly challenging. Embodiments of this disclosure are designed to minimize the latency introduced by the trajectory determination process of step 104. Embodiments can also be used to make step 104 efficient in terms of memory requirements and processing load, thereby reducing the cost of the hardware used to implement trajectory determination and limiting power consumption. Reference is made below. FIG. 5A to FIG. 9 Various implementations of step 104 are described in detail.

[0046] In step 105, the processing device 60 processes the trajectory data TD from step 104 to determine one or more characteristics of one or more objects in the scene. The result is provided in the form of the aforementioned object characteristic data OD. Therefore, step 105 can be performed by the second module 62, for example, according to any non-limiting example given above.

[0047] At least some implementations of step 104 rely on the presence of light pulses in the scanning beam 21. FIG. 3This is a timing diagram of the optical pulses 22 generated by the beam scanner. Each optical pulse 22 has a duration P, and the optical pulses 22 are generated periodically with pulse-to-pulse intervals PP. The interval PP is typically much longer than the pulse duration P. In some embodiments, the duration P of the corresponding optical pulse is in the range of 1-100 ns. In some embodiments, the interval PP is in the range of 1-20 μs, corresponding to a pulse rate in the range of 20-1000 kHz. The pulsed light can be generated by a laser source in the beam scanner 20. The laser source can include one or more solid-state laser devices, such as laser diodes, VCSELs (vertical-cavity surface-emitting lasers), DPSSLs (diode-pumped solid-state lasers), silicon lasers, etc. In some embodiments, the laser source is configured to emit IR light, for example, in the range of 700-1000 nm.

[0048] FIG. 4A This is a top-view plan view of an example pixel array 31 comprising a grid of pixels 33. Each pixel 33 responds to an incident photon. Each pixel has a known position (“pixel position”) in a local coordinate system 10' and is represented by a unique identifier, index, or address, given here as a pair of coordinates (x, y) in the local coordinate system 10'. In some implementations, each pixel 32 stores a reference brightness level and continuously compares it with the current brightness level detected by the pixel. If the brightness difference exceeds a preset threshold, pixel 32 resets its reference level and generates an event. The event may also include an instantaneous measurement of the polarity (increase or decrease) of the brightness change and / or the current illumination level (“intensity”). The operation and structure of an event camera are beyond the scope of this disclosure. For a detailed description of different types of event cameras and their operation, refer to the article “Event-based Vision: A Survey”, published by Gallego et al. as arXiv:1904.08405v3[cs.CV], August 2020, the entire contents of which are incorporated herein by reference.

[0049] FIG. 4A A grid of cells 32 is also shown, defined for the pixel array 31 and used in the calculation of step 104. Each cell 32 includes multiple pixels 33 and has a predefined location on the pixel array 31. Cells 32 are not physically defined on the pixel array 31, but are computational constructs or partitions used in the calculation of step 104. During the calculation, each cell 32 can be assigned an identifier that uniquely identifies the cell 32 and its included pixels 33.

[0050] exist FIG. 4AIn the example, cells 32 are arranged in a linear grid that forms a tessellation or tiling of cells 32 on the pixel array 31. Therefore, cells 32 are arranged without overlap and without gaps. For computational efficiency, cells 32 can have equal shapes and sizes, as shown, and can be rectangular, such as the squares shown. However, it is conceivable that cells may differ in shape and / or size and be arranged in another form of regular grid, or even in an irregular grid, with or without overlap.

[0051] When the cross-scene sweep scanning beam 21 ( FIG. 1A When ), pixel 32 is activated by the reflected light received by pixel array 31. FIG. 4B In the image, the path of reflected light from scanning beam 21 is superimposed on array 31, as shown by dashed arrow 21'. The shaded areas indicate units 32 containing pixels activated when objects sweep across the scanning beam in the scene. It can be seen that each point in arrow 21' represents reflected light from a single light pulse (see [link to image]). FIG. 3 22). The reflected light of the light pulse (“reflected pulse”) has a pulse size on pixel array 31, which depends on the spot 2 on object 1 ( FIG. 1A The size of the spot on the pixel array 31. Therefore, if the scanning beam 21 is divergent, the pulse size on the pixel array 31 can vary depending on the distance to the object. The pulse size can also vary depending on the surface structure and orientation of the object 1. However, for each system, a nominal pulse size of the reflected pulse on the pixel array 31 can be defined, for example, for the maximum useful distance between the system and the object 1. The nominal pulse size typically corresponds to a number of pixels on the pixel array 31. For computational efficiency, the size of the corresponding cell 32 can be defined as exceeding the nominal pulse size of the reflected pulse. In some embodiments, each cell 32 can have a size that accommodates at least 2 to 5 reflected pulses arranged side by side. FIG. 4B It is recognized that each reflection pulse can be assigned to at least one cell 32 by the location of the pixel activated by the reflection pulse. If the reflection pulse is located near the cell boundary, a portion of the reflection pulse can also activate pixels in one or more adjacent cells.

[0052] As noted, the system can be configured to sweep multiple beams simultaneously across a scene, with these beams separated by a minimum angular distance for eye safety reasons. This is in FIG. 4BAs shown, the path of reflected light from another scanning beam is indicated by dashed arrow 21''. In this configuration, it may be advantageous to have the size of the corresponding cell 32 smaller than the known spatial spacing of reflected pulses from different scanning beams on the pixel array 31. In other words, the cell grid is defined to prevent pixels 33 within the corresponding cell 32 from being activated by reflected light from more than one scanning beam at any given time. This simplifies the determination of the 2D trajectory in the presence of multiple scanning beams.

[0053] In the following text, references will be made to, for example FIG. 4A to FIG. 4B The grid of cell 32 shown is used to describe step 104 of method 100. FIG. 5A The flowchart corresponding to step 104 is shown below. For further illustration, block arrows are added to the right side of the flowchart to indicate the input data used and / or the output data provided by the corresponding step. The corresponding block arrows may correspond to operations accessing memory (see [link to flowchart documentation]). FIG. 10 (1002 in the middle).

[0054] Initially, step 201 is performed to define the cell grid of the pixel array. As described above, the cell grid is a computational construct. Through steps 202-205, the use of cells enables fast and efficient classification and data reduction of the incoming event stream from the pixel array. Data reduction involves generating summary data for each cell. The summary data of a cell represents the incoming event, where the pixel position is within the corresponding first time period. During t1 (hereinafter referred to as the "summary cycle"), data falls within the cell. Steps 202-205 can be performed consecutively to generate a set of summary data for multiple consecutive summary cycles for subsequent processing in steps 206-210.

[0055] In the example shown, during the summary cycle in step 202 During period t1, events are collected for each cell. Step 202 involves mapping the pixel location of the corresponding event to a cell, and assigning the event to a cell when the pixel location falls within the cell. When the summary period expires (step 203), summary data of the events within the corresponding cell during the summary period is output in step 204. FIG. 5A In this context, the summary data for the cell with index i is designated as SDi. The summary data includes at least the representative location of the event within the corresponding cell. It should be recognized that if in... If no events are collected within a specific unit during period t1, the summary data can be empty. If a second time period follows step 204... If t2 has not yet expired, the process returns to step 202. Therefore, steps 202-205 are performed to generate a set or "frame" summary data for each cell during the second time period (referred to herein as the "frame period"). If the frame period... If t2 has expired, then step 205 proceeds to step 206 to process the set of summary data generated during the frame period.

[0056] In efficient computer implementations, summary data is computed as an in-run aggregation of data from incoming events. For example, one or more computed variables can be assigned to each unit to be reset at the start of each summary cycle and configured to accumulate data from incoming events. In other words, computed variables are recursively and in real-time updated as new events arrive. For example, aggregation can involve summation, multiplication, maximum extract, minimum extract, or any combination thereof.

[0057] The determination of the summary data of unit Ci in FIG. 6A Further examples are provided. FIG. 6A In the diagram, cell Ci is shown over three consecutive summary cycles, where the filled squares indicate the pixels activated during the corresponding summary cycle. It should be understood that activated pixels will result in corresponding events being output by the pixel array. For example, in the leftmost illustration, cell Ci contains 10 activated pixels, and thus results in 10 events being collected by step 202 during the corresponding summary cycle. Step 204 will output summary data for each summary cycle, thereby generating... FIG. 6A SDi_t1, SDi_t2, and SDi_t3 in the example.

[0058] In some implementations, the summary cycle t1 is set to include a single reflected pulse. In other words, t1 is set to be greater than the pulse duration P and less than the pulse-to-pulse interval PP. FIG. 3 This allows the summary data from the summary period to represent a single reflected pulse, which maximizes the spatial and temporal resolution of the summary data. However, it is conceivable that the summary period is set to include multiple reflected pulses.

[0059] Back FIG. 6A The activated pixels within a cell can thus represent a pulse reflection, and the summary data can specify the representative location of each pulse reflection within cell Ci. In the following text, the term "representative location" is abbreviated as RP.

[0060] In some implementations, frame period t2 is set to exceed the time required for a reflected pulse to sweep across any individual cell on the pixel array. This allows the reflected pulse to form a complete path across at least one cell during the frame period. It may also be desirable to... t2 is set to be less than the time required to sweep the reflected pulse across the entire pixel array in the x or y direction, i.e., half the sweep period. A longer t2 may be advantageous because a longer t2... t2 reduces the number of inter-unit evaluations (step 208 below), as unit evaluation is expensive in terms of processing. The delay between optical detection and trajectory data output is related to... t2 is proportional. In some implementations, to balance these conflicting requirements, the frame period is set to include at least 50, 100, or 150 profile periods, while being less than half the sweep period.

[0061] In some implementations, regardless of each The number of pulse reflections at t1 determines how RP can be calculated as an approximation of or correspondence to the center point of the pixel position within the cell. For example, RP can be calculated as the average pixel position of the incoming events within the cell. If the pixel position is determined by a first coordinate and a second coordinate (see...), then... FIG. 4A to FIG. 4B Given the x and y coordinates in the local coordinate system, the average can be calculated separately for the first and second coordinates. Therefore, RP can be given by the first and second values ​​in the local coordinate system, where the first value is calculated as the average of the first coordinates of the pixel location, and the second value is calculated as the average of the second coordinates of the pixel location. Any average value can be used, including but not limited to the arithmetic mean, geometric mean, median, root mean square, intermediate range, centroid, etc.

[0062] In some implementations, step 204 is configured to if in If the number of events within the cell during period t1 exceeds a threshold, the summary data of the output cell is then processed. The threshold can be set to reduce the influence of ambient light and noise and / or increase the likelihood that the summary data accurately represents at least one reflected pulse. In a non-limiting example, the threshold is set to 2, 3, 4, or 5.

[0063] As will be further elaborated below, the summary data may include further representative data on the events falling within the cell. FIG. 6B This shows an example of the summary data for unit Ci, SDi. FIG. 6B In this context, RPi specifies the representative location, ti specifies the representative time, and SEi is a range parameter. The representative time ti is determined by... The timestamps of events collected during period t1 are given. For example, ti can be set to the minimum, median, maximum, or average of the timestamps. The range parameter SEi is an estimate of the spatial range of events falling within the cell, specifically the range of pixel locations. If the pixel locations are given in the first and second coordinates (see...), then... FIG. 4A to FIG. 4B If x and y are given, then SEi can be determined separately for the first and second coordinates. Therefore, in FIG. 6AIn the examples, SEi can be defined as representing the horizontal and vertical range of combinations of black pixels within a cell Ci for a given summary period. In one example, SEi can be given by the maximum and minimum pixel positions, or equivalently by the difference (“range”) between the maximum and minimum pixel positions. This range is a measure of statistical deviation. It is conceivable that SEi can be given by another measure of statistical deviation of pixel positions, such as variance, standard deviation, range, interquartile range, median absolute deviation, mean absolute deviation, entropy, etc. In some implementations, SEi includes two or more distinct range parameters for at least one dimension.

[0064] return FIG. 5A Step 206 involves the following: Evaluation of the summary dataset generated for the corresponding cell during t2. Step 206 aims to determine one or more trajectories of the RP within the cell. FIG. 5A In this context, the summary data set determined for cell Ci during the frame period is designated as [SDi]. Step 206 is limited to a single cell and is therefore an "intra-cell evaluation". Step 206 generates one or more trajectories for the corresponding cell. The trajectory comprises the RP that forms a spatially and temporally coherent trajectory within the cell. Such cell-specific trajectories are referred to below as "cell trajectories" or CTs. FIG. 5A In this context, the set of element trajectories determined for element Ci is designated as [CTi]. See below for reference. FIG. 5B The implementation of step 206 is described further.

[0065] The trajectory can be stored in any suitable data structure during the execution of the trajectory determination process 104, such as a one-dimensional array, table, dynamic array, associative array, linked list, etc.

[0066] As used herein, "spatial coherence" means that the RPs are separated from each other by a distance less than the maximum distance. Similarly, "temporal coherence" means that the RPs are separated from each other by a time less than the maximum time difference over their representative time. The maximum distance and maximum time difference depend, for example, on the sweeping speed of the scanning beam, the pulse rate of the light, and the optical system of the event camera. Nominal values ​​for the distance and time difference can be determined analytically or experimentally for the system, and maximum values ​​can be set based on these nominal values.

[0067] Step 206 can be followed by step 207, in which the endpoints of the corresponding unit trajectory are processed for adjustment. See below for reference. FIG. 5D Describe optional step 207.

[0068] In step 208, the cell trajectories generated in step 206 (and optionally adjusted in step 207) are processed to stitch together one or more trajectories (if possible) extending across multiple cells. Step 208 is performed across cells and is therefore an “inter-cell evaluation.” The corresponding trajectories contain representative locations of spatially and temporally coherent trajectories formed within cells and potentially extending across cells. The trajectories from step 208 are hereinafter referred to as “mesh trajectories” or GTs. FIG. 5A In this context, the set of mesh trajectories determined for all elements is designated as [GT]. It should be understood that GT does not need to extend across boundaries between elements, but it can be the same as CT if the CT cannot be merged with another CT in step 208.

[0069] Step 208 can be followed by a further evaluation step 209, referred to as inter-frame evaluation, where the grid trajectory [GT] is processed to be merged with the grid trajectory [GT]p determined for the previous frame period (if possible). See below for reference. FIG. 5E Describe optional step 209.

[0070] In step 210, optionally after merging in step 209, trajectory data TD is output based on the grid trajectory [GT] from step 208. In some embodiments, step 210 simply includes the grid trajectory [GT] in the TD. In some embodiments, such as FIG. 5A As shown by the dashed lines, step 210 includes step 210A, which involves performing a curve fitting algorithm on the corresponding ground truth (GT) to determine a curve that matches the RP with sufficient accuracy. Any conventional curve fitting algorithm can be used. For example, the curve fitting algorithm can be based on least squares. The curve fitting algorithm generates curve definition data for the resulting curve. In step 210B, the curve definition data is provided for output in the trajectory data. By replacing the GT in the curve definition data, the size of the TD can be significantly reduced. Furthermore, curve fitting can produce a TD with sub-pixel precision. The curve fitting algorithm can be configured to fit a predefined function to the RP, such as a polynomial function of a predefined order, a spline function, a trigonometric function, etc. In one example, the predefined function is a third-order polynomial. In a variant, the curve fitting algorithm can test multiple predefined functions of the RP and output curve definition data to obtain the best fit.

[0071] In some implementations, step 210 may include calculating and evaluating the residual between the resulting curve and the RP of the ground truth (GT). If the residual exceeds a threshold, the resulting curve may be discarded instead of being included in the trajectory data. This reduces the risk of errors in determining object characteristics during downstream processing in module 62.

[0072] FIG. 5B It corresponds to FIG. 5AThe flowchart illustrates an example procedure for intra-cell evaluation 206. Procedure 206 evaluates all RPs within a given cell, considering spatial constraints and possibly temporal constraints, to assign each RP to one or more trajectories within the cell (“cell trajectory”). FIG. 5A In the example, process 206 includes step 206A of initiating a new trajectory for the first RP selected in step 206B. In step 206B, an RP is selected from the available RPs. In step 206C, the selected RP is evaluated to detect another RP within a spatial search area (SSA) surrounding the selected RP. The SSA comprises multiple pixels in both the x and y directions and can have any shape, such as a circle, ellipse, square, rectangle, etc. If another RP is detected within the SSA by step 206C, step 206D can be performed to evaluate whether the representative time difference between the selected RP and the other RP is less than the maximum time difference (…). (T_MAX). If so, then if another RP is not already included in the track, step 206E is executed to add the other RP to the track of the selected RP. After step 206E, process 206 returns to step 206B to select a new RP from the unselected RPs. Process 206 continues until all RPs have been selected. If another RP fails the test in step 206B or step 206C, this means that the other RP does not belong to the same track as the selected RP, and a new track is started for the other RP in step 206F before the process returns to step 206B.

[0073] FIG. 5B The process described is given as a non-limiting example. For example, the search process can be optimized to avoid redundant comparisons. Furthermore, the applicant has found that, at least for some configurations of the system, the time evaluation of step 206C can be omitted without significantly reducing the accuracy of the trajectory data generated by step 210. In principle, any processing-efficient algorithm for spatial or spatiotemporal clustering can be used in step 206.

[0074] exist FIG. 5A In the example, the intra-cell evaluation in step 206 is shown as being performed for a frame period ( The corresponding batch of summary data generated during t2) is executed. Therefore, step 206 is executed once per frame period. In a variant, once an instance of summary data is generated in step 204, the intra-cell evaluation of step 206 is performed. In such a variant, the intra-cell evaluation is therefore performed once per summary period (t2). t1) is executed once. In other variations, it will be executed once. t1 and Intra-unit evaluation is performed on timescales between t2.

[0075] exist FIG. 7The process 206 is further illustrated, showing cell Ci and its pixel 33. The hollow circle indicates the RP determined for cell Ci by steps 202-205 during the frame period. A spatial search region SSA 40, in the form of a circle with radius r, is centered on the selected RP. In the illustrated example, two other RPs exist within SSA 40. Given SSA 40, process 206 will identify a first trajectory consisting of RP 41 and a second trajectory consisting of RP 42.

[0076] In some implementations, process 206 includes a termination step of evaluating the number of RPs in the corresponding trajectory and discarding any trajectory containing fewer than a predefined number of RPs. The applicant has found that this significantly improves the overall processing efficiency of process 104 by removing potentially irrelevant trajectories earlier. In one example, the predefined number is in the range of 2-10.

[0077] FIG. 5C It corresponds to FIG. 5A The flowchart of an example procedure for inter-cell evaluation 208 is provided. Procedure 208 evaluates the endpoints of cell trajectories between spatially adjacent cells with respect to spatial and temporal constraints, and selectively combines (“stitches”) the cell trajectories into a mesh trajectory extending across multiple cells. As used herein, an “endpoint” is either the start or end point of a trajectory. The start point is given by the RP with the earliest representative time, and the end point is given by the RP with the latest representative time.

[0078] Process 208 can be limited to evaluating cell trajectories with endpoints sufficiently close to the boundary between two cells. This improves processing efficiency by reducing the number of evaluations in process 208. For example, for a cell trajectory to be evaluated by process 208, the distance between one of its endpoints and the cell boundary can be less than a distance limit. The distance limit can correspond to the nominal distance between the center points of reflected pulses on the pixel array. Furthermore, process 208 can be limited to evaluating pairs of cell trajectories with endpoints sufficiently close to the same boundary. This further reduces the number of evaluations in process 208. Therefore, as... FIG. 5B As shown, process 208 may include an initial step 208A of determining a set of candidate pairs of cell trajectories to be evaluated.

[0079] exist FIG. 5BIn the example, process 208 further includes a step 208B of selecting a pair of cell trajectories for evaluation. The selected pair includes a first cell trajectory in a first cell and a second cell trajectory in a second cell adjacent to the first cell. The first trajectory has an endpoint, and the second cell trajectory has a starting point located near the boundary between the first and second cells. In step 208B, the time difference between the endpoint of the first trajectory and the starting point of the second trajectory is determined, given by the difference between their representative times. In step 208C, the distance between the endpoint of the first trajectory and the starting point of the second trajectory on the pixel array is determined, given by the difference between their RP. Therefore, step 208B determines the time interval between the two cell trajectories, and step 208C determines the spatial interval between the two cell trajectories. In step 208D, regarding the maximum time difference ( The time interval (T_MAX) is evaluated from step 208B, and this maximum time difference can be the same as the time difference used in step 206D. If the time interval exceeds... If T_MAX, the process returns to step 208B, which selects another pair of cell trajectories for evaluation. If the time interval is acceptable, step 208E is performed to evaluate the spatial interval from step 208C with respect to the maximum distance (D_MAX). If the spatial interval exceeds D_MAX, the process returns to step 208B, which selects another pair of cell trajectories for evaluation. If the spatial separation is acceptable, step 208F is performed to merge the first and second trajectories to form a mesh trajectory. When merging the trajectories, the endpoint of the first trajectory is logically connected to the starting point of the second trajectory. Step 208E may or may not involve including the RPs of the first and second trajectories in a common data structure. Process 208 continues until all relevant cell trajectory pairs have been evaluated.

[0080] exist FIG. 8A The process 208 is further illustrated, showing two cells Ci and Cj with a common boundary Bij. Hollow circles indicate the RPs determined for cells Ci and Cj during the frame period by steps 202-205. As indicated, the RP in cell Ci has been assigned to cell trajectory CTi_1, and the RP in cell Cj has been assigned to cell trajectories CTj_1 and CTj_2. Through step 208A, CTi_1 and CTj_1 have been identified as candidate pairs for merging. In the illustrated example, as in steps 208C and 208E (… FIG. 5C In the spatial evaluation portion, a circular SSA 40 with radius r is centered at the endpoint 43 of CTi_1, and the existence of the starting point 44 of CTj_1 is evaluated. Therefore, in the example shown, radius r corresponds to D_MAX. FIG. 8A In the process, spatial evaluation will lead to the decision not to merge CTi_1 and CTj_1.

[0081] Another example of the operation of procedure 208 is in FIG. 8B As shown in the figure, FIG. 8B This is a planar diagram of element Cn, where n is the element index. Element pairs Cn and Cm are separated by boundaries specified by Bnm. Element trajectories CTn have been determined for each of C1, C2, C3, C6, and C7 via step 206. Each element trajectory has a start point and an end point specified by Sn and En, respectively. Assume that step 208A has identified the following CT pairs as candidate pairs for merging: (CT 1, CT 2), (CT 2, CT 3), and (CT 6, CT7). In the spatial evaluation of (CT 1, CT 2), SSA 40 is arranged on E1 and the presence of S2 is evaluated. Assume that (CT 1, CT 2) is also evaluated temporally, and CT1 and CT2 will be merged to form a first GT. Correspondingly, in the spatial evaluation of (CT 2, CT 3), SSA 40 is arranged on E2 and the presence of S3 is evaluated. Therefore, the spatial evaluation will result in a decision not to merge CT2 and CT3, and CT3 will thus form a second GT. In the case of (CT 2, CT 3) through both spatial and temporal evaluations, CT3 will be merged with CT2, thus being included in the first GT. In the evaluation of (CT 6, CT 7), SSA 40 is arranged on E6 and the presence of S7 is evaluated. The spatial evaluation will result in a decision not to merge CT6 and CT7, thus forming the third GT and the fourth GT, respectively.

[0082] Now refer to FIG. 5D and FIG. 8C to FIG. 8E Explanation, inspiration, and illustration FIG. 5A Adjustment step 207. Adjustment step 207 is performed to spatially adjust the endpoints of the cell trajectory determined in step 206. The motivation for step 207 is dual. Proceed to... FIG. 8D The figure shows a series of light spots 23 on object 1 as the light beam 21' sweeps across object 1. Each light spot 23 corresponds to a light pulse ( FIG. 3 22). In FIG. 8D In the process, one of the light spots 23 strikes the outer edge or contour 1a of the object 1. This means that only a portion of the light spot 23 will be reflected by the object 1. FIG. 8E A portion of the pixel array is shown, which receives signals from... FIG. 8D The reflected pulse of the light spot 23 impacting the edge 1a. Pixels activated by the reflected pulse are indicated by black squares. FIG. 8E RP 47, which can be calculated based on the pixel position of the activated pixel, is also shown. FIG. 8EThe dotted area in the figure indicates the additional pixels that would have been activated if spot 23 had been completely reflected by object 1. Reference numeral 47' indicates the true RP calculated for the combination of black and dashed pixels. It should be recognized that the calculated RP 47 is not an exact representation of the position of edge 1a located at the true RP 47'. Therefore, one motivation for step 207 is to improve the determination of edge positions on the object. A second motivation is to improve the positional accuracy of endpoints located near the boundaries between cells. This can also be achieved if the black and dashed squares are seen to be located in different cells separated by boundaries. FIG. 8E This is understood in the context of the boundary extending vertically through the reflected pulses on the pixel array. Clearly, there will be a difference between the calculated RP 47 and the actual RP 47' for the pixel position in the left cell. The same applies to the right cell. Adjusting the endpoints can improve the accuracy of the inter-cell evaluation in step 208.

[0083] Adjustment step 207 is based on the following understanding: regarding the summary cycle ( Information about the range of activated pixels during t1) can be used to adjust the RP in the summary data to make the RP closer to the true RP. This is why the range parameter SEi is included in the summary data.

[0084] FIG. 5D It corresponds to FIG. 5A A flowchart of an example process for step 207 is provided. Process 207 can be performed for each endpoint of each cell trajectory generated by step 206, or only for endpoints located sufficiently close to the boundary between two cells (e.g., within D_MAX). In step 207A, a range parameter is retrieved from the summary data of the current endpoint to be adjusted. In step 207B, the RP of the current endpoint is adjusted based on the range parameter from step 207A. Adjustment is typically used to extend the cell trajectory, regardless of whether the adjustment is performed for the start or end point. In some implementations, step 207B includes determining the direction of movement of the cell trajectory at the current endpoint and adjusting the RP of the current endpoint by spatial shifting in the direction of movement. The RP of the start point will be shifted in the opposite direction of movement, and the RP of the end point will be shifted in the direction of movement. The direction of movement of the cell trajectory is given directly by the time series of the RPs in the trajectory. In some implementations, for example, if it is known that the scan beam scans back and forth approximately parallel to the coordinate axes of the pixel array (e.g., horizontally), one of the first and second coordinates can be ignored during step 207. In this case, adjustments can be made only along this coordinate axis for all cell trajectories.

[0085] In some implementations, spatial shift is determined as a function of a range parameter and optionally a direction of movement. For example, spatial shift can be a shift vector in the local coordinate system of the pixel array. As used herein, a “vector” has both a magnitude and a direction. The direction of the shift vector can be parallel to the direction of movement. In step 207B, the shift vector can be added to the RP of the current endpoint.

[0086] In some implementations, step 207 can switch between using different range parameters to determine the adjustment. For example, the summary data may include a first range parameter and a second range parameter, where the first range parameter is considered a better approximation of the range but is more sensitive to outliers than the second range parameter. The first range parameter may be the minimum and maximum pixel positions, or the corresponding range values. The second range parameter may be any other outlier suppression measure, such as standard deviation or statistical deviation. Step 207B may, by default, determine the adjustment based on the first range parameter. However, if the ratio between the range value and the standard deviation exceeds a predefined value, step 207B may switch to determining the adjustment based on the second range parameter. A large ratio may indicate that the first parameter is affected by outliers and does not correctly represent the range.

[0087] exist FIG. 8C The effect of process 207 is further illustrated in the text. FIG. 8C Two elements Ci and Cj with a common boundary Bij are shown. An endpoint 45 has been determined in element Ci, and a starting point 46 has been determined in element Cj. In the inter-element evaluation performed in step 208, an SSA 40 is positioned at the endpoint 45, and the presence of the starting point is evaluated. As shown, the starting point 46 is located outside the SSA 40, and even though the element trajectories should be merged, they are not merged. In step 207, a first shift vector SV1 is determined for the endpoint 45, and a second shift vector SV2 is determined for the starting point 46. An adjusted endpoint 43' is determined by adding SV1 to the endpoint 45, and an adjusted starting point 45' is determined by adding SV2 to the starting point 46. After this adjustment, the SSA 40' positioned at the adjusted endpoint 46' will include the adjusted starting point 46'. Therefore, the inter-element evaluation in step 208 will correctly merge the element trajectories.

[0088] Now refer to FIG. 5E and FIG. 9 Explanation, inspiration, and illustration FIG. 5A Inter-frame evaluation 209. The motivation for step 209 is that the grid trajectory can continue beyond the frame period ( The expiration of t2). In other words, even if the grid trajectory has an apparent end point on the pixel array, the same grid trajectory can actually continue from the apparent starting point determined by steps 206 and 208 when processing the summary data in subsequent frames (i.e., for subsequent frame periods).

[0089] Since it is desirable to provide trajectory data as quickly as possible, step 209 of the inter-frame evaluation is preferably performed by comparing the start point of the current frame's grid trajectory with the end point of the previous frame's grid trajectory. This is in FIG. 9 The following example illustrates this. Here, the mesh trajectory of the current frame GT has an apparent starting point 48 in cell Ci and extends across boundary Bij into cell Cj, while the mesh trajectory of the previous frame GTp extends across boundary Bhi from cell Ch into cell Ci and has an apparent ending point 49 within cell Ci. In inter-frame evaluation, circle SSA 40 is centered on the starting point 48 of GT, and the existence of the ending point is evaluated. In the illustrated example, the ending point 49 of GTp is located within SSA 40. If the time difference between points 48 and 49 meets a time criterion, GTp and GT can be combined (stitched together) to form an updated GT.

[0090] Now go to FIG. 5E As can be seen from the example, step 209 is similar to step 208. Step 209 can be limited to the end point in a previous frame that is temporally close to the end of the previous frame, and the start point in the current frame that is temporally close to the start of the current frame. In this context, "temporally close" can imply that the time difference is less than a predefined limit. This will improve processing efficiency by reducing the number of evaluations in step 209. Therefore, as... FIG. 5E As shown, process 209 may include an initial step 209A of determining a set of candidate pairs of grid trajectories to be evaluated.

[0091] exist FIG. 5E In the example, process 209 further includes step 209B of selecting a pair of grid trajectories for evaluation. The selected pair includes a first grid trajectory from a previous frame and a second grid trajectory from the current frame. The first trajectory has an endpoint, and the second grid trajectory has a starting point that is temporally close to the location during the transition from the previous frame to the current frame. In step 209B, the time difference between the endpoint of the first trajectory and the starting point of the second trajectory is determined, which is given by the difference between their representative times when they are transferred to a common time scale. In step 209C, the distance on the pixel array between the endpoint of the first trajectory and the starting point of the second trajectory is determined, given by the difference between their RPs. Therefore, step 209B determines the time interval between the two grid trajectories in different frames, and step 209C determines the spatial interval between the two grid trajectories. In step 209D, regarding the maximum time difference ( The time interval (T_MAX) is evaluated from step 209B, and this maximum time difference can be the same as the time difference used in step 208D. If the time interval exceeds... If T_MAX, the process returns to step 209B, which selects another pair of mesh trajectories for evaluation. If the time interval is acceptable, step 209E is performed to evaluate the spatial interval from step 209C with respect to a maximum distance (D_MAX), which can be the same as the distance used in step 208E. If the spatial interval exceeds D_MAX, the process returns to step 209B. If the spatial separation is acceptable, step 209F is performed to merge the first and second mesh trajectories. Process 209 continues until all relevant mesh trajectory pairs have been evaluated.

[0092] In the above text, we assume a summary cycle ( t1) is a fixed period. In the variant, the summary period is variable and determined by the presence of incoming events. FIG. 5A In the example, step 203 can be replaced by the step of checking whether the time since the last received event has exceeded the time limit for each cell. If the time limit has been exceeded, step 204 is performed for that cell. The time limit can be set to be greater than the pulse duration P and less than the pulse-to-pulse interval PP. This means that the summary period can vary between cells and also changes over time for the respective cells.

[0093] In another variant, step 203 can be synchronized with the generation time of the light pulse via a synchronization signal provided from the beam scanner 21 to the processing subsystem 61. Therefore, a summarization cycle can be proactively initiated when it is known that a light pulse has been generated and therefore a reflected pulse should be received by the pixel array. This can improve the accuracy of the summarization data and the robustness of trajectory determination. For example, in the event that another system happens to illuminate the scene with a scanning or non-scanning beam, time synchronization can significantly reduce interference from the other system on the summarization data.

[0094] Although the foregoing description has been given for pulsed light, the technique is equally applicable to scanning beams composed of continuous light (also known as continuous wave (cw) light). For cw light, the summary period ( t1) cannot be defined in terms of the pulsation of light, but it can still be defined as the appropriate spatial and temporal resolution for achieving summary data. However, compared to pulsed light, cw light may result in more noise-affected summary data.

[0095] FIG. 10 This is a block diagram of an exemplary device or machine 1000, which can be configured to perform any of the methods, processes, and functions or portions thereof described herein via a combination of software and hardware circuitry or specifically via particular hardware circuitry. FIG. 10In this document, device 1000 includes processor circuitry 1001, which may be or include a central processing unit (CPU), graphics processing unit (GPU), microcontroller, microprocessor, ASIC, FPGA, or any other specific or general-purpose processing device. Device 1001 can execute instructions 1002A stored in memory (such as memory 1002) to control the operation of device 1000. Instructions 1002A, when executed by processor circuitry 1001, can cause device 1000 to perform any of the methods described herein or any part thereof. Instructions 1002A may be provided to device 1000 on a computer-readable medium 1010, which may be a tangible (non-transitory) product (e.g., magnetic media, optical disc, read-only memory, flash memory, etc.) or a propagating signal. FIG. 10 As shown, the memory 1002 can also store data 1002B for use by the processor circuit 1001, such as cell definitions, t1、 t2, D_MAX, SSA, r, The value of T_MAX, etc. Memory 1002 may include one or more of a buffer, flash memory, hard disk drive, removable media, volatile memory, non-volatile memory, random access memory (RAM), or other suitable data storage devices. Such memory 1002 is considered a non-transitory computer-readable medium. Device 1000 may also include an I / O interface 1003, which may include any conventional communication interface for wired or wireless communication. For example, device 1000 may be arranged to receive a stream of events (see...). FIG. 1B (ES in the middle) and provides trajectory data TD via interface 1003 (see FIG. 1A ).

[0096] While the subject matter of this disclosure has been described in conjunction with embodiments currently considered to be most practical, it should be understood that the subject matter of this disclosure is not limited to the disclosed embodiments, but rather is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

[0097] Furthermore, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order shown or sequentially, or to perform all the operations shown in order to achieve the desired result.

[0098] The following description outlines some aspects and implementation methods disclosed above.

[0099] C1. A computer-implemented method for processing a data stream of events from an event-based sensor (30), the event-based sensor (30) comprising a pixel array (31) and arranged to receive photons reflected or scattered by a region on an object (1) when irradiated by a scanning beam (21), wherein each event in the data stream originates from a pixel (33) in the pixel array (31) and includes the pixel position of the pixel (33) and a timestamp associated with the event, the method comprising: defining (201) a grid of cells (32) of the pixel array (31), each cell (32) comprising a plurality of pixels (33); generating (202-205) for a corresponding cell (32) in the grid, comprising a first time period ( 1) Summary data of incoming events that fall within the corresponding unit (32) during the first time period, the summary data including the pixel positions of the incoming events during the first time period (32) Representative positions of the pixels falling within the corresponding unit during t1); Execute (206) for the corresponding unit (32) for multiple first time periods ( The second time period of t1) t2) Intra-cell evaluation of the generated summary data to generate cell trajectories at representative locations forming spatially and temporally coherent trajectories within the corresponding cells; Execution (208) in the second time period ( t2) Inter-cell evaluation of multiple cell trajectories generated for multiple cells in the grid to generate grid trajectories at representative locations forming spatially and temporally coherent trajectories within the multiple cells; and provides (210) trajectory data representing the grid trajectories.

[0100] C2. The method according to C1, wherein the summary data includes representative timestamps of the incoming events, and wherein the spatial coherence trajectory and the temporal coherence trajectory include a sequence of representative locations, the representative locations being separated from each other by less than a maximum time difference and by less than a maximum distance at the representative locations.

[0101] C3. The method according to C2, wherein the evaluation within the unit of execution (206) includes: evaluating (206C) a corresponding representative position in the summary data for detecting another representative position within the search area (40) at a maximum distance around the corresponding representative position, and if another representative position is detected within the search area (40), including (206E) the other representative position in the unit trajectory of the corresponding representative position.

[0102] C4. The method according to C3, wherein the inclusion (206E) of the other representative position is performed only if the corresponding representative position and the other representative position are separated in representative time (206D) by less than the maximum time difference.

[0103] C5. The method according to any one of C2 to C4, wherein each unit trajectory extends from a starting point to an ending point, wherein the inter-unit evaluation (208) includes: with regard to the maximum time difference evaluation (208B) the time difference between the ending point of the first unit trajectory of the first unit in the grid and the starting point of the second unit trajectory of the second unit in the grid, wherein the second unit is spatially adjacent to the first unit; and with regard to the maximum distance evaluation (208C) the distance between the representative position of the ending point and the representative position of the starting point, and if the time difference is less than the maximum time difference (208D) and the distance is less than the maximum distance (208E), then the first unit trajectory and the second unit trajectory are included in the grid trajectory (208F).

[0104] C6. The method according to C5, wherein the inter-unit evaluation (208) is performed only when the representative locations of the endpoints and the representative locations of the starting points are within the distance limit from the boundary between the first unit and the second unit.

[0105] C7. The method according to any of the foregoing clauses, wherein the summary data further includes a range parameter, the range parameter representing the range within the first time period ( The method further includes an adjustment process (207) comprising: obtaining (207A) the range parameters of the endpoints of the corresponding unit trajectory from the summary data, and adjusting (207B) the representative position of the endpoints based on the range parameters.

[0106] C8. The method according to C7, wherein the endpoint is at least one of the start point or the end point of the corresponding unit trajectory.

[0107] C9. The method according to C7 or C8, wherein the adjustment (207B) of the representative position includes determining a shift vector in the coordinate system of the pixel array based on the at least one range parameter, wherein the representative position of the endpoint is adjusted by adding the shift vector to the representative position of the endpoint.

[0108] C10. The method according to any one of C7 to C9, wherein the adjustment process results in the extension of the corresponding unit trajectory.

[0109] C11. The method according to any one of C7 to C10, wherein the range parameter includes one or more of the following: minimum and maximum values ​​of pixel positions; range of pixel positions; standard deviation of pixel positions; variance of pixel positions; quartile range of pixel positions; median absolute deviation of pixel positions; or mean absolute deviation of pixel positions.

[0110] C12. The method according to any of the foregoing clauses, the method further comprising: performing a (210A) fitting algorithm on the representative position of the grid trajectory to generate curve definition data of a curve fitted to the representative position of the grid trajectory, wherein the trajectory data includes the curve definition data.

[0111] C13. The method according to any of the foregoing clauses, wherein the representative position corresponds to the first time period ( During t1), the center point of the pixel position falls within the corresponding unit.

[0112] C14. According to the method described in C13, wherein the center point is calculated in the first time period ( t1) is the average value of the pixel positions that fall into the corresponding cell during the period.

[0113] C15. The method according to any of the preceding clauses, wherein the pixel position is defined by a first coordinate and a second coordinate in coordinate system (10'), and wherein the representative position is defined by a first value and a second value in coordinate system (10'), wherein the first value and the second value are respectively calculated as follows during the first time period ( The average of the first and second coordinates of the pixel position falling within the corresponding unit during period t1).

[0114] C16. The method according to any of the foregoing clauses, wherein only when the first time period ( When the count of pixel positions falling into the corresponding unit (32) during t1 exceeds the threshold, the generation of summary data (202-205) is performed.

[0115] C17. The method according to any of the preceding clauses, wherein the cells (32) in the grid are defined as non-overlapping.

[0116] C18. The method according to any of the foregoing clauses, wherein the light in the scanning beam (21) is pulsed light.

[0117] C19. The method according to C18, wherein the pulsed light comprises a sequence of optical pulses, and wherein the first time period ( t1) includes a predefined number of light pulses.

[0118] C20. The method according to C19, wherein the predefined quantity is one.

[0119] C21. The method according to any one of C18 to C20, wherein the generation (202-205) of the summary data is performed in time synchronously with the pulsation of the pulsed light.

[0120] C22. The method according to any of the foregoing clauses, wherein the second time period ( t2) includes at least 50, 100, or 150 first time periods ( t1).

[0121] C23. A computer-readable medium comprising computer instructions (1002A) that, when executed by processor circuitry (1001), cause processor circuitry (1001) to perform any one of C1 to C22.

[0122] C24. A processing apparatus comprising an interface (1003) for receiving a data stream of events from event-based sensors (30; 40; 50), and configured to perform the method described in any one of C1 to C22.

[0123] C25. The processing apparatus according to C24, wherein the processing apparatus is or includes an FPGA, an ASIC, or a GPU.

[0124] C26. A system comprising: at least one beam scanning device (20) configured to generate a scanning beam (21) of light to illuminate an object (1); at least one event-based sensor (30, 40, 50) comprising a pixel array (31, 41, 51) and arranged to receive photons reflected or scattered by a region on the object (1) when illuminated by the scanning beam (21), wherein the at least one event-based sensor (30, 40, 50) is configured to receive a number of photons received by a pixel (32). When the amount exceeds a threshold, an event is generated for pixel (32) in the pixel array (31, 41, 51), wherein the at least one event-based sensor (30, 40, 50) is configured to output the event as a corresponding data stream, wherein each event in the corresponding data stream includes the pixel position of the pixel (32) and a timestamp associated with the event; and a processing subsystem (61) comprising at least one processing device according to C24 or C25 and configured to receive the corresponding data stream from at least one event-based sensor (30, 40, 50) and output trajectory data.

[0125] C27. The system according to C26 further includes a detection subsystem (62) configured to receive trajectory data and determine the characteristics of the object (1) based thereon.

Claims

1. A computer-implemented method for processing a data stream of events from an event-based sensor (30), the event-based sensor (30) comprising a pixel array (31) and arranged to receive photons reflected or scattered by regions on an object (1) when irradiated by a scanning beam (21), wherein, Each event in the data stream originates from a pixel (33) in the pixel array (31) and includes the pixel position of the pixel (33) and a timestamp associated with the event. The method includes: The pixel array (31) defines a grid of units (32) of (201), each unit (32) comprising a plurality of pixels (33). For the corresponding cell (32) in the grid, generate (202-205) including in the first time period ( Summary data of incoming events that fall within the corresponding unit (32) during the first time period (t1), the summary data including the pixel positions of the incoming events during the first time period (t1). The representative position of the pixel that falls within the corresponding unit during period t1) Execute (206) for the corresponding unit (32) for multiple first time periods ( The second time period of t1) t2) Intra-cell evaluation of the generated summary data to generate cell trajectories at representative locations within the corresponding cells that form spatially and temporally coherent trajectories. Execution (208) during the second time period ( t2) Inter-cell evaluation of multiple cell trajectories generated for multiple cells in the grid, to generate grid trajectories at representative locations forming spatially and temporally coherent trajectories within the multiple cells, and Provide (210) trajectory data representing the grid trajectory.

2. The method according to claim 1, wherein, The summary data includes representative timestamps of the incoming events, and the spatial and temporal coherent trajectories include a sequence of representative locations that are spaced apart by a maximum time difference and by a maximum distance at the same time.

3. The method according to claim 2, wherein, The execution (206) of the intra-cell evaluation includes: evaluating (206C) a corresponding representative location in the summary data for detecting another representative location within the search area (40) that defines the maximum distance around the corresponding representative location, and if the other representative location is detected within the search area (40), including (206E) the other representative location in the cell trajectory of the corresponding representative location.

4. The method according to claim 3, wherein, The inclusion of (206E) the other representative position is performed only if the corresponding representative position and the other representative position are separated in representative time (206D) by less than the maximum time difference.

5. The method according to claim 2, wherein, Each unit trajectory extends from a starting point to an ending point, wherein performing the inter-unit evaluation (208) includes: evaluating (208B) the time difference between the ending point of the first unit trajectory of the first unit in the grid and the starting point of the second unit trajectory of the second unit in the grid, wherein the second unit is spatially adjacent to the first unit; and evaluating (208C) the distance between the representative position of the ending point and the representative position of the starting point, and if the time difference is less than the maximum time difference (208D) and the distance is less than the maximum distance (208E), then the grid trajectory includes (208F) the first unit trajectory and the second unit trajectory.

6. The method according to claim 5, wherein, The inter-cell evaluation (208) is performed only if the representative location of the endpoint and the representative location of the starting point are within the distance limit from the boundary between the first cell and the second cell.

7. The method according to claim 1, wherein, The summary data also includes a range parameter, which represents the range within the first time period ( The method further includes an adjustment process (207) comprising: obtaining (207A) the range parameters of the endpoints of the corresponding unit trajectory from the summary data, and adjusting (207B) the representative position of the endpoints based on the range parameters.

8. The method according to claim 7, wherein, The endpoint is at least one of the start or end point of the corresponding unit trajectory.

9. The method according to claim 7, wherein, The adjustment (207B) of the representative position includes determining a shift vector in the coordinate system of the pixel array based on the at least one range parameter, wherein the representative position of the endpoint is adjusted by adding the shift vector to the representative position of the endpoint.

10. The method according to claim 7, wherein, The adjustment process results in the extension of the corresponding unit trajectory.

11. The method according to claim 7, wherein, The range parameter includes one or more of the following: the minimum and maximum values ​​of the pixel position; the range of the pixel position; the standard deviation of the pixel position; the variance of the pixel position; the quartile range of the pixel position; the median absolute deviation of the pixel position; or the mean absolute deviation of the pixel position.

12. The method according to claim 1, further comprising: The representative position of the grid trajectory is operated on by a fitting algorithm (210A) to generate curve definition data of a curve fitted to the representative position of the grid trajectory, wherein the trajectory data includes the curve definition data.

13. The method according to claim 1, wherein, The representative position corresponds to the first time period ( The center point of the pixel location that falls within the corresponding unit during t1).

14. The method according to claim 13, wherein, The center point is calculated as being within the first time period ( The average value of the pixel position falling within the corresponding unit during the period t1).

15. The method according to claim 1, wherein, The pixel position is defined by a first coordinate and a second coordinate in the coordinate system (10'), and the representative position is defined by a first value and a second value in the coordinate system (10'), wherein the first value and the second value are respectively calculated as follows during the first time period ( The average of the first and second coordinates of the pixel position falling within the corresponding unit during period t1).

16. The method according to claim 1, wherein, Only when the first time period ( When the count of the pixel positions falling into the corresponding unit (32) during t1) exceeds the threshold, the summary data (202-205) is generated.

17. The method according to claim 1, wherein, The cells (32) in the grid are defined as non-overlapping.

18. The method according to claim 1, wherein, The light in the scanning beam (21) is pulsed light.

19. The method according to claim 18, wherein, The pulsed light comprises a sequence of light pulses, and wherein the first time period ( t1) includes a predefined number of light pulses.

20. The method according to claim 19, wherein, The predefined quantity is one.

21. The method according to claim 18, wherein, The generation of the summary data (202-205) is performed synchronously with the pulsation time of the pulsed light.

22. The method according to claim 1, wherein, The second time period ( t2) includes at least 50, 100, or 150 first time periods ( t1).

23. A computer-readable medium comprising computer instructions (1002A) that, when executed by a processor circuit (1001), cause the processor circuit (1001) to perform the method of any one of claims 1 to 22.

24. A processing apparatus comprising an interface (1003) for receiving a data stream of events from event-based sensors (30; 40; 50), and configured to perform the method of any one of claims 1 to 22.

25. The processing apparatus of claim 24, wherein the processing apparatus is or includes an FPGA, an ASIC, or a GPU.

26. A system comprising: At least one beam scanning device (20) is configured to generate a scanning beam (21) to illuminate the object (1). At least one event-based sensor (30, 40, 50), the at least one event-based sensor comprising a pixel array (31, 41, 51) and arranged to receive photons reflected or scattered by a region on the object (1) when illuminated by the scanning beam (21), wherein the at least one event-based sensor (30, 40, 50) is configured to generate an event for a pixel (32) in the pixel array (31, 41, 51) when the number of photons received by the pixel (32) exceeds a threshold, wherein the at least one event-based sensor (30, 40, 50) is configured to output the event as a corresponding data stream, wherein each event in the corresponding data stream includes the pixel position of the pixel (32) and a timestamp associated with the event; and The processing subsystem (61) includes at least one processing device according to claim 24, and is configured to receive corresponding data streams from the at least one event-based sensor (30, 40, 50) and output trajectory data.

27. The system of claim 26, further comprising a detection subsystem (62) configured to receive the trajectory data and determine characteristics of the object (1) based on the trajectory data.

Images