System, event sensor, data processing device and method

Abstract

The present disclosure relates to systems, event sensors, data processing devices and methods. A system comprising: an event sensor comprising a plurality of event pixels, each event pixel configured to detect a change in luminance of a scene based on a reference value and a threshold value to generate an event, the event sensor configured to output an event stream; a data source configured to provide a data stream comprising sensor data or control data; a data processing device comprising circuitry configured to: receive the event stream from the event sensor and the data stream from the data source; predict a change in the scene based on the event stream and the data stream; generate a luminance prediction model for at least one event pixel based on the predicted change in the scene and the event stream; and send the luminance prediction model to the event sensor, the event sensor configured to receive the luminance prediction model for the at least one event pixel from the data processing device and adapt the reference value of the at least one event pixel according to the received luminance prediction model.

Description

Technical Field

[0001] This disclosure generally relates to a system, event sensor, data processing device, and method. Background Technology

[0002] Typically, event sensors or cameras are known. Compared to classic cameras, event cameras (which may also be called neuromorphic cameras, silicon retinas, or dynamic vision sensors) report pixel-by-pixel changes in brightness asynchronously with wait times typically on the order of microseconds.

[0003] Its function can be summarized as assuming a static scene (e.g., including fixed scene settings, objects, camera positions, lighting, and reflection characteristics) and reporting any deviations from this, where such deviations result in brightness changes detected by the event camera.

[0004] Known or regular event cameras implement their functionality by storing pixel-by-pixel reference values ​​for anticipated measurements, and generating an event if the measurement deviates from that reference value and a predetermined range around it (possibly represented by a threshold). The reference value of the event pixel is then updated to match the new measurement.

[0005] However, in scenarios where changes are normal (i.e., triggering a large number of events), data transmission and signal processing require high bandwidth. While such infrastructure can meet these peak requirements—however, this infrastructure increases production and energy costs—there is also a risk of information loss if limits are exceeded.

[0006] Despite the existence of technologies for event sensors and for controlling the generation of events by event sensors, improvements to existing technologies are generally desired. Summary of the Invention

[0007] According to the first aspect, this disclosure provides a system comprising: An event sensor includes multiple event pixels, each event pixel being configured to detect brightness changes in the scene based on a reference value and a threshold to generate an event, and wherein the event sensor is configured to output an event stream; The data source is configured to provide a data stream, wherein the data stream includes sensor data or control data; A data processing device, including circuitry, configured to: Receive event streams from event sensors and data streams from data sources; Predict changes in the scenario based on this event stream and data stream; Based on the predicted changes in the scene and the event flow, a brightness prediction model is generated for at least one event pixel. The brightness prediction model is sent to the event sensor; and The event sensor is configured to receive a brightness prediction model for at least one event pixel from a data processing device, and to adapt a reference value for the at least one event pixel according to the received brightness prediction model.

[0008] According to a second aspect, this disclosure provides an event sensor, comprising: Multiple event pixels, wherein each event pixel is configured to detect changes in scene brightness based on a reference value and a threshold to generate an event; The circuit is configured as follows: Output event stream; Receive a brightness prediction model for at least one event pixel; and The reference value of the at least one event pixel is adapted according to the received brightness prediction model.

[0009] According to a third aspect, the present invention provides a data processing apparatus, including a circuit configured to: Receive event streams from event sensors and data streams from data sources, wherein the event streams include events generated based on detected changes in scene brightness, and wherein the data streams include sensor data or control data; Predict changes in the scenario based on this event stream and data stream; Based on the predicted changes and event flow of the scene, a brightness prediction model is generated for at least one event pixel; and The brightness prediction model is sent to the event sensor to adapt the reference value of the at least one event pixel according to the brightness prediction model.

[0010] According to a fourth aspect, the present invention provides a method comprising: Receive an event stream from an event sensor and a data stream from a data source, wherein the event stream includes events generated based on detected changes in brightness of the scene, and wherein the data stream includes sensor data or control data; Predict changes in the scenario based on the event stream and data stream; Based on the predicted changes and event flow of the scene, a brightness prediction model is generated for at least one event pixel; and The brightness prediction model is sent to the event sensor to adapt the reference value of the at least one event pixel according to the brightness prediction model.

[0011] Further aspects are set forth in the dependent claims, the drawings and the following description. Attached Figure Description

[0012] Referring to the accompanying drawings, embodiments are illustrated by way of example, wherein: Figure 1A block diagram schematically illustrating an embodiment of the system is shown; Figure 2 A block diagram schematically illustrates an embodiment of a regular event pixel; Figure 3 A block diagram schematically illustrates an embodiment of the event pixel; Figure 4 A block diagram schematically illustrating an embodiment of the system is shown; Figure 5 A line graph illustrating an embodiment of the brightness prediction model is shown schematically; Figure 6 A line graph illustrating an embodiment of the brightness prediction model is shown schematically; Figure 7 A line graph illustrating an embodiment of the brightness prediction model is shown schematically; Figure 8 A line graph illustrating an embodiment of the brightness prediction model is shown schematically; Figure 9A and Figure 9B An embodiment of a method for generating a brightness prediction model is illustrated schematically, wherein, Figure 9A A block diagram schematically illustrates an embodiment of a matrix of event pixels, and Figure 9B A line graph schematically illustrates an embodiment of future time-dependent brightness associated with an event pixel; Figure 10A and Figure 10B An embodiment of a method for generating a brightness prediction model is illustrated schematically, wherein, Figure 10A A block diagram schematically illustrates an embodiment of a matrix of event pixels, and Figure 10B A line graph schematically illustrating an embodiment of future time-dependent brightness associated with an event pixel; and Figure 11 A flowchart illustrating an embodiment of the method is shown schematically. Detailed Implementation

[0013] Provide a reference Figure 3 Before a detailed description of the embodiments, a general explanation will be given.

[0014] As mentioned at the beginning, event sensors or cameras are usually known.

[0015] To enhance the overall understanding of this disclosure, reference will be made below to the following details. Figure 1 and Figure 2 This paper discusses embodiments of a system using event sensor 2 and embodiments of regular event pixels 10. Figure 1 and Figure 2 Corresponding embodiments are shown.

[0016] refer to Figure 1Scene 1 has a specific brightness detected by event sensor 2, which has multiple event pixels typically arranged in a matrix. If the brightness fluctuates and the fluctuation exceeds a certain threshold, the event pixel that has detected the brightness change generates a signal indicating the event.

[0017] Then, event sensor 2 generates an event based on the signal from the corresponding event pixel. The event includes the location or pixel coordinates of the event pixel that triggered the event, a timestamp, and a polarity indicating whether the brightness is increasing or decreasing. The event may also include the actual brightness value, such as a measurement and output as intensity.

[0018] Event sensor 2 can then output an event stream that includes events generated based on brightness changes detected in scene 1.

[0019] The event stream can be provided to downstream module 3. Downstream module 3 can be an application that uses the event stream; for example, the application could be a monitoring application or an application used to generate a 3D representation of a scene for navigating vehicles in the environment.

[0020] Now for reference Figure 2 Event pixel 10 includes a light detector 11 configured to detect the brightness of a portion of the scene imaged on event pixel 10. Brightness can be measured as intensity I.

[0021] Event pixel 10 determines at certain points in time, based on clock signal 12, whether the measured light intensity or brightness is within a predetermined range given by the current reference value R (which may also be referred to as the expected brightness value) and thresholds ε (+) and ε (-).

[0022] Once event pixel 10 determines that the brightness exceeds the predetermined range, the event is triggered and the reference value is updated to R'=R+(ε(+)+ε(-)) for a positive event (increase in brightness) and to R'=R-(ε(+)+ε(-)) for a negative event (decrease in brightness).

[0023] However, returning to the general interpretation, as mentioned at the beginning, in scenarios where scene changes are normal (i.e., many events are triggered), data transmission and signal processing require high bandwidth.

[0024] Typically, event sensors can be static or mobile. In the case of a static event sensor, only the dynamic parts of the scene will generate events, such as moving objects (e.g., cars or pedestrians), and the static parts of the scene (e.g., buildings) will not generate events if the illumination does not change. In the case of a mobile event sensor, the static parts of the scene also contribute to event generation because the static parts of the scene move relative to the event sensor, causing many events to be generated due to the movement of the event sensor in the static world. However, the dynamic parts of the scene also cause event generation. Thus, events generated due to movement in the static world can overshadow events generated due to movement in the dynamic world (e.g., cars). This makes it more difficult to separate static world events from subsequent dynamic world events.

[0025] It has been recognized that event generation by event sensors should be controlled in a way that reduces or removes events caused by predictable scene changes, and therefore does not result in information gain when understanding the scene. Residual events are caused by deviations from predictions, and may be important, for example, for downstream applications because these residual events may be primarily caused by the dynamic world.

[0026] Therefore, it has been recognized that conventional event sensors should be modified to accept models of future expected brightness values ​​for each pixel from external sources.

[0027] Therefore, some embodiments relate to an event sensor, wherein the event sensor includes: Multiple event pixels, wherein each event pixel is configured to detect changes in scene brightness based on a reference value and a threshold to generate an event; The circuit is configured as follows: Output event stream; Receive a brightness prediction model for at least one event pixel; and The reference value of the at least one event pixel is adapted according to the received brightness prediction model.

[0028] In some embodiments, each event pixel is equipped with additional logic (which may be referred to as a reference value generator) that stores the received brightness prediction model and generates a reference value that may change over time based on the brightness prediction model to replace the constant reference value of a conventional event sensor.

[0029] In addition, in some embodiments, the threshold used to identify significant deviations from the reference value based on the brightness prediction model can be adjusted.

[0030] As described above, the accepted brightness prediction model can be time-varying and can be defined for each event pixel, a set of event pixels, or the entire event sensor (i.e., each of the plurality of event pixels). The brightness prediction model includes reference values ​​to be used and optionally also includes thresholds to be used.

[0031] The following will refer to Figure 3 An embodiment of event pixel 20 is discussed. Figure 3 An embodiment is illustrated schematically with a block diagram.

[0032] Event pixel 20 is with Figure 2 The modified event pixels are compared to the regular event pixels of 10.

[0033] Event pixel 20 includes a reference value generator 21 that stores the received brightness prediction model.

[0034] Then, the reference value generator 21 adapts the reference value R over time using the stored brightness prediction model. Thresholds ε(+) and ε(-) can also be adapted and included in the brightness prediction model.

[0035] like Figure 3 As shown, the reference value can, for example, increase linearly with time. The measured brightness is currently within the boundaries at each time point, thus not triggering the generation of events. When the actual brightness evolution matches the predicted brightness evolution, no event is generated, and the reference value in the brightness prediction model is determined based on the predicted brightness evolution.

[0036] Returning to the general interpretation, it has been further recognized that, in order to predict the brightness measured by event pixels in the future, changes in the scene should be predicted.

[0037] It has been recognized that additional data processing equipment or processing units or prediction modules can be used to generate a brightness prediction model that takes into account information from at least two sources: event sensors—that is, the event stream of generated (i.e., previous) events; and information from another data source (e.g., from another sensor or from a downstream application).

[0038] Some embodiments relate to a data processing apparatus, wherein the data processing apparatus includes circuitry configured to: Receive event streams from event sensors and data streams from data sources, wherein the event streams include events generated based on detected changes in scene brightness, and wherein the data streams include sensor data or control data; Predict changes in the scenario based on this event stream and data stream; Based on the predicted changes and event flow of the scene, a brightness prediction model is generated for at least one event pixel; and The brightness prediction model is sent to the event sensor to adapt the reference value of the at least one event pixel according to the brightness prediction model.

[0039] Data processing devices can be embedded data processing modules, computers, servers, etc.

[0040] The circuit may be based on, may include, or may be implemented by typical electronic components configured to perform the functions described herein.

[0041] The circuit may be based on, may include, or may be implemented as integrated circuit logic, and its functions may be implemented by software executed by a processor or the like. The circuit may be based on, may include, or may be implemented by: CPU (Central Processing Unit), microcontroller, FPGA (Field Programmable Gate Array), ASIC (Application-Specific Integrated Circuit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), etc.

[0042] The circuit may be based on, may include, or may be implemented by typical electronic components and integrated circuit logic, and by software.

[0043] The circuit may include storage capabilities, such as magnetic storage, semiconductor storage, etc.

[0044] The circuit may include a data bus for transmitting and receiving data, and may implement the corresponding communication protocol.

[0045] The circuit may include one or more communication interfaces for communication via wired or wireless networks.

[0046] Therefore, some embodiments relate to a system comprising: An event sensor includes multiple event pixels, each event pixel being configured to detect brightness changes in the scene based on a reference value and a threshold to generate an event, and wherein the event sensor is configured to output an event stream; The data source is configured to provide a data stream, wherein the data stream includes sensor data or control data; A data processing device, including circuitry, configured to: Event streams from event sensors and data streams from data sources; Predict changes in the scenario based on this event stream and data stream; Based on the predicted changes in the scene and the event flow, a brightness prediction model is generated for at least one event pixel. The brightness prediction model is sent to the event sensor; and The event sensor is configured to receive a brightness prediction model for at least one event pixel from a data processing device, and to adapt a reference value for the at least one event pixel according to the received brightness prediction model.

[0047] The data source can be a sensor, such as an inertial measurement unit (“IMU”), a global navigation satellite system (“GNSS”), a LiDAR (“laser detection and ranging”) sensor, another event sensor, an image sensor, or a camera (e.g., an RGB (“red-green-blue” camera)).

[0048] Therefore, sensor data can be or may include velocity and acceleration data, depth data, event data, or image data.

[0049] The data source can be an application or a clock. The application can use event streams.

[0050] Applications can be control applications for vehicles (e.g., robots or cars) or surveillance cameras or traffic lights.

[0051] In some embodiments, the circuitry of the data processing device is configured to run an application.

[0052] Therefore, control data may or may include vehicle speed and acceleration data, or vehicle steering, braking, and acceleration commands. Control data may or may include rotation or zoom data from monitoring cameras. Control data may or may include commands for switching traffic lights.

[0053] Therefore, in some embodiments, the data source is an application that uses an event stream, and the data stream includes control data indicating the movement of the event sensor, and the circuitry is configured to provide the application with a brightness prediction model of at least one event pixel.

[0054] The system can be implemented as a distributed system, in which event sensors, data sources, and data processing devices are physically separated from or far apart from each other.

[0055] Event sensors and data sources can be implemented as devices.

[0056] The system (i.e., event sensors, data sources, and data processing devices) can be implemented as a device.

[0057] The device can be a camera (e.g., a surveillance camera), a vehicle (e.g., a robot or a car), a mobile electronic device (e.g., a smartphone), etc.

[0058] As described above, the circuit is configured to predict changes in the scene based on event streams and data streams.

[0059] In some embodiments, the circuit is configured to predict, based on the data stream, the relative movement of a static portion of the scene with respect to an event sensor as a change in the scene.

[0060] For example, control data from a vehicle control application can indicate the vehicle's speed, acceleration, and steering, enabling the circuitry to predict the relative movement of an event sensor with respect to a static part of the scene. Therefore, the circuitry can predict how much and in which direction the current image on the event sensor will move within a given time interval. Thus, the circuitry can predict for a specific event pixel that the current brightness of neighboring event pixels might exist on that specific event pixel in the future, after a given time interval.

[0061] In some embodiments, the circuit is configured to predict changes in scene illumination as changes in the scene based on at least one of an event stream and a data stream.

[0062] For example, events in an event stream can indicate the periodic generation of the same event sequence by one or more event pixels, such as traffic lights that are periodically turned on and off, and thus can predict changes in the scene's illumination.

[0063] In addition, control data may include traffic light switching commands, enabling predictable changes in scene illumination.

[0064] The circuit can predict the sun's movement based on time information received from the clock, thus enabling the prediction of changes in scene illumination.

[0065] As described above, the circuit is configured to generate a brightness prediction model for at least one event pixel based on scene prediction changes and event flow, and to send the brightness prediction model to the event sensor.

[0066] In some embodiments, the circuit is configured to determine the current reference value of each of a plurality of event pixels (specifically, at least one event pixel and one or more neighboring event pixels of at least one event pixel) based on an event stream.

[0067] In other words, in some embodiments, generating a brightness prediction model includes: determining a current reference value for each of a plurality of event pixels (specifically, at least one event pixel and one or more neighboring event pixels of at least one event pixel) based on an event stream.

[0068] The events of a stream generated by multiple event pixels (specifically, by at least one event pixel and one or more adjacent event pixels) may include the actual brightness value at the time the event is triggered.

[0069] In other embodiments, the circuit may sum all events relative to a reference point (e.g., the initial configuration of the reference value may be reported by the event sensor during initialization) to determine the current reference value for each of a plurality of event pixels (specifically, at least one event pixel and one or more adjacent event pixels of at least one event pixel).

[0070] In some embodiments, the brightness prediction model includes a time-related reference value to be used by at least one event pixel. In some embodiments, the brightness prediction model includes a time-related threshold to be used by at least one event pixel.

[0071] Given periodic or predictable variations in scene illuminance, as discussed above, the circuitry can determine low and high reference values ​​for event pixels detecting a periodic event sequence, and these reference values ​​can be used in a luminance prediction model. The low reference value can be used for when traffic lights are off, while the high reference value can be used for when traffic lights are on.

[0072] The current reference value can be used as the starting point for the brightness prediction model.

[0073] In the case of relative movement of the event sensor relative to a static part of the scene, as discussed above, the circuitry can predict for a specific event pixel that the current brightness of neighboring event pixels is likely to exist on that specific event pixel in the future after a specific time interval. The circuitry can then use the current reference values ​​of the neighboring event pixels of the specific event pixel as a target reference value that should be reached after the specific time interval. The change toward the target reference value can be linear or non-linear (e.g., spline or quadratic or cubic function).

[0074] In some embodiments, the brightness prediction model indicates a segmented constant reference value or a constant rate of change of the reference value.

[0075] In some embodiments, the brightness prediction model indicates a reference value that changes continuously from the current reference value to the target reference value, and the target reference value is based on the current reference value of at least one neighboring event pixel among at least one event pixel.

[0076] In some embodiments, the brightness prediction model indicates a list of target reference values ​​to be used sequentially, and each target reference value corresponds to the current reference value of a different neighboring event pixel.

[0077] In some embodiments, the luminance prediction model includes a sequence of luminance prediction models or an indication of periodic repetition of one or more luminance prediction models.

[0078] In some embodiments, the circuit is configured to update the luminance prediction model at each fixed time interval or when a preset time limit has elapsed or when an event stream indicates that the current luminance prediction model is no longer appropriate.

[0079] Some embodiments relate to a method, wherein the method includes: Receive an event stream from an event sensor and a data stream from a data source, wherein the event stream includes events generated based on detected changes in brightness of the scene, and wherein the data stream includes sensor data or control data; Predict changes in the scenario based on the event stream and data stream; Based on the predicted changes and event flow of the scene, a brightness prediction model is generated for at least one event pixel; and The brightness prediction model is sent to the event sensor to adapt the reference value of the at least one event pixel according to the brightness prediction model.

[0080] This method can be performed by a data processing device as described herein.

[0081] In some embodiments, when executed on a computer and / or processor, the methods described herein are also implemented as a computer program that causes the computer and / or processor to perform the methods. In some embodiments, a non-transitory computer-readable recording medium is also provided in which a computer program product, when executed by a processor (such as the processor described above), performs the methods described herein.

[0082] return Figure 4 The block diagram of an embodiment of system 30 discussed below is schematically shown.

[0083] System 30 includes an event sensor 31, a data processing device 32 for an operation prediction module 33, and a downstream module 34. The system also includes one or more sensors 35.

[0084] Event sensor 31 outputs an event stream to prediction module 33 and downstream module 34, which is used by applications that utilize the event stream.

[0085] Event sensor 31 includes, as referenced Figure 3 The discussion focuses on multiple event pixels. Each event pixel is configured to detect brightness changes in Scene 1 based on a reference value and a threshold to generate an event.

[0086] Therefore, the event stream includes events generated based on the brightness changes detected in Scene 1.

[0087] Downstream module 34 sends control data to prediction module 33, and one or more sensors 35 send sensor data to prediction module 33.

[0088] Therefore, the prediction module 33 receives event streams and data streams from data sources, wherein the data streams include sensor data or control data.

[0089] The prediction module 33 determines the current reference value of at least one event pixel based on the event stream.

[0090] In addition, the prediction module 33 predicts changes in scene 1 based on the event stream and data stream, and generates a brightness prediction model for at least one event pixel based on the predicted changes in the scene and the determined current reference value.

[0091] Then, the prediction module 33 sends the brightness prediction model to the event sensor 31.

[0092] The event sensor 31 receives a brightness prediction model of at least one event pixel from the prediction module 33, and adapts a reference value of at least one event pixel according to the received brightness prediction model.

[0093] When the brightness prediction model is generated and sent to the event sensor 31, the prediction module 33 may include information such as the model type, the affected pixels, and the timestamps from when they become valid until when they become valid.

[0094] Predictive models are effective within a reasonable timeframe, making the increased effort required to compute, transfer, and implement the model still more efficient than transferring events in their original form.

[0095] In the following examples, the process behind the changes is less dynamic than the induced changes in brightness values, and therefore can be well approximated by a model.

[0096] The following section discusses examples of brightness prediction models.

[0097] "Parameters" are values ​​that describe the detailed behavior of the model, and "Model" describes how time-related reference values ​​are generated. "Variations" are possible modifications and extensions to the basic version of the model. "Application Examples" describe possible scenarios where the described model might be useful.

[0098] Figure 5 An embodiment of the brightness prediction model 40 is schematically illustrated in the graph, which is discussed below.

[0099] During time period T, the event pixel uses the current reference value 41 for the event pixel targeted by the brightness prediction model. Then, a constant new reference value 42 is used.

[0100] Model Example 1: Segmented constant brightness.

[0101] Parameters: Time period T; New reference value P.

[0102] Model: Maintain the current reference value R for time period T, and then change the reference value to P.

[0103] Variant: Time period T indicates a point in time relative to a common reference point, such as when the sensor is started.

[0104] Application example: Moving an object through the scene and obscuring the background.

[0105] Figure 6 An embodiment of the brightness prediction model 50 is schematically illustrated in the graph, which is discussed below.

[0106] First, the event pixel can use the current reference value 51 of the event pixel targeted by the brightness prediction model. Then, at 52, the reference value will continuously change (e.g., increase) at a specific slope during the time period T until it reaches the target reference value 53 and is used by the event pixel.

[0107] Model Example 2: Constant Rate of Change.

[0108] Parameters: Initial reference value P; brightness slope S.

[0109] Model: Immediately set the reference value R to P, and then continuously update the reference value R over time with a rate of change S.

[0110] Variant: Limit the reference value change to a time period T; keep the reference value constant after the duration of the period T changes; multiply rather than add the rates of change.

[0111] Application example: Illuminance variation.

[0112] Figure 7 An embodiment of the brightness prediction model 60 is schematically illustrated in the graph, which is discussed below.

[0113] Brightness prediction model 60 is another example of a piecewise constant reference value used by event pixels.

[0114] First, the event pixel uses the current reference value 61 of the event pixel targeted by the brightness prediction model. Then, a constant new reference value 62 is used. Then, after a certain amount of time, the initial current reference value 61 is used again. Then, the constant new reference value 62 is used again.

[0115] The following section discusses application examples.

[0116] Application Example 1: Compensation for changes in illuminance.

[0117] Setup: A camera with an event sensor 31, with a fixed position and viewpoint; typically a static scene, but with varying illumination, such as controlled (active lighting) or predictable (sunlight).

[0118] Example applications: surveillance; automobiles, used to suppress hazard lights or turn signal indicators.

[0119] Objective: To suppress events caused by predictable changes in brightness; to preserve events associated with any other scene changes, such as moving people.

[0120] Implementation method: (For example, via prediction module 33) receive information about future lighting changes from active lighting control or from solar motion prediction or from sensors or from detectors of periodically repeating illuminance patterns (e.g., flashing lights).

[0121] For each event pixel, (e.g., by prediction module 33) a brightness prediction model is generated based on the reference image (at the reference illuminance), the predicted illuminance change is generated, and the event pixel behavior is modeled under the brightness change.

[0122] For each event pixel, (e.g., via prediction module 33) a periodic sequence of constant changing brightness prediction models (flickering lights) is generated as a reference value.

[0123] The brightness prediction model is sent (e.g., via prediction module 33) to event sensor 31 and downstream module 34.

[0124] The event pixel receives the brightness prediction model, and the reference value generator 13 (see [reference]). Figure 3 The reference value is continuously updated based on the brightness prediction model and timer information.

[0125] Downstream module 34 receives brightness prediction models and events.

[0126] Alternatively, Figure 6 The brightness prediction model 50 can be used in this application example.

[0127] Figure 8 An embodiment of the brightness prediction model 70 is schematically illustrated in the graph, which is discussed below.

[0128] For event pixel #1: First, event pixel #1 uses the current reference value 71 of the luminance prediction model. Then, event pixel #1 linearly changes its reference value towards a target reference value 72, which is based on the current reference value of its neighboring event pixel (here, event pixel #2), as indicated by the upper left arrow. Then, event pixel #1 linearly changes its reference value towards a target reference value 73, which is based on the current reference value of its neighboring event pixel (here, event pixel #3), as indicated by the upper right and middle left arrows.

[0129] Therefore, the brightness prediction model for event pixel #1 indicates a list of target reference values ​​used sequentially, where each target reference value corresponds to the current reference value of a different neighboring event pixel (here, event pixel #2 and then event pixel #3).

[0130] Similarly, for event pixel #2: First, event pixel #2 uses the current reference value 74 of the luminance prediction model for event pixel #2. Then, event pixel #2 linearly changes its reference value towards a target reference value 75, which is based on the current reference value of its neighboring event pixel (here, event pixel #3), as indicated by the upper left arrow. Next, event pixel #2 linearly changes its reference value towards a target reference value 76, which is based on the current reference value of its neighboring event pixel (here, event pixel #4), as indicated by the upper right and middle left arrows.

[0131] Therefore, the brightness prediction model for event pixel #2 indicates a list of target reference values ​​used sequentially, where each target reference value corresponds to the current reference value of a different neighboring event pixel (here, event pixel #3 and then event pixel #4).

[0132] Similarly, for event pixel #3: First, event pixel #3 uses the current reference value 77 of the luminance prediction model for event pixel #3. Then, event pixel #3 linearly changes its reference value towards a target reference value 78, which is based on the current reference value of its neighboring event pixel (here, event pixel #4), as indicated by the upper left arrow. Next, event pixel #3 linearly changes its reference value towards a target reference value 79, which is based on the current reference value of its neighboring event pixel (here, event pixel #5), as indicated by the upper left arrow.

[0133] Therefore, the brightness prediction model for event pixel #3 indicates a list of target reference values ​​to be used sequentially, where each target reference value corresponds to the current reference value of a different neighboring event pixel (here, event pixel #4 and then event pixel #5).

[0134] The model example will be discussed below.

[0135] Model Example 3: Pixel Transmission.

[0136] Parameters: Address of another pixel on event sensor 31, ADR, duration T.

[0137] Model: Immediately copy the current reference R_ADR of the neighboring pixel indicated by the ADR—this is the target reference value. Then, start with the current reference value of the event pixel. Then, linearly and continuously transfer the reference value to the target reference value over time for a duration T. Then, repeat the same process for other reference values.

[0138] Variant: The transfer curve from the current reference value of an event pixel to the target reference value is non-linear, for example, a quadratic or cubic function. The actual brightness value is used instead of the reference values ​​of neighboring event pixels. As mentioned above, a list of event pixels is used instead of a single event pixel, and the target reference value is defined based on a weighted combination of the reference values ​​of neighboring event pixels.

[0139] Application examples will be discussed below.

[0140] Application Example 2: Camera rotation compensation.

[0141] Setup: A camera with event sensor 31 rotates around a fixed pivot point, for example, through controlled articulated motion. Typically, a static scene is assumed.

[0142] Example application: monitoring.

[0143] Objective: To suppress events caused by camera rotation and viewing direction; to preserve events that are relevant to any other scene changes, such as moving objects.

[0144] Implementation method: (For example, through prediction module 33) receive information about future camera movement, such as from camera rotation control.

[0145] For each event pixel, the apparent motion of the projected scene point is calculated as a two-dimensional velocity vector in the image plane based on the camera rotation direction and speed, the camera projection model, and the event pixel position on the event sensor 31 (e.g., by the prediction module 33).

[0146] For each pixel, a brightness prediction model is generated (e.g., via prediction module 33).

[0147] For example, a pixel transport model where the apparent motion direction indicates the direction of neighboring pixels, and the apparent motion speed indicates the duration of the change. This is based on a constant rate of change sequence from a reference image and its apparent projection onto an event sensor based on predicted camera motion.

[0148] The event pixel receives a brightness prediction model and continuously updates the reference value based on the prediction model.

[0149] Downstream module 34 receives: predicted camera motion; pixel-by-pixel predicted motion; pixel-by-pixel brightness prediction model; events.

[0150] Figure 9A and Figure 9B An embodiment of a method for generating a brightness prediction model is illustrated schematically, and is discussed below, wherein, Figure 9A A block diagram schematically illustrating an embodiment of the matrix of event pixels 80a and... Figure 9B Figure 90a schematically illustrates an embodiment of future time-dependent brightness associated with an event pixel.

[0151] refer to Figure 9A The matrix of event pixels includes event pixels 81-1 whose brightness prediction model is determined by prediction module 33.

[0152] The prediction module 33 predicts changes in the scene caused by the relative movement of the event sensor 31 relative to static parts of the scene, causing the scene image to move along the matrix of event pixels. Figure 9A In the image, the relative movement of the event sensor 31 is indicated by arrow 82, so that the image of the scene moves in the opposite direction to that indicated by arrow 82.

[0153] Adjacent event pixels 81-2 and 81-3 are arranged along the relative movement direction of event sensor 31, so that they can be used to predict the future brightness value of event pixel 81-1, where circle 83 represents the starting point and circle 84 represents the future point after a specific time interval.

[0154] Now for reference Figure 9B Line graph 90a shows the actual brightness evolution of future event pixel 81-1, represented by the dashed line.

[0155] Prediction module 33 now uses the current reference value of event pixel 81-1 as the starting point 93 for brightness prediction, and the weighted combination of the current reference values ​​of event pixels 81-2 and 81-3 as the ending point 94 for brightness prediction.

[0156] The prediction module 33 performs linear interpolation between the starting point 93 and the ending point 94 to obtain the slope of the linear prediction 92a.

[0157] The prediction module 33 generates a brightness prediction model based on linear prediction 92a, starting point 93 and ending point 94, such that the brightness prediction model includes the current reference value of event pixel 81-1 and the target reference value of ending point 94, which corresponds to a weighted combination of the current reference values ​​of event pixels 81-2 and 81-3 and the slope of linear prediction 92a.

[0158] The following section discusses application examples.

[0159] Application Example 3: Compensation for Complex Vehicle / Camera Motion

[0160] Setup: A camera with event sensor 31 is mounted on a vehicle moving through a scene (typically a static scene).

[0161] Example applications: driver assistance, autonomous driving, robotics, visual navigation, UAV (“unmanned aerial vehicle”).

[0162] Objective: To suppress events caused by the movement of one's own camera / vehicle; to preserve events associated with any other changes, such as moving objects.

[0163] Implementation method: (For example, via prediction module 33) receive information about the current camera motion (instantaneous 3D translation and rotation velocities), such as from IMU, GNSS, wheel odometer, visual / LiDAR / radar odometer, or event-based camera odometer. Update (e.g., via prediction module 33) the camera motion model for 3D translation and rotation velocities, accelerations, jerkiness, or possibly more attributes, such as unscented / extended Kalman filters or Bayesian recursive filters.

[0164] Predict the camera trajectory in the near future based on filter state prediction (e.g., by prediction module 33).

[0165] For each event pixel, information about the distance to the projected scene point (e.g., measured from LiDAR) is received (e.g., via prediction module 33), and a motion structure method is applied to the camera data, and / or to the event camera data.

[0166] For each event pixel, the apparent motion of the projected scene point is calculated as a two-dimensional velocity vector in the image plane based on the predicted camera trajectory, camera projection model, event pixel distance, and event pixel position on the event sensor 31 (e.g., by the prediction module 33).

[0167] For each pixel, a brightness prediction model is generated (e.g., via prediction module 33).

[0168] For example, a pixel transport model where the apparent motion direction indicates the direction of neighboring pixels, and the apparent motion speed indicates the time period of change. This is based on predicted camera motion, based on the change sequence of a reference image and its apparent projection onto event sensor 31.

[0169] The event pixel receiving model continuously updates the reference value based on the prediction model.

[0170] Downstream module 34 receives: predicted camera motion; pixel-by-pixel predicted motion; pixel-by-pixel brightness prediction model; events.

[0171] Variant: Uses a less detailed motion prediction model that directly determines the apparent motion in the image plane, for example, based on previous measurements (optical flow), and therefore does not require depth information.

[0172] Figure 10A and Figure 10B An embodiment of a method for generating a brightness prediction model is illustrated schematically, and is discussed below, wherein, Figure 10A A block diagram schematically illustrating an embodiment of the matrix of event pixels 80b and... Figure 10B Figure 90b schematically illustrates an embodiment of future time-dependent brightness associated with an event pixel.

[0173] The implementation example is similar to Figure 9A and Figure 9B The embodiment is shown. However, the difference lies in the use of nonlinear prediction 92b.

[0174] Endpoint 96 is the current reference value based on event pixels 81-8 of circle 86 along the direction of movement.

[0175] Nonlinear prediction 92b is now based on two additional sampling points along arrow 82.

[0176] Sampling point 94 is based on the current reference values ​​of event pixels 81-2 and 81-3, which are consistent with circle 84 along the movement direction.

[0177] Sampling point 95 is based on the current reference values ​​of event pixels 81-4, 81-5, 81-6, and 81-7, which are consistent with the circle 85 along the direction of movement.

[0178] Then, prediction module 33 generates a brightness prediction model based on nonlinear prediction 92b, starting point 93, and ending point 96.

[0179] Figure 11 A flowchart illustrating an embodiment of method 100 discussed below is shown schematically.

[0180] Method 100 can be performed by a data processing device as described herein.

[0181] At 101, an event stream from an event sensor and a data stream from a data source are received, wherein the event stream includes events generated based on detected changes in brightness of the scene, and wherein the data stream includes sensor data or control data, as discussed herein.

[0182] In 102, changes in the scenario are predicted based on event streams and data streams, as discussed in this paper.

[0183] In 103, a brightness prediction model for at least one event pixel is generated based on the predicted scene changes and event flow, as discussed in this paper.

[0184] At 104, the brightness prediction model is sent to the event sensor for use in adapting the reference value of the at least one event pixel to the brightness prediction model, as discussed herein.

[0185] Returning to the general explanation, the following outlines some aspects of some embodiments.

[0186] A modified event sensor is provided, which receives a model from an external source for the expected future brightness value in each event pixel.

[0187] Additional data processing equipment or processing unit or prediction module that considers information from previous event camera events, more than one event pixel (e.g., the event pixel neighborhood or all event pixels), information from additional sensors (e.g., IMU, GNSS, other event cameras, grayscale or RGB cameras), and / or information from downstream processing stages (e.g., visual odometry, self-motion estimation, motion structure) to analyze the current situation and derive a brightness prediction model for a single event pixel, a group of event pixels, or all event pixels from the event sensor.

[0188] The brightness prediction model is provided to the event sensor to implement a new model or replace some or all of the previous models.

[0189] The brightness prediction model can be provided to any downstream processing block (possibly in a generalized form).

[0190] When the model reaches a preset time limit, that is, when the validity period of the model's prediction is exceeded and / or when an event indicates that the model is no longer suitable for describing changes in the measurement, the brightness prediction model can be updated by a timer at fixed intervals.

[0191] Predictive models need to be designed to match the application scenario and the sensor interface.

[0192] For example, in application scenarios where a large number of changes can be predicted and measured, at least one of the following can be implemented and / or provided: Events are filtered based on importance. Predictable measurements are not informative, and related events are filtered out. Unpredictable events are informative and are retained in the event stream.

[0193] The complexity requirements of downstream modules are reduced because they are not overwhelmed by uninformative events, but can instead focus only on important events.

[0194] Compression: Measurement changes described by the filtered events are not lost to downstream modules but are summarized in the brightness prediction model. The brightness prediction model provided to downstream modules summarizes events over time and thus reduces data transmission. The brightness prediction model can be used as a preprocessing step in downstream modules.

[0195] The total number of triggered events has decreased.

[0196] It should be recognized that the embodiments describe a method having an exemplary order of method steps. However, the specific order of method steps is given for illustrative purposes only and should not be construed as a constraint.

[0197] Unless otherwise stated, all units and entities described in this specification and claimed in the appended claims may be implemented as integrated circuit logic, for example, on a chip, and unless otherwise stated, the functionality provided by these units and entities may be implemented by software.

[0198] With regard to the implementation of the embodiments of the present disclosure at least in part using a software-controlled data processing apparatus, it should be understood that providing such a software-controlled computer program and the transmission, storage or other medium through which such a computer program is provided are conceived as aspects of the present disclosure.

[0199] It should be noted that this technology can also be configured as follows.

[0200] (1) A system comprising: An event sensor includes multiple event pixels, each event pixel being configured to detect brightness changes in the scene based on a reference value and a threshold to generate an event, and wherein the event sensor is configured to output an event stream; The data source is configured to provide a data stream, wherein the data stream includes sensor data or control data; A data processing device, including circuitry, configured to: Receive event streams from event sensors and data streams from data sources; Predict changes in the scenario based on this event stream and data stream; Based on the predicted changes and event flow of the scene, a brightness prediction model is generated for at least one event pixel. The brightness prediction model is sent to the event sensor; and The event sensor is configured to receive a brightness prediction model for at least one event pixel from a data processing device, and to adapt a reference value for the at least one event pixel according to the received brightness prediction model.

[0201] (2) The system according to (1), wherein the circuit is configured to predict, based on the data stream, the relative movement of the static part of the scene with respect to the event sensor as a change in the scene.

[0202] (3) The system according to (1) or (2), wherein the circuit is configured to predict the illuminance change of the scene as a change of the scene based on at least one of an event stream and a data stream.

[0203] (4) The system according to any one of (1) to (3), wherein the brightness prediction model includes a time-related reference value to be used by at least one event pixel.

[0204] (5) The system according to (4), wherein the brightness prediction model indicates a segmented constant reference value or a constant rate of change of the reference value.

[0205] (6) The system according to (4), wherein the brightness prediction model indicates a reference value that changes continuously from a current reference value to a target reference value, and wherein the target reference value is based on the current reference value of at least one neighboring event pixel among at least one event pixel.

[0206] (7) The system according to (6), wherein the brightness prediction model indicates a list of target reference values ​​to be used sequentially, and wherein each target reference value corresponds to the current reference value of a different neighboring event pixel.

[0207] (8) The system according to any one of (1) to (7), wherein the brightness prediction model includes a sequence of brightness prediction models or an indication of periodic repetition of one or more brightness prediction models.

[0208] (9) The system according to any one of (1) to (8), wherein the circuit is configured to update the luminance prediction model at each fixed time interval or when a preset time limit has passed or when an event stream indicates that the current luminance prediction model is no longer appropriate.

[0209] (10) The system according to any one of (1) to (9), wherein the data source is an application using an event stream, the data stream including control data indicating the movement of the event sensor, and wherein the circuit is configured to provide the application with a brightness prediction model of at least one event pixel.

[0210] (11) The system according to any one of (1) to (10), wherein the system is implemented as an apparatus.

[0211] (12) An event sensor, wherein the event sensor comprises: Multiple event pixels, wherein each event pixel is configured to detect changes in scene brightness based on a reference value and a threshold to generate an event; The circuit is configured as follows: Output event stream; Receive a brightness prediction model for at least one event pixel; and The reference value of the at least one event pixel is adapted according to the received brightness prediction model.

[0212] (13) A data processing apparatus, wherein the data processing apparatus includes a circuit configured to: Receive event streams from event sensors and data streams from data sources, wherein the event streams include events generated based on detected changes in scene brightness, and wherein the data streams include sensor data or control data; Predict changes in the scenario based on this event stream and data stream; Based on the predicted changes and event flow of the scene, a brightness prediction model is generated for at least one event pixel; and The brightness prediction model is sent to the event sensor to adapt the reference value of the at least one event pixel according to the brightness prediction model.

[0213] (14) A method, wherein the method comprises: Receive an event stream from an event sensor and a data stream from a data source, wherein the event stream includes events generated based on detected changes in brightness of the scene, and wherein the data stream includes sensor data or control data; Predict changes in the scenario based on the event stream and data stream; Based on the predicted changes and event flow of the scene, a brightness prediction model is generated for at least one event pixel; and The brightness prediction model is sent to the event sensor to adapt the reference value of the at least one event pixel according to the brightness prediction model.

[0214] (15) The method according to (14) includes: predicting the relative movement of a static part of the scene relative to an event sensor as a change in the scene based on a data stream.

[0215] (16) The method according to (14) or (15) includes: predicting the illuminance change of the scene as a change of the scene based on at least one of the event stream and the data stream.

[0216] (17) The method according to any one of (14) to (16), wherein the brightness prediction model includes a time-related reference value to be used by at least one event pixel.

[0217] (18) According to the method of (17), wherein the brightness prediction model indicates a segmented constant reference value or a constant rate of change of the reference value.

[0218] (19) The method according to (17), wherein the brightness prediction model indicates a reference value that changes continuously from the current reference value to the target reference value, and wherein the target reference value is based on the current reference value of at least one neighboring event pixel among at least one event pixel.

[0219] (20) According to the method of (19), wherein the brightness prediction model indicates a list of target reference values ​​to be used sequentially, and wherein each target reference value corresponds to the current reference value of a different neighboring event pixel.

[0220] (21) A computer program comprising program code that, when executed on a computer, causes the computer to perform the method according to any one of (14) to (20).

[0221] (22) A non-transitory computer-readable recording medium storing a computer program product which, when executed by a processor, causes the method according to any one of (14) to (20) to be performed.

Claims

1. A system for data processing, comprising: An event sensor includes multiple event pixels, wherein each event pixel is configured to detect brightness changes in a scene based on a reference value and a threshold to generate an event, and wherein the event sensor is configured to output an event stream; A data source, configured to provide a data stream, wherein the data stream includes sensor data or control data; and A data processing device, including circuitry, the circuitry being configured to: Receive the event stream from the event sensor and the data stream from the data source; Predict changes in the scenario based on the event stream and the data stream; Based on the predicted changes in the scene and the event flow, a brightness prediction model is generated for at least one event pixel. The brightness prediction model is sent to the event sensor; and The event sensor is configured to receive the brightness prediction model for the at least one event pixel from the data processing device, and to adapt the reference value of the at least one event pixel according to the received brightness prediction model.

2. The system according to claim 1, wherein, The circuit is configured to predict, based on the data stream, the relative movement of a static portion of the scene relative to the event sensor as a change in the scene.

3. The system according to claim 1, wherein, The circuit is configured to predict changes in illuminance of the scene as changes in the scene based on at least one of the event stream and the data stream.

4. The system of claim 1, wherein, The brightness prediction model includes time-related reference values ​​used by the at least one event pixel.

5. The system of claim 4, wherein, The brightness prediction model indicates a segmented constant reference value or a constant rate of change of the reference value.

6. The system of claim 4, wherein, The brightness prediction model indicates a reference value that changes continuously from the current reference value to the target reference value, wherein the target reference value is based on the current reference value of at least one neighboring event pixel among the at least one event pixel.

7. The system of claim 6, wherein, The brightness prediction model indicates a list of target reference values ​​to be used sequentially, wherein each target reference value corresponds to the current reference value of a different neighboring event pixel.

8. The system of claim 1, wherein, The brightness prediction model includes a sequence of brightness prediction models or an indication of the periodic repetition of one or more brightness prediction models.

9. The system of claim 1, wherein, The circuit is configured to update the brightness prediction model at each fixed time interval, or when a preset time limit has passed, or when the event stream indicates that the current brightness prediction model is no longer suitable.

10. The system according to claim 1, wherein, The data source is an application using the event stream, the data stream including control data indicating the movement of the event sensor, and wherein the circuitry is configured to provide the application with the brightness prediction model for the at least one event pixel.

11. The system of claim 1, wherein, The system is implemented as a device.

12. An event sensor, comprising: Multiple event pixels, wherein each event pixel is configured to generate an event by detecting brightness changes in the scene based on a reference value and a threshold; and The circuit is configured as follows: Output event stream; Receive a brightness prediction model for at least one event pixel; and The reference value of the at least one event pixel is adapted according to the received brightness prediction model.

13. A data processing apparatus, comprising a circuit, the circuit being configured to: receiving an event stream from an event sensor and a data stream from a data source, wherein, The event stream includes events generated based on detected changes in scene brightness, and the data stream includes sensor data or control data. Predict changes in the scenario based on the event stream and the data stream; Based on the predicted changes in the scene and the event flow, a brightness prediction model is generated for at least one event pixel; and The brightness prediction model is sent to the event sensor to adapt the reference value of the at least one event pixel according to the brightness prediction model.

14. A method for data processing, comprising: Receive an event stream from an event sensor and a data stream from a data source, wherein the event stream includes events generated based on detected changes in brightness of the scene, and wherein the data stream includes sensor data or control data; Predict changes in the scenario based on the event stream and the data stream; Based on the predicted changes in the scene and the event flow, a brightness prediction model is generated for at least one event pixel; and The brightness prediction model is sent to the event sensor to adapt the reference value of the at least one event pixel according to the brightness prediction model.

15. The method of claim 14, comprising: Based on the data stream, the relative movement of the static portion of the scene with respect to the event sensor is predicted as a change in the scene.

16. The method of claim 14, comprising: Based on at least one of the event stream and the data stream, the change in illuminance of the scene is predicted as a change in the scene.

17. The method of claim 14, wherein, The brightness prediction model includes time-related reference values ​​used by the at least one event pixel.

18. The method of claim 17, wherein, The brightness prediction model indicates a segmented constant reference value or a constant rate of change of the reference value.

19. The method of claim 17, wherein, The brightness prediction model indicates a reference value that changes continuously from the current reference value to the target reference value, wherein the target reference value is based on the current reference value of at least one neighboring event pixel among the at least one event pixel.

20. The method of claim 19, wherein, The brightness prediction model indicates a list of target reference values ​​to be used sequentially, wherein each target reference value corresponds to the current reference value of a different neighboring event pixel.

Images