Method for transmitting data originating from an event-based image sensor

Abstract

The present invention relates to a method for transmitting data originating from an event-based image sensor (1) via a data channel (4). In this process, a camera stream (2) from the event-based image sensor (1) is converted and compressed into a bit stream (11) using event prediction (7) and delta coding (8) as well as subsequent entropy coding (10), and the bit stream (11) is transmitted via the data channel (4) to a receiver side (5).

Description

[0001] The present invention relates to a method for transmitting data originating from an event-based image sensor and containing events.

[0002] Event-based image sensors, also known as "event cameras," are used in a wide variety of applications. The transmission of data from the event-based image sensor to an application can encounter limitations in the capacity of a data channel.

[0003] US patent 2025 / 0078508 A1 discloses a method for compressing a data stream originating from an event camera. In this process, the data stream is compressed and stored in a memory.

[0004] The present invention addresses the problem of specifying improved or at least other embodiments for a method for transmitting a data stream from an event-based image sensor.

[0005] This problem is solved according to the invention by the subject matter of independent claim 1. Advantageous embodiments are the subject matter of the dependent claims.

[0006] The present invention is therefore based on the basic idea of ​​predicting events from a data stream of an event-based image sensor and generating a coded and compressed data stream from the data stream and the predicted events by means of delta coding and subsequent entropy coding and transmitting this data stream to a receiver, wherein the receiver generates a decoded data stream by applying the analog coding measures and makes it available to an application, in particular in a vehicle.

[0007] This enables real-time compression and transmission of the event-based image sensor's data stream, while simultaneously reducing bandwidth requirements. This results in both improved utilization of the data from the event-based image sensor and reduced resource consumption. Entropy coding and delta coding are the lossless components of the framework. This ensures that the meta-properties of the event-based image sensor's data stream are largely preserved, allowing the transmitted data to be enhanced and evaluated with higher quality in the associated application.

[0008] According to the inventive method, the event-based image sensor provides event-containing data that generates a data stream. This data stream is hereinafter also referred to as the camera stream. The camera stream thus originates from the event-based image sensor and contains the events supplied by the image sensor. The event-based image sensor is also commonly known as an "event camera" and will hereinafter be referred to as such. A data stream containing events predicted by the event prediction is generated from the camera stream. The event prediction is hereinafter also referred to as the transmitter event prediction, and the data stream as the event prediction stream. Furthermore, a data stream is generated from the camera stream and the event prediction stream by means of delta coding, which is hereinafter also referred to as the transmitter residual stream. The transmitter residual stream is converted into a bit stream by means of entropy coding.The aforementioned measures are performed at the sender's end. The bitstream is transmitted via a data channel to a receiver. The receiver converts the bitstream into a data stream using entropy decoding; this is subsequently referred to as the receiver residual stream. From the receiver residual stream, a data stream is generated using event prediction and delta decoding, so that the data stream contains the predicted events from the sender's event prediction as well as events from the camera stream. This data stream is subsequently referred to as the user stream. The user stream can then be provided to an application.

[0009] The procedure is expediently implemented using computers.

[0010] The method can be used in any field.

[0011] For example, the method can be used in a vehicle. The event camera could be used to monitor the vehicle's surroundings, particularly in a vehicle assistance system, for example, for at least partially autonomous driving.

[0012] The event prediction performed by the receiver corresponds to the event prediction by the sender.

[0013] It is conceivable to generate the event prediction stream directly from the camera stream. This means that the event prediction stream is generated directly from the camera stream using the transmitter's event prediction. This results in a transmission to the receiver that is at least essentially lossless.

[0014] It is also conceivable to filter the camera stream before generating the event prediction stream, thus converting it into a filtered data stream. This filtered stream is then converted into the transmitter's residual stream using event prediction and delta coding. Irrelevance reduction is employed for this purpose. This means that the camera stream is converted into the filtered data stream before generating the event prediction stream, and the filtered data stream is then converted into the transmitter's residual stream using event prediction and delta coding. The filtered data stream, and therefore also the transmitter's residual stream, thus contains less data. As a result, a reduced bandwidth is required in the data channel for transmitting the bit stream. In this way, the resources required for transmission are further reduced.Since event prediction, delta coding, and entropy coding are lossless, precise lossy compression adapted to the respective application can also be achieved through irrelevance reduction.

[0015] The camera stream is preferably transmitted in an event-standard format. Preferably, this involves an event e. i , hereinafter also referred to as "event", by the tuple {x i , yi, p i , t i} or another arrangement of the same fields. Here, x and y describe the position of the pixel at which a light intensity threshold was exceeded and / or fallen below, p describes the polarity of the event, where, for example, 0 can correspond to a negative change in intensity and 1 to a positive change in intensity. t describes the timestamp assigned to the event. This is advantageously necessary for processing events on synchronous computing systems.

[0016] In an advantageous embodiment, the procedure can also be carried out entirely asynchronously, thereby eliminating the need for the timestamp information.

[0017] The respective event prediction can be carried out, for example, as a so-called "Same-Event Prediction", a so-called "Optical-Flow-based Prediction" and a so-called "Intensity-Estimation Prediction".

[0018] In "same-event prediction," the next event predicted is the last event that occurred. This very simple approach takes advantage of the fact that an event source often generates an event more than once and / or a second event from the same source will occur spatiotemporally close to the first event. Even if the event does not occur exactly the same way again, this results in a smaller delta, which can then be transmitted more efficiently. Furthermore, this potentially reduces the standard deviation of the probability distribution of the information to be transmitted, since the entire pixel space only needs to be considered as the value space in the worst-case scenario. In better cases, small delta values ​​are the norm and can thus be encoded more effectively. Depending on the architecture and software of the event-based image sensor, a readout method is also used, which reads the events line by line.This already reduces redundancies in the coordinates, since the y-coordinate does not change from event to event, resulting in a delta of 0.

[0019] In "optical flow-based prediction," the next event is predicted based on the optical flow of the receptive field of the event camera. This method exploits the fact that the world is based on physical relationships. The movement of an object continues and does not disappear. Therefore, the movement can be estimated. Particularly in a vehicle context, a large proportion of events in the moving scene are also generated by stationary objects. Since these events result from the recorded scene and the displacement caused by the movement of the ego-vehicle, they are redundant and can therefore also be predicted.

[0020] In an advantageous implementation, "Optical-Flow-based Prediction" can be supplemented by other data, such as data from an inertial measurement unit, to improve the determination of the optical flow.

[0021] In "Intensity Estimation Prediction," the light intensity that will affect the sensor in the future is estimated. This, in turn, determines the next events. For example, the so-called "Kalman filter" can be used, as it can combine various information sources, such as past events, motion estimations like optical flow, and potentially data from an inertial measurement unit, and provide a continuous estimation of the intensity values ​​for all pixels.

[0022] Alternatively or additionally, a machine learning model can be used for event prediction. This model receives the current / recent light intensities as input and predicts the next light intensities as output. Such a model can be trained, particularly unsupervised, since the labels are derived from the light intensity in the next time step. Here, "continual learning" methods are advantageously used to continuously improve the model even while the method is in operation.

[0023] Entropy coding can be done arbitrarily.

[0024] For example, entropy coding can include or correspond to the so-called "Huffman coding" and / or the so-called "arithmetic coding".

[0025] Irrelevance reduction filters out events that are not, or are unlikely to be, useful for the subsequent application and are therefore irrelevant. This reduces the absolute number of events and consequently the corresponding data rate.

[0026] Such irrelevant events can be noise and / or so-called "hot pixels," which generate an event without any external influence. Hot pixels, due to their lower threshold, generate more events than a "normal" pixel. Therefore, external factors are still required. The information generated by such a pixel is redundant, which is similar to noise.

[0027] As explained above, the reduction of irrelevance is located on the sender's side.

[0028] However, it is also conceivable to implement an analogous irrelevance reduction on the receiving side. This would allow for lower hardware utilization on the transmitting side.

[0029] Preferably, the filtering strength for irrelevance reduction is adjustable. This adjustment is preferably dependent on the available bandwidth of the data channel. With higher available bandwidth, the filtering strength can be reduced, and / or with lower available bandwidth, the filtering strength can be increased. This results in optimal utilization of the available bandwidth and improved quality of the user data stream provided to the application.

[0030] Irrelevance reduction can, for example, be carried out at least partially using a so-called "spiking neural network".

[0031] Alternatively or additionally, irrelevance reduction is achieved, for example, by means of so-called "Density-Driven Random Dropping" and / or by means of so-called "Fourier-based Event Filtering" and / or by means of so-called "Bio-Inspired Sensitivity Control".

[0032] In "Density-Driven Random Dropping," a probability is defined at the pixel level that indicates the likelihood of an event occurring at that point being filtered out and not processed further. This method is based on the assumption that the relevance of a single event is related to the activity in the surrounding spatiotemporal space. Specifically, a single event provides little information about a process in the receptive field, since a movement or other relevant process would generate more than one event. Therefore, it can be assumed that the corresponding event is merely noise or irrelevant to the downstream application. Conversely, if many events occur in a spatiotemporal neighborhood, a single event carries little weight, as only the semantic structure of the movement or the spatiotemporal neighborhood needs to be preserved.This allows a certain portion of these events to be filtered out randomly.

[0033] Fourier-based event filtering removes events in the frequency domain. The motivation for this approach is the flickering characteristic of artificial light, which, due to pulse-width modulation, typically flickers at a similar, usually constant frequency, thus generating a large number of redundant events. Specifically, an adaptive notch and / or bandpass filter could be used here.

[0034] Another possibility is certain complex visual patterns which, due to their nature, trigger many events. These could also be detected and filtered.

[0035] Bio-Inspired Sensitivity Control draws inspiration from human visual perception, which exhibits high resolution in the center of the receptive field while lower resolution in the periphery. This principle is applied to event-based visual perception by defining one or more attention fields with high spatial and / or temporal resolution. For example, in a vehicle's environment, one attention field could be placed on the region representing the destination of the current trajectory, and additional fields could be placed on regions with high event activity. These could include pedestrians or other road users that must be clearly observable by the downstream application. Thus, events are implicitly filtered by prioritizing those occurring in peripheral areas over those generated within attention fields.

[0036] Alternatively or additionally, so-called "Application-Based Irrelevance Reduction" (ABR) can be used for irrelevance reduction. Similar to MP3 compression, this removes data that is irrelevant to the application. In MP3s, there are sounds that are inaudible to humans. Similarly, there is data that is irrelevant to the application. This can be achieved through AI-based irrelevance reduction, which is trained to compress as much data as possible without degrading application performance. To reduce the complexity of the AI-based filter, it is advantageously used as a control element for the parameters of the aforementioned irrelevance methods. This allows for real-time processing in optimized hardware units, and the control of individual parameters can be AI-optimized without having to meet the strict timing requirements of real-time event processing.

[0037] Preferably, to reduce irrelevance, the aforementioned methods are fine-tuned automatically and / or manually, so that the performance of the application with and without irrelevance reduction is as indistinguishable as possible.

[0038] Further important features and advantages of the invention will become apparent from the dependent claims, the drawings and the associated description of the figures based on the drawings.

[0039] It is understood that the features mentioned above and those to be explained below can be used not only in the combination specified, but also in other combinations or on their own, without leaving the scope of the present invention.

[0040] Preferred embodiments of the invention are shown in the drawings and are explained in more detail in the following description, wherein identical reference numerals refer to identical or similar or functionally identical components.

[0041] The single figure shows: Fig. 1. A flowchart to explain a procedure for transferring data from an event-based image sensor.

[0042] In a process that is suitably implemented by computer, an event-based image sensor 1, hereinafter also referred to as the "event camera" 1, provides data during operation from which a data stream 2 is generated. The data, and thus the data stream, contains events detected by the event camera, the so-called "events". The data stream 2 is hereinafter also referred to as the camera stream 2. The event camera 1 is located on a transmitter 3. The transmitter 3 transmits data via a data channel 4 to a receiver 5, which comprises an application 6 or provides data for this application 6.

[0043] The method can be used in a vehicle 100. The event camera 1 can be part of a driver assistance system 101 of the vehicle 100 (not shown), which supports a driver (not shown) while driving the vehicle 100 and / or by means of which the vehicle 100 drives at least partially autonomously.

[0044] The bandwidth available for this data transmission on data channel 4 may be limited, or the total bandwidth of data channel 4 may be insufficient for the data transmission. To achieve reliable data transmission despite the reduced or insufficient bandwidth of data channel 4, the following procedure is used.

[0045] How Fig. As can be seen from Figure 1, an unseen data stream is generated on the transmitter side 3 from the camera stream 2 by means of an event prediction 7. The event prediction 7 is subsequently also referred to as transmitter event prediction 7 and the data stream as event prediction stream.

[0046] Furthermore, a data stream 9 is generated from the camera stream 2 and the event prediction stream using delta coding 8; this data stream is subsequently also referred to as the transmitter residual stream 9. In the representation of the Fig. Figure 1 shows the transmitter event prediction 7 and the delta coding 4 together.

[0047] The residual transmitter stream 9 is converted into a bit stream 11 using entropy coding 10. The entropy coding 10 can include or correspond to the so-called "Huffman coding" and / or the so-called "arithmetic coding".

[0048] Bitstream 11 is transmitted to receiver 5 via data channel 4.

[0049] On the receiver side 5, the bitstream 11 is converted into a data stream 13 by means of entropy decoding 12, which is subsequently also referred to as the receiver residual stream 13. From the receiver residual stream 13, a data stream 16 is generated by means of event prediction 14 and delta decoding 15, which contains the predicted events of the transmitter event prediction 7 as well as events of the camera stream 2. Analogous to the representation on the transmitter side 3, the event prediction 14 and the delta decoding 15 are shown together on the receiver side 5. The data stream 16 is subsequently also referred to as the user stream 16. The user stream 16 is provided to the application 6.

[0050] Event prediction 14 can correspond to transmitter event prediction 7. The respective event predictions 7 and 14 can be executed, for example, using a so-called "Same-Event Prediction" and / or a so-called "Optical-Flow-based Prediction" and / or a so-called "Intensity-Estimation Prediction".

[0051] Camera stream 2 can be processed unfiltered using transmitter event prediction 7 and delta decoding. This means that the event prediction stream is generated directly from camera stream 2 using transmitter event prediction 7.

[0052] As in Fig. As shown in Figure 1, the camera stream 2 can also be subjected to filtering by means of irrelevance reduction 17 before further processing. This means that the camera stream 2 is converted into a filtered data stream 18 by means of irrelevance reduction 17 before the event prediction stream is generated. The filtered data stream 18 is then converted into the residual transmitter stream 9 by means of transmitter event prediction 7 and delta coding 8.

[0053] Irrelevance reduction 17 can be implemented, at least partially, using a so-called "spiking neural network." Similarly, irrelevance reduction can be implemented, for example, using so-called "density-driven random dropping," "Fourier-based event filtering," and / or "bio-inspired sensitivity control."

[0054] The strength of the irrelevance reduction filtering 15 is preferably adjustable, wherein the strength of the filtering is set depending on the available bandwidth of the data channel 4, as in Fig. 1 is indicated by an arrow leading back to irrelevance reduction 15. The strength of the filtering is expediently increased when the available bandwidth of data channel 4 is reduced and reduced when the available bandwidth of data channel 4 increases.

[0055] The process is preferably executed in real time. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 2025 / 0078508 A1

[0003]

Claims

[1] Method for transmitting data originating from an event-based image sensor (1) and containing events, in particular in a vehicle (100), wherein a camera stream (2) containing the events is generated from the data of the event-based image sensor (1), wherein an event prediction stream is generated from the camera stream (2) by means of transmitter event prediction (7), wherein a transmitter residual stream (9) is generated from the camera stream (2) and the event prediction stream by means of delta coding (8), wherein the transmitter residual stream (9) is converted into a bit stream (11) by means of entropy coding (10) and the bit stream (11) is transmitted via a data channel (4) of a receiver side (5), wherein on the receiver side (5) the bitstream (11) is converted into a receiver residual stream (13) by means of entropy decoding (12), wherein a user stream (16) is generated from the receiver residual stream (13) by means of receiver event prediction (14) and delta decoding (15), such that the user stream (16) contains the predicted events of the transmitter event prediction (7) as well as events of the camera stream (2), where the user stream (16) is provided to an application (6). [2] Method according to claim 1, characterized by , that the event prediction stream is generated directly from the camera stream (2) by means of the transmitter event prediction (7). [3] Method according to claim 1, characterized by , that the camera stream (2) is converted into a filtered data stream (18) by means of irrelevance reduction (17) before the generation of the event prediction stream, wherein the filtered data stream (18) is converted into the transmitter residual stream (9) by means of event prediction (7) and delta coding (8). [4] Method according to claim 3, characterized by, that the strength of the irrelevance reduction filtering (17) is adjustable, the strength of the filtering being adjusted depending on the available bandwidth of the data channel (4).

Images