Transmitting event streams over MIPI camera serial interface

Abstract

A method for transmitting a stream of events generated by an event-based visual sensor (10) over an interface (14, 16) configured to transmit data in consecutive packets (H-CS) grouped in consecutive frames (FS-FE), the method comprising the steps of: allocating periods to the frames; reading events with corresponding timestamps from the event stream; and populating packets of the current frame with only events having timestamps within the period of the current frame.

Description

Technical Field

[0001] The present invention relates to transmitting events generated by an event-based camera to a terminal device via a standard interface. Background Art

[0002] Standard camera interfaces in principle allow interoperability between cameras and end devices from different manufacturers. For example, the MIPI Camera Serial Interface (CSI-2) is designed to transmit video produced by a camera to an end device such as a display. The interface provides a proven and robust way to transfer data via differential lines, error detection and correction capabilities, and low power modes.

[0003] Such interfaces were originally designed for frame-based camera systems and rely primarily on the concept of frames that transfer a fixed amount of pixel data according to a raster scan pattern of the sensor array. Event-based cameras produce a stream of events, where events are transmitted in the order they occur rather than in the spatial order of the sensor array. Furthermore, only events that have been triggered are transmitted, which means that the transmission has a variable rate and does not represent all pixels of the camera, which is in principle incompatible with the concept of frames.

[0004] It would make sense to use features of a standard camera interface such as MIPI CSI-2 to stream the events generated by an event-based camera to the end device. Summary of the invention

[0005] Generally disclosed is a method for transmitting an event stream generated by an event-based visual sensor through an interface, the interface being configured to transmit data grouped in consecutive packets in consecutive frames, the method comprising the steps of: assigning a period to a frame; reading events with corresponding timestamps from the event stream; and filling the packets of the current frame only with events with timestamps within the period of the current frame.

[0006] The method may comprise the further step of inserting a major frame end marker immediately after the event in the last packet of the frame.

[0007] The method may further include the steps of: allocating fixed capacity events to each packet; continuously filling the packets to their capacity; and when the number of events is insufficient to fill the last packet of the frame, filling the remaining space of the last packet with filler events.

[0008] The method may further include the steps of: transmitting a fixed number of packets in each frame; and when the number of events is insufficient to fill all packets of the frame, filling the remaining packets with filler events.

[0009] The method may further include the steps of: assigning a period to the packets; and filling a current packet of the current frame with only events having a timestamp within the period of the current packet.

[0010] The event stream may include at least synchronization events occurring at a period that is smaller than a packet period, thereby forming a fixed number of packets for each frame, wherein each packet includes at least one synchronization event.

[0011] The method may further include the steps of: allocating fixed capacity events to each group; and filling the remaining capacity of the group with filling events when the number of events filling the current group is less than the capacity of the group.

[0012] A padding event for a packet may be inserted before the checksum suffix of the packet.

[0013] The interface may be a MIPI CSI-2 interface, where each frame has a frame start prefix, a frame end suffix, and each packet also has a header prefix and a checksum suffix. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Embodiments will be presented in the following description provided for exemplary purposes only, in conjunction with the accompanying drawings, in which:

[0015] Figure 1 is a block diagram of the transmission chain from an event-based camera to an end device or application using the MIPI interface;

[0016] Figure 2 exemplifies a MIPI CSI-2 frame filled in accordance with the first embodiment for transmitting data of an event stream;

[0017] Figure 3 exemplifies a MIPI CSI-2 frame filled according to the second embodiment for transmitting data of an event stream;

[0018] Figure 4 illustrates a MIPI CSI-2 frame filled in accordance with the third embodiment for transmitting data of an event stream; and

[0019] Figure 5 A MIPI CSI-2 frame filled according to the fourth embodiment for transmitting data of an event stream is illustrated. DETAILED DESCRIPTION

[0020] The MIPI CSI-2 interface is designed in practice to transmit video data based on frames and raster scanned pixels. More specifically, each frame is defined by a frame start prefix FS and a frame end suffix FE. In addition, each frame is subdivided into multiple packets that actually correspond to corresponding pixel rows. Therefore, the frames transmitted continuously define the frame rate, and the continuous packets transmitted for each frame correspond to the row scan rate. The number and size of the packets can be configured according to the type of frame-based video to be transmitted.

[0021] Figure 1 is a block diagram of a transmission chain from an event-based camera 10 to an end device or application 12 via a MIPI interface. The MIPI interface includes a transmitter 14 on the camera side and a receiver 16 on the end device side connected via a differential link. The MIPI transmitter receives the event stream from the camera 10 via a formatter circuit 18 that is responsible for converting the event stream into frames and packets according to the MIPI standard.

[0022] In principle, a standard-compliant receiver interface simply extracts the payload from the packets and transmits the concatenated payload to a terminal device or application that is specifically designed to process the payload.

[0023] In a common approach for transmitting event streams over the MIPI CSI-2 interface, the event data may simply be inserted as payload into consecutive packets of the same length, and each frame transmitted when all of its packets have been so filled. In this case, there is no direct relationship between the timestamps of the events in the stream being transmitted and the MIPI frame and packet timing.

[0024] However, this approach is not feasible because the events do not occur periodically, and thus the frame rate will vary with significantly variable event rates that exceed the standard specification.

[0025] Figure 2 A better method is illustrated as a first implementation. The figure shows the general structure of a MIPI CSI-2 frame. As mentioned before, each frame is delimited by a frame start prefix FS and a frame end suffix FE. The frame is subdivided into packets, each of which carries a payload with a header prefix H and a checksum suffix CS.

[0026] The formatter 18 is configured to generate frames from the event stream at a constant rate, whereby each frame is assigned a fixed transmission period. In practice, a frame rate between 10fps and 1Kfps may be selected. When forming a current frame for transmission, the formatter 18 reads the timestamps of the incoming events and inserts only events with timestamps within the period of the frame into the current frame, thereby establishing a relationship between the timestamps of the events in the stream and the MIPI frame timing.

[0027] More specifically, if Figure 2 As shown, the groups of frames are filled to the same specified length until no additional events match the timestamp constraint. Therefore, as shown, the last group is usually not completely filled, and the number of groups may be less than the traditional expected number of frame-based cameras.

[0028] Although the resulting frame may be MIPI compliant, its structure is uncommon in the field where the MIPI CSI-2 interface is most commonly used (i.e., frame-based cameras) because the length of the last packet is variable and the number of packets in a frame is also variable. Therefore, many receiver devices designed for frame-based cameras do not have the additional complexity of handling frames with variable length packets, whereby the last packet of a frame (which is typically shorter than the rest) can cause processing failures in the receiver device.

[0029] In particular, in such receivers, the specification of the MIPI CSI-2 standard that the packet header H includes a packet "word count" is often ignored, whereby the receiver interface does not inform the terminal application of the end of the packet or the amount of data transferred.

[0030] like Figure 2 As shown, the end of the event data in a frame can be marked by a major frame end suffix MFE inserted at the end of the payload of the last packet of the frame, just before the checksum CS is inserted in the last packet. The checksum calculation therefore also involves the MFE marking. Therefore, the MFE marking can be used by the terminal application to identify the end of the event data, even if the receiver does not inform the application of the received payload data size of the frame.

[0031] Figure 3 An embodiment is illustrated that is compatible with a receiver interface that expects frames with fixed-length packets or ignores word count information in the packet header. The payload of the last packet including event data is filled with stuffing events until the expected packet length is reached between the MFE marker and the checksum CS. This last checksum CS is therefore also calculated by the stuffing event and aligned with the other checksums at the position expected by the receiver.

[0032] The MIPI standard does not specify padding events and MFE markers, because they are part of the payload and can be arbitrary from the point of view of the standard. Therefore, the MFE markers and payload data can be specified for the design of the event stream formatter 18 and the corresponding terminal application.

[0033] The filling event can be a null event or an event that only transmits a timestamp, or simply any dummy data, since the data after the MFE marker is in principle ignored by the terminal application. The dummy data can be a series of bits set to the same state 1 or 0, which has the advantage of reducing the transmission power, since such data does not generate a state transition.

[0034] In the case where the receiver device also expects a fixed number of packets per frame, the padding event is used only to create up to the expected number of lost packets, such as Figure 3 The last group in is shown.

[0035] although Figure 2 and Figure 3 The implementation scheme provides frames at a fixed frame rate, but the time required to fill each packet varies with the event rate. In some cases, a given packet may start to fill with a series of events, and then pause as the event rate drops to zero momentarily before the packet is filled. Therefore, packet transmission is delayed until the event rate rises again to complete the packet filling, which is unsatisfactory in most cases where events should be received immediately after they occur. This results in more delay and jitter in the packet transmission time.

[0036] Figure 4 A preferred event stream formatting implementation is illustrated to avoid large delays in transmitting event data. The formatter 18 is configured to generate a specified number of packets for each frame, whereby each packet is assigned a fixed transmission period equal to the frame transmission period divided by the number of packets. When forming a current packet for transmission, the formatter 18 reads the timestamps of the incoming events and inserts only events with timestamps within the period assigned to the packet into the current packet, thereby establishing a relationship between the timestamps of the events in the stream and the MIPI packet timing.

[0037] More specifically, if Figure 4 As shown, each packet is filled until no further events match the timestamp constraint, after which the next packet is selected for filling. Therefore, depending on the event rate variation, the packet has a varying length. Therefore, the word count in the header H of the packet is set by the transport interface to reflect this situation.

[0038] In practice, a maximum size is set for the packet. The MIPI CSI-2 standard allows a maximum size of 64KB. However, the trade-off between latency and transmission efficiency is a smaller value, such as 8KB, which provides satisfactory latency and 97% transmission efficiency.

[0039] In some cases, the number of events that match the timestamp constraint of the current packet may exceed the packet capacity. In this case, three possibilities arise. As a first alternative, events that exceed the packet capacity are discarded and not transmitted. As a second alternative, events that exceed the packet capacity are inserted into the next packet, and the next packet is assigned to the same period, whereby multiple consecutive packets may include events with timestamps corresponding to the same packet period. As a third alternative, events that exceed the packet capacity are inserted into the next packet, and the next packet is assigned to the current period plus the subsequent period. Therefore, the next packet is assigned to events with timestamps spanning two periods.

[0040] The timestamp constraint can be expressed as follows. Each packet is assigned a time stamp equal to T f / N fixed period T p , where Tf is the frame period and N is the number of packets per frame. i Included in TS0 and TS0+T p An event is inserted into the current packet only when the timestamp of the event is contained between TS0 and TS0+2T, where TS0 is the timestamp of the first event inserted into the current packet or, if applicable, into the previous packet assigned to the same period. When the third alternative described above is implemented, after the first packet has been filled, when the timestamp of the event is contained between TS0 and TS0+2T, p When between packets, the event is inserted into the next packet.

[0041] This allocation of cycles to packets is transparent to the terminal application and does not require the transmission of corresponding information. The receiver interface will in principle extract the payload from the packets and concatenate them for transmission to the terminal device or application. This mechanism is only used to form standard-compliant packets that can be transmitted as fast as possible based on the timestamps of the events, resulting in a MIPI stream with less jitter in the packet transmission time interval than ordinary methods.

[0042] In another case where the event rate is low or drops significantly when filling a frame, there may not be enough events to insert into each packet, and thus the number of packets may not reach the number specified for the frame. This situation is avoided in practice by the fact that even if no events have been generated for a long period of time, event-based cameras will periodically insert synchronization events in the stream, which only convey a timestamp. The period of such events may be smaller than the period Tp allocated to the packets, and thus all packets must be used, each containing at least one synchronization event.

[0043] Figure 4 The implementation requires the receiver interface to be fully compliant with the standard and therefore use the packet word count in order to correctly extract the payload. As mentioned earlier, many receiver interfaces expect fixed-length packets and ignore the word count.

[0044] Figure 5 Illustrated Figure 4 A variation of the embodiment of the invention is compatible with receiver interfaces that expect a fixed packet length and a fixed number of packets per frame, resulting in a fixed frame size. A fixed capacity is allocated to packets within the limits specified by the standard, for example 8KB. When the current received event has a timestamp outside the current packet period, the current event is kept for the next packet while filling the remainder of the current packet with fill events until the packet capacity is reached. The fill events are inserted before the checksum CS, whereby the checksum is in the expected position and the fill events are also involved in its calculation.

[0045] To maintain compatibility with fully compliant receiver interfaces, the word counts in the packet headers are all set to fixed size packets, ie, they take into account both payload and padding events in each packet.

[0046] As shown, in all packets except the last one, there are no insertion markers between the payload and the padding events. This improves transmission efficiency, but the terminal application will not know where the padding events begin in each packet unless the padding events have a recognizable format. In practice, the receiver interface will also extract and forward the padding because the padding is indistinguishable from the payload of the receiver interface. The padding events may have the appropriate event format but transmit no information, or simply transmit a timestamp. Such events will be correctly parsed by the terminal application, but will be ignored because they do not require further processing.

Claims

1. A method for transmitting an event stream generated by an event-based vision sensor (10) via an interface (14, 16), the interface being configured to transmit data grouped in consecutive packets (H-CS) in consecutive frames (FS-FE), the method include: allocating a period to the frame; Reading events with corresponding timestamps from the event stream; as well as The packets of the current frame are populated only with events having timestamps within the period of the current frame.

2. The method according to claim 1, comprising the further step of inserting a Master Frame End marker (MFE) immediately after the event in the last packet of the frame.

3. The method according to claim 1, further comprising: include: Allocate a fixed volume of events to each group; continuously filling the packets to the capacity of the packets; and When the number of events is insufficient to fill the last packet of the frame, the remaining space of the last packet is filled with filler events.

4. The method according to claim 3, further comprising: include: A fixed number of packets are transmitted in each frame; and When the number of events is insufficient to fill all of the packets of the frame, the remaining packets are filled with filler events.

5. The method according to claim 1, further comprising: include: assigning a period to the grouping; as well as The current packet of the current frame is populated only with events having timestamps within the period of the current packet.

6. The method according to claim 5, wherein the event stream comprises at least synchronization events occurring at a period smaller than a packet period, thereby forming a fixed number of packets for each frame, wherein each packet comprises at least one synchronization event.

7. The method according to claim 6, further comprising: include: Allocate a fixed volume of events to each group; and When the number of events filling the current group is less than the capacity of the group, the remaining capacity of the group is filled with filling events.

8. The method of claim 7, wherein the padding event of the packet is inserted before a checksum suffix (CS) of the packet.

9. The method of claim 1, wherein the interface is a MIPI CSI-2 interface, wherein each frame has a frame start prefix (FS), a frame end suffix (FE), and each packet also has a header prefix (H) and a checksum suffix (CS).

10. An interface for transmitting an event stream generated by an event-based vision sensor (10), the interface being configured to: Transmitting data grouped in consecutive packets (H-CS) in consecutive frames (FS-FE); allocating a period to the frame; Reading events with corresponding timestamps from the event stream; as well as The packets of the current frame are populated only with events having timestamps within the period of the current frame.

11. The interface according to claim 10, further configured to: assigning a period to the grouping; and The current packet of the current frame is populated only with events having timestamps within the period of the current packet.

12. The interface according to claim 11, wherein the event stream includes at least synchronization events occurring at a period smaller than the packet period, thereby forming a fixed number of packets for each frame, wherein each packet includes at least one synchronization event.

13. The interface according to claim 12, further configured to: assigning a fixed volume of events to each group; and When the number of events filling the current group is less than the capacity of the group, the remaining capacity of the group is filled with filling events.

Images