Asynchronous readout encoding for event-based vision sensors

Abstract

Asynchronous readout encoding for event-based vision sensors (and related systems, devices, and methods) is disclosed herein. In one embodiment, a method of encoding event data from an event-based vision sensor includes encoding row address information using row address MSB packets and row address LSB packets each including a different prefix code. The method can further include encoding column address information using column address MSB packets and column address LSB packets each including a different prefix code. The method can further include encoding per-row information using a row additional packet positioned after the row address LSB packets, encoding per-pixel information using a column additional packet positioned after the column address LSB packets, or a combination thereof.

Description

Cross-reference of related applications

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 780,112, filed March 28, 2025; International Patent Application No. PCT / CN2025 / 098427, filed May 30, 2025; and International Patent Application No. PCT / CN2025 / 098453, filed May 30, 2025, the full disclosure of each of which is incorporated herein by reference. Technical Field

[0002] This technology generally relates to image sensors. For example, several embodiments of this technology described in detail below relate to encoding formats for event-based visual sensors that utilize variable-length prefix codes or flags, packet sharing methods, and / or flexible data structures to reduce data overhead while maintaining temporal and spatial information of events detected by pixels of an event visual sensor array. Background Technology

[0003] Image sensors have become ubiquitous and are now widely used in digital cameras, mobile phones, surveillance cameras, and medical, automotive, and other applications. As image sensors are integrated into a wider range of electronic devices, there is a desire to enhance their functionality, performance metrics, and the like in as many ways as possible (e.g., resolution, power consumption, dynamic range, etc.) through device architecture design and image acquisition and processing.

[0004] A typical image sensor operates in response to incident image light from an external scene. The image sensor comprises an array of pixels having photosensitive elements (e.g., photodiodes) that absorb a portion of the incident image light and generate image charge upon absorption. The image charge generated by the pixel light can be measured as an analog output image signal on the column lines, which varies depending on the incident image light. In other words, the amount of image charge generated is proportional to the intensity of the image light, and this amount of image charge is read out as an analog image signal from the column lines and converted into a digital value to provide information representing the external scene. Summary of the Invention

[0005] On one hand, this disclosure relates to a method for encoding event data from an event-based visual sensor, the method comprising: encoding row address information using row address MSB packets and row address LSB packets, the row address MSB packets containing a first prefix code and the row address LSB packets containing a second prefix code different from the first prefix code; and encoding column address information using column address MSB packets and column address LSB packets, the column address MSB packets containing a third prefix code and the column address LSB packets containing a fourth prefix code different from the third prefix code, wherein the method further comprises: encoding row-by-row information using one or more row append packets located after the row address LSB packets, encoding pixel-by-pixel information using one or more column append packets located after the column address LSB packets, or a combination thereof.

[0006] In another aspect, this disclosure relates to an event-based visual sensor system comprising: an event-based visual sensor pixel array configured to detect events; and an event signal processor configured to encode the detected events using an encoding format comprising coordinate data packets, the coordinate data packets comprising row address most significant bit (MSB) data packets, row address least significant bit (LSB) data packets, column address MSB data packets, and column address LSB data packets, wherein each coordinate data packet contains a prefix code for data packet identification, and wherein the encoding format further comprises: one or more row appended data packets for encoding row-by-row information; one or more column appended data packets for encoding pixel-by-pixel information; or a combination thereof. Attached Figure Description

[0008] Many aspects of this disclosure can be better understood with reference to the accompanying drawings. The components in the drawings are not necessarily to scale. Instead, the focus should be on clearly illustrating the principles of this disclosure. The drawings should not be construed as limiting this disclosure to the specific embodiments shown, but are provided for explanation and understanding.

[0009] Figure 1A This is a partial schematic diagram of a stacked hybrid complementary metal-oxide-semiconductor (CMOS) image sensor (CIS) and event-based vision sensor (EVS) system configured according to various embodiments of the present technology.

[0010] Figure 1B yes Figure 1A A partial schematic diagram of a specific instance of the system.

[0011] Figure 2 This describes an encoding format for transmitting event data from an event-based visual sensor, which is configured according to various embodiments of the present technology.

[0012] Figure 2A and 2B Description of various embodiments according to the present technology Figure 2 Various data packet sharing methods with different encoding formats.

[0013] Figure 2C Description of various embodiments according to the present technology Figure 2 The pixel-level encoding method for the encoding format.

[0014] Figure 2D Description of various embodiments according to the present technology Figure 2 A group-based encoding method for the encoding format.

[0015] Figure 2E Description of various embodiments according to the present technology Figure 2 The event run-length encoding method of the encoding format.

[0016] Figure 2F Description of various embodiments of the present technology for use Figure 2 Various methods for conveying timestamp information using encoding formats.

[0017] Figure 2G (a) illustrates various embodiments according to the present technology. Figure 2 (a) The embedded event grouping structure of the encoding format and (b) an instance of embedded event grouping that can be used to convey timestamp information.

[0018] Figure 3 This illustration demonstrates several block sequences of compression efficiencies achieved through various encoding techniques according to various embodiments of the present technology.

[0019] Figure 4 Description of various embodiments according to the present technology Figure 2 The possible order of data packets in the encoding format.

[0020] Figure 5 This describes another encoding format configured for pixel-by-pixel encoding according to various embodiments of the present technology.

[0021] Figure 6 This describes another encoding format for pixel-by-pixel encoding with per-event timestamps configured according to various embodiments of the present technology.

[0022] Figure 7 This describes various embodiments of the present technology that configure encoding formats with both row-by-row and column-by-column timestamps.

[0023] Figure 8 This describes another encoding format configured for pixel-by-pixel encoding according to various embodiments of the present technology.

[0024] Figure 9 This describes another encoding format configured according to various embodiments of the present technology for pixel group vector encoding in sparse event scenarios.

[0025] Figure 10 This describes the encoding formats configured for pixel group vector encoding in dense event scenarios according to various embodiments of the present technology.

[0026] Figure 11 This is a block diagram of an event vision sensor system configured according to various embodiments of the present technology.

[0027] Figure 12 This is a block diagram of a hybrid image sensor system configured according to various embodiments of the present technology.

[0028] Figure 13 This describes another encoding format for transmitting event data from an event-based visual sensor, which is configured according to various embodiments of the present technology.

[0029] Figure 13A Description of various embodiments according to the present technology Figure 13 The encoding method of the encoding format.

[0030] Figure 13B Description of various embodiments according to the present technology Figure 13 A group-based encoding method for the encoding format.

[0031] Figure 13C Description of various embodiments according to the present technology Figure 13 The event run-length encoding method of the encoding format.

[0032] Figure 14 This describes another encoding format for transmitting event data from an event-based visual sensor, which is configured according to various embodiments of the present technology.

[0033] Figures 14A to 14F This describes the various embodiments of the present technology that are feasible. Figure 14 The encoding format is used to convey various instances of embedded event grouping.

[0034] Figure 15 This describes unit and multi-bit techniques for encoding event polarity information for inline vector event encoding and line vector encoding, according to various embodiments of the present technology.

[0035] Those skilled in the art should understand that the components in the figures are illustrated for simplicity and clarity and are not necessarily drawn to scale. For example, the dimensions of some components in the figures may be exaggerated relative to other components to aid in the understanding of various aspects of the art. Furthermore, common but easily understood components or methods that are useful or necessary in commercially feasible embodiments are generally not depicted in the figures or described in detail below to avoid unnecessarily obscuring the description of various aspects of the art. Detailed Implementation

[0036] This technology relates to an encoding format for event-based visual sensors that utilizes variable-length prefix codes or flags, packet sharing methods, and flexible data structures to reduce data overhead while maintaining the temporal and spatial information of events detected by pixels in an event-based visual sensor array. The encoding format includes various packet types, such as row address, column address, and timestamp packets, that can be shared across multiple pixels or events to improve encoding efficiency. Different encoding modes are provided, including pixel-level encoding, group-based encoding, and event run-length encoding, allowing optimization based on event density and sensor characteristics. The technology also incorporates embedded group identifiers and flexible timestamp implementations to achieve additional functionality and synchronization capabilities in hybrid image sensor systems.

[0037] In the following description, specific details are set forth to provide a thorough understanding of aspects of this art. However, those skilled in the art will recognize that the systems, apparatuses, and techniques described herein can be practiced without one or more of the specific details set forth herein, or with other methods, components, materials, etc.

[0038] Throughout this specification, references to "example" or "implementation" mean that a particular feature, structure, or characteristic described in connection with an example or embodiment is included in at least one example or embodiment of the present technology. Therefore, the use of the phrases "for example," "as an example," or "implementation" herein does not necessarily refer to the same example or embodiment and is not necessarily limited to the specific example or embodiment discussed. Furthermore, the features, structures, or characteristics of the present technology described herein can be combined in any suitable manner to provide other examples or embodiments of the present technology.

[0039] For ease of description, spatial relative terms (e.g., “below,” “under,” “above,” “under,” “above,” “top,” “bottom,” “left,” “right,” “center,” “middle,” and the like) may be used herein to describe the relationship of an element or feature relative to one or more other elements or features, as illustrated in the figures. It should be understood that spatial relative terms are intended to cover different orientations of the device or system in use or operation other than those depicted in the figures. For example, if the device or system illustrated in the figures is rotated, turned, or flipped about a horizontal axis, then an element or feature described as “below,” “under,” or “below” one or more other elements or features may then be oriented “above” one or more other elements or features. Therefore, the exemplary terms “below” or “under” are non-limiting and may cover both above and below orientations. The device or system may also be oriented in other ways (e.g., rotated 90 degrees about a vertical axis, or otherwise), as illustrated in the figures, and the spatial relative descriptive terms used herein shall be interpreted accordingly. Additionally, it should be understood that when an element is referred to as being “between” two other elements, it may be the only element between the other two elements, or there may be one or more intermediary elements.

[0040] It should be understood that although the terms first, second, third, etc., may be used in this disclosure and claims to describe various elements, these elements should not be limited by these terms and should not be used to determine the process sequence or formation order of related elements. Unless otherwise indicated, these terms are used only to distinguish one element from another. Therefore, without departing from the teachings of the disclosed embodiments, the first element discussed below may be referred to as the second element.

[0041] It should be understood that the terms "photosensor" or "photodiode" may correspond to a doped region disposed within a semiconductor material, the doped region being configured to generate an image charge (e.g., one or more electrons or holes) in response to incident light. For example, a photodiode may correspond to an n-doped region disposed within a p-type semiconductor material, or an n-doped region within a semiconductor material surrounded by a p-type well, or a p-doped region within an n-type semiconductor.

[0042] Throughout this specification, several terms used in the field are employed. These terms should have their general meaning in the field, unless specifically defined herein or the context in which they are used will clearly indicate otherwise. It should be noted that throughout this document, element names and symbols are used interchangeably (e.g., Si and silicon); however, they have the same meaning.

[0043] A. Overview

[0044] Event-based vision sensors (EVS) have emerged as a promising technology for capturing dynamic visual information with high temporal resolution and low latency. Unlike traditional frame-based image sensors, EVS pixels asynchronously detect and report local brightness changes, generating sparse event data that can efficiently represent visual motion and temporal contrast. However, the asynchronous nature of EVS data presents challenges for data transmission and processing, especially when integrating EVS capabilities into hybrid image sensor systems that combine CMOS image sensor (CIS) imaging with event-based sensing.

[0045] Many existing EVS coding schemes use fixed-length flags to represent event data, which can lead to inefficient bandwidth usage and increased data overhead. For example, some methods employ Address Event Representation (AER) format, with fixed word lengths for flag bits and address information. While these methods provide a simple coding structure, they do not optimally compress event data, especially in scenarios with varying event densities or distributions across pixel arrays.

[0046] Other EVS encoding techniques implement a row-by-row grouping strategy to reduce data redundancy. However, these methods typically require sending event blocks for each column, which can lead to significant data overhead. Furthermore, many existing encoding schemes lack flexibility in representing different types of event information (such as polarity, timestamps, or additional sensor metadata) within a unified data structure.

[0047] This technology addresses the challenges associated with encoding event-based visual sensor data by introducing a novel encoding format that utilizes variable-length prefix codes or flags, packet sharing methods, and flexible data structures. This approach significantly reduces data overhead while preserving the temporal and spatial information of events detected by pixels in an event-based visual sensor array. The encoding format includes various packet types that can be shared across multiple pixels or events to improve encoding efficiency, including row address, column address, and timestamp packets. Different encoding modes are supported, such as pixel-level encoding, group-based encoding, and event run-length encoding, allowing for optimization based on event density and sensor characteristics.

[0048] The implementation of variable-length prefix codes for packet flags is expected to allow for more efficient use of bit space by assigning shorter flag sequences to frequently occurring packets and longer sequences to less common packets. The encoding format also incorporates a packet sharing method that enables multiple events sharing the same row or column segment to reuse address information, further reducing data redundancy. In addition to or instead of address information, the packet sharing method can also be used to share timestamps and / or other information. Furthermore, the technology introduces embedded packet identifiers and flexible timestamp implementations to enable additional functionality and synchronization capabilities in hybrid image sensor systems that combine CMOS image sensor (CIS) imaging and event-based sensing.

[0049] The disclosed encoding format offers several advantages over existing methods. Unlike fixed-length flag schemes, which can lead to inefficient bandwidth usage, variable-length prefix codes optimize bit allocation based on packet frequency. The packet-sharing approach addresses the limitations of row-by-row grouping strategies that require sending event blocks for each column, significantly reducing data overhead. Furthermore, the flexible data structure allows for the efficient representation of various types of event information, including polarity, timestamps, and additional sensor metadata, within a uniform format. This versatility enables the encoding scheme to adapt to different event distributions, sensor configurations, and application requirements, potentially improving overall system performance in areas such as low-light imaging, high-speed motion capture, and dynamic range optimization. Therefore, the encoding format (and associated systems, apparatus, and methods) disclosed herein are expected to reduce power consumption and data bandwidth usage overhead compared to other existing encoding formats. By reducing the amount of data transmitted from the image sensor via the MIPI transmitter, this technique is also expected to reduce data loss in event data, which in turn is expected to improve temporal information reconstruction (e.g., video frame interpolation, image deblurring).

[0050] B. Selected embodiments of encoding formats for event-based visual sensors and associated systems, devices, and methods.

[0051] Figure 1AThis is a partial schematic diagram of a stacked complementary metal-oxide-semiconductor (CMOS) image sensor (CIS) 130 (“Stacked System 130” or “Image Sensor 130”) configured with an event-based vision sensor (EVS) system according to various embodiments of the present technology. As shown, the stacked system 130 includes a first die 132, a second die 134, and a third die 136 stacked and coupled together in a stacked chip scheme. In some embodiments, the first die 132, the second die 134, and the third die 136 are semiconductor dies comprising a suitable semiconductor material (e.g., silicon). In the illustrated embodiment, the first die 132 (also referred to herein as the “top die”) includes a pixel array 138. The third die 136 (also referred to herein as the “bottom die”) includes an image readout circuitry system 146 (also referred to herein as the “image readout mixed-signal circuitry system”). The image readout circuitry system 146 can be coupled to the pixel array 138 of the first die 132 via a column-level connection 140 for normal image readout. In some embodiments, the column-level connection 140 for normal image readout is implemented from the column bit line of the pixel array 138 by extending between the first die 132 and the third die 136 and wiring through a through silicon via (TSV) of the second die 134.

[0052] In some embodiments, pixel array 138 is a two-dimensional (2D) array comprising a plurality of pixel units (also referred to as “pixels”), each pixel unit comprising at least one photoelectric sensor (e.g., at least one photodiode) exposed to incident light. As shown in the illustrated embodiments, pixels are arranged in rows and columns. Some of the pixels may be configured as CMOS image sensor (CIS) pixels, configured to acquire image data of people, locations, objects, etc., which can then be used to render images and / or videos of people, locations, objects, etc. For example, each CIS pixel is configured to generate an image charge in response to incident light. After each CIS pixel has acquired its image charge, the corresponding analog image charge data can be read out by image readout circuitry system 146 in third die 136 via column lines. In some embodiments, the image charge from each row of pixel array 138 can be read out in parallel via column lines by image readout circuitry system 146. As discussed in more detail below, others of the pixels in pixel array 138 may be configured as event vision sensor (EVS) pixels.

[0053] The image readout circuitry system 146 in the third die 136 may include amplifiers, analog-to-digital converter (ADC) circuitry, associated analog support circuitry, associated digital support circuitry, etc., for normal image readout and processing. In some embodiments, the image readout circuitry system 146 may also include an event-driven readout circuitry system, which will be described in more detail below. In operation, photogenerated analog image charge signals are read from the pixel units of the pixel array 138, amplified, and converted into digital values ​​in the image readout circuitry system 146. In some embodiments, the image readout circuitry system 146 may read out one line of image data at a time. In other instances, the image readout circuitry system 146 may use various other techniques (not described), such as serial readout or simultaneous fully parallel readout of all pixels, to read out image data. The image data may be stored or even manipulated through post-image effects such as cropping, rotation, red-eye removal, brightness adjustment, contrast adjustment, and the like.

[0054] In the illustrated embodiment, the second die 134 (also referred to herein as the "intermediate die") includes an event-driven sensing array 142 coupled to at least some of the pixels (e.g., EVS pixels) in the pixel array 138 of the first die 132. In some embodiments, the event-driven sensing array 142 is coupled to the pixels of the pixel array 138 via a hybrid bonding between the first die 132 and the second die 134. The event-driven sensing array 142 may include an array of event-driven circuitry. In some embodiments, according to the teachings of this disclosure, each of the event-driven circuitry in the event-driven sensing array 142 is coupled to at least one of a plurality of pixels in the pixel array 138 via a hybrid bonding between the first die 132 and the second die 134 to asynchronously detect events occurring in light incident on the pixel array 138.

[0055] In some embodiments, the corresponding event detection signal is generated by the event-driven circuitry in the event-driven sensing array 142. The event detection signal can be received and processed by the event-driven peripheral circuitry system 144. In some embodiments, the event-driven peripheral circuitry system 144 is arranged around the periphery of the event-driven sensing array 142 in a second die 134, such as... Figure 1A It is displayed in the middle. Figure 1A The embodiments described also illustrate a column-level connection 140 for normal image readout via a second die 134 between the first die 132 and the third die 136.

[0056] Figure 1B yes Figure 1A A partial schematic diagram of a specific instance of the stacked system 130. (See attached diagram.) Figure 1B As shown, the stacked system 130 includes a pixel array 138 on a first die 132. Figure 1BThe image shows only a portion of the pixel array 138, the event-driven circuitry 100 of the event-driven sensing array 142 on the second die 134, and the image readout circuitry system 146 on the third die 136. The image readout circuitry system 146 includes an analog-to-digital converter 151 (“ADC 151”), an image signal processor 152, a scan readout circuitry system 153, an event signal processor 154, a synchronization communication interface 155 (e.g., a Mobile Industry Processor Interface (MIPI) transmitter and / or receiver), and various auxiliary circuits 156. As discussed in more detail below, the image readout circuitry system 146 may also include deblurring circuitry (e.g., for performing event-driven deblurring of CIS data).

[0057] Figure 1B The portion of pixel array 138 shown corresponds to a 4×4 pixel cluster within pixel array 138. This cluster may repeat across pixel array 138. In the illustrated embodiment, fifteen (15) pixels in the cluster are configured as active (CIS) pixels 135 to capture CIS information (e.g., intensity information) corresponding to light incident on the photoelectric sensors of those pixels. Additionally, one pixel in the cluster is configured as an EVS pixel 137 to capture non-CIS information (e.g., contrast information, event data) corresponding to light incident on the photoelectric sensor of EVS pixel 137.

[0058] The CIS pixels 135 and EVS pixels 137 of the cluster are read out independently. More specifically, the CIS information captured by the CIS pixels 135 is read out via the ADC 151 on the third die 136 using the corresponding row / column control circuitry system (not shown) through the second die 134. The non-CIS information captured by the EVS pixels 137 is read out via the corresponding row / column control circuitry system (not shown) to the event-driven circuitry 100 on the second die 134, and events detected by the event-driven circuitry 100 are read out by the scan readout circuitry system 153 on the third die 136. The CIS information captured by the CIS pixels 135 of the pixel array 138 is frame-based and can be read out row by row from the CIS pixels 135 at the end of the exposure period. In contrast, the non-CIS information captured by the EVS pixels 137 is used by the event-driven circuitry 100 to asynchronously detect / trigger events, and the events can be read out according to either the row scan readout scheme or the column scan readout scheme. The row scan readout scheme will be discussed in more detail below.

[0059] In some embodiments, the row / column control circuitry system corresponding to CIS pixel 135 may be allocated on the same or different die as the die on which ADC 151 is allocated (e.g., third die 136). In these and other embodiments, the row / column control circuitry system corresponding to EVS pixel 137 may be allocated on the same or different die as the die on which scan readout circuitry system 153 is allocated (e.g., third die 136). In these and other embodiments, ADC 151 and / or the row / column control circuitry system corresponding to CIS pixel 135 may be allocated on the same or different die as the die on which scan readout circuitry system 153 and / or the die on which row / column control circuitry system corresponding to EVS pixel 137 is allocated (e.g., third die 136).

[0060] In the illustrated embodiment, EVS pixel 137 is dedicated to capturing non-CIS (EVS) information, while CIS pixel 135 is dedicated to capturing CIS information. In other embodiments, one or more of EVS pixel 137 and / or CIS pixel 135 may be configured to switch between capturing CIS information and non-CIS information. This allows the stacked system 130 to operate in the following modes: CIS-only mode, where all pixels 135 and 137 are used to capture CIS information; EVS-only mode, where all pixels 135 and 137 are used to capture non-CIS (EVS) information; and / or a hybrid CIS and EVS mode, where a first subset of pixels 135 and 137 is used to capture CIS information and a second subset of pixels 135 and 137 is used to capture non-CIS (EVS) information.

[0061] In some embodiments, the event-driven circuitry 100 on the second die 134 has the same die size as the 4×4 pixel cluster on the first die 132. In other embodiments, the event-driven circuitry 100 may have a different die size than the 4×4 pixel cluster. Additionally or alternatively, although the ratio of CIS pixels to EVS pixels in the 4×4 pixel cluster is 15:1, other ratios of CIS pixels to EVS pixels (e.g., 14:2, 12:4, 8:8, 4:12, 2:14, 15:1) are possible and fall within the scope of this technology. Furthermore, although... Figure 1B The EVS pixel 157 corresponds to a 4×4 pixel cluster, but other arrangements (e.g., EVS pixels corresponding to a 1×1 pixel cluster, a 4×2 pixel cluster, etc.) are possible and within the scope of this technology. Furthermore, although in Figure 1BOne row of EVS pixels (e.g., a row containing EVS pixels 157) corresponds to four rows of CIS pixels, but other arrangements are possible and within the scope of this technology. For example, each row of EVS pixels may correspond to (a) one row of CIS pixels, (b) two rows of CIS pixels, (c) three rows of CIS pixels, or (d) more than four rows of CIS pixels.

[0062] Figure 2 This section describes an encoding format 200 for transmitting event data from an event-based vision sensor, according to various embodiments of the present technology. Encoding format 200 provides a structured approach for efficiently transmitting event information, such as pixel location (x and y address information), polarity, and timestamps, while reducing data overhead. Format 200 supports stream-sequential encoding and decoding, making it well-suited for address-ordered, row-based streaming of event sequences. Encoding format 200 assumes that only EVS pixels with detected events are read out. Therefore, it can be used when asynchronously reading out detected events from an EVS pixel array. Figure 2 The image sensor is encoded using the encoding format 200 described herein, and is subsequently provided to the MIPI transmitter for transmission from the image sensor.

[0063] Encoding format 200 includes the following data packet sequence: row address most significant bit (MSB) data packet 202, row address least significant bit (LSB) data packet 204, optional row append data packet 206, column address MSB data packet 208, column address LSB data packet 210, optional column append data packet 212, and optional embedded packet identifier data packet 214. Each of these data packets is described in more detail below.

[0064] In the illustrated embodiment, encoding format 200 utilizes a data packet with a length of 8 bits. However, this data packet length can be adjusted to suit different system requirements or design preferences. For example, in some implementations, the data packet length can be extended to 16 bits, 32 bits, or any other suitable bit length.

[0065] Format 200 further utilizes variable-length prefix codes for packet flags that identify packet types / distinguish packet types from one another. These variable-length prefix codes are inspired by Huffman coding principles, which allow for efficient use of bit space. More specifically, by utilizing variable-length prefix codes for packet flags, the encoding format achieves improved efficiency in terms of bit space utilization. This approach allows for shorter flag sequences for more frequently occurring packets, potentially reducing overall data overhead. In some cases, frequently occurring packets can be assigned shorter flag sequences, while less frequent packets can have longer flag sequences. This variable-length flag scheme enables more efficient compression of event data, potentially leading to reduced bandwidth requirements and improved transmission speeds. Furthermore, the flexibility of variable-length flags allows the encoding format to be optimized for different event distribution patterns or sensor configurations, adapting to various use cases and scenarios in event-based visual sensing applications. Flags help maintain the internal state of the encoding sequence to properly encode or decode data, ensuring the correct interpretation of the packet sequence.

[0066] As described above, flags can be used to identify data packets and distinguish them from other data packets defined in encoding format 200. For data packets of encoding format 200 that are assigned a flag and transmitted with the flag, the position of the data packet in the encoded data stream can vary because the flag can be used by the decoder to identify the data packet and distinguish it from other data packets contained in the encoded data stream. For data packets of encoding format 200 that are not assigned a flag (or are assigned an optional flag that is not used when transmitting the data packet), the position of the data packet in the encoded data stream can be fixed, meaning that its preceding neighbor (e.g., another data packet immediately before the data packet) and / or its following neighbor (e.g., another data packet immediately after the data packet) is always the same. For example, a packet without a flag, when transmitted in an encoded data stream, can be positioned in the encoded data stream such that it is always (a) immediately following another packet with a first flag, (b) immediately following another packet of the same type (e.g., which also lacks a flag), (c) immediately preceding another packet with a second flag, and / or (d) immediately preceding another packet of the same type (e.g., which also lacks a flag). Continuing this example, the flags of packets positioned before or after a packet without a flag in the encoded data stream can be used by the decoder to identify the packet without a flag and distinguish it from other packets transmitted in the encoded data stream.

[0067] Figure 2The specific packet flag pattern shown represents one possible configuration, but alternative flag patterns can be used. In fact, various alternative flag patterns are described in more detail below. The flexibility of the encoding format described herein allows for customization of both packet length and flag pattern, provided that the resulting implementation conforms to the underlying hardware encoding algorithm. This adaptability enables the encoding format of this technique to be optimized for various applications and hardware configurations while maintaining its core functionality and efficiency.

[0068] exist Figure 2 At the beginning of the sequence shown, row address information is conveyed using two packets: Row Address MSB packet 202 and Row Address LSB packet 204. In the example shown, 4 bits of Row Address MSB packet 202 are used for the actual address information, while the remaining 4 bits (shown as '0110') are used as a flag to identify the packet type. Row Address LSB packet 204 uses 6 bits for address information and 2 bits (shown as '00') for the flag. This allocation allows for a total of 10 bits for the row address information, enabling addressing up to 1024 rows. For larger arrays, additional row address packets (such as one or more row append packets 206) can be added to the sequence to extend the addressable range. In some embodiments, an optional flag bit (shown as '1') can be used to identify row append packets 206 and distinguish them from other packets in the sequence. Optional row append packets 206 may follow row address LSB packets. As a specific instance, when a flag is not used, line append packet 206 may need to be followed by another packet (such as line address LSB packet 204 or another line append packet 206), for example, to help the decoder identify line append packet 206 (e.g., using the flag of the previous line address LSB packet 204). If enabled, then line append packet 206 may contain custom content, such as line-by-line information or timestamps, providing flexibility in data representation.

[0069] The sequence continues with column address packets. More specifically, the column address information is split between column address MSB packet 208 and column address LSB packet 210. Column address MSB packet 208 uses 5 bits for address information and 3 bits (shown as '010') for its flag, while column address LSB packet 210 uses 5 bits for address information and 1 bit (shown as '1') for its flag. Additionally, two bits of column address LSB packet 210 can be used for column embedded information. These column embedded bits 216 can be used to convey various information, such as event polarity or other pixel-specific attributes. One or both of the column embedded bits 216 can also be used to provide additional column address bit width, thereby extending the per-column resolution to up to 4096 columns. For row-level addressing, additional column address packets (such as one or more row append packets 212) can be added to the sequence to extend the addressable range to more than 1024 columns. In some embodiments, an optional flag (displayed as '1') can be used to identify column append data packet 212 and distinguish it from other data packets in the sequence. Optional column append data packet 212 may follow column address LSB data packet 210. As a specific instance, when a flag is not used, column append data packet 212 may need to follow another data packet (e.g., column address LSB data packet 210 or another column append data packet 212), for example, to help the decoder identify column append data packet 212 (e.g., using the flag of a previous column address LSB data packet 210). If enabled, column append data packets may contain custom content, such as pixel-by-pixel information, timestamps, or pixel group vectors, providing additional flexibility in event representation.

[0070] Encoding format 200 may optionally include an embedded packet identifier data packet 214, which may appear anywhere in the sequence and can be used to identify the start of an embedded event packet stream. See below for reference. Figure 2G In more detail, the embedded event packet stream may contain a packet sequence, the packet sequence including an embedded packet identifier packet 214 and an embedded packet header packet ( Figure 2 (not shown in the text) and one or more embedded packet data packets ( Figure 2 (Not shown in the text). Embedded event packet streams can be used to encode specific types of events (such as externally triggered events) at a low generation frequency.

[0071] Figure 2 The encoding format 200 described herein enables a packet sharing method for both row and column address information. As discussed in detail below, this method allows multiple events to reuse address information within the same row or column segment, reducing redundancy and improving compression efficiency. This sharing method can also be applied, or alternatively, to sharing timestamps and / or other information, as described in the reference below. Figure 2F and Figure 2GMore detailed description. In fact, this technology facilitates the sharing of any event information across several events using the principles of the packet sharing method described herein.

[0072] This encoding format supports various encoding modes. "Event pixel" encoding allows for the representation of individual pixel events, while "pixel group vector" encoding in column append data packet 212 enables efficient representation of events for pixel groups. Additionally, event run-length representation can be implemented in column address LSB data packet 210 to efficiently compress consecutive events.

[0073] By utilizing Figure 2 The encoding format 200 enables high encoding efficiency in event-based vision sensor systems while maintaining the flexibility to represent various types of event information. The format's design allows for on-chip integration at low cost, low latency, and low power consumption, making it suitable for a wide range of applications in event-based vision sensing.

[0074] Figure 2A Description of various embodiments according to the present technology Figure 2 Various packet sharing methods 220 of the encoding format 200. The encoded event sequence shown can be row-major and column-ordered (e.g., such that in each row, the event is encoded and streamed using an ascending sequence address). Figure 2A The data packet sharing method 220 shown in the figure utilizes Figure 2 The encoding format 200 uses row address MSB data packets 202, row address LSB data packets 204, column address MSB data packets 208, and column address LSB data packets 210 to efficiently encode the address information of multiple pixels or events. More specifically, Figure 2ASeven scenarios 221 to 227 are described. In scenario 221, event information from two rows is encoded and streamed without packet sharing. In scenario 222, event information from two rows is encoded and streamed using shared row address packets. Specifically, row address MSB packets 202 and row address LSB packets 204 are shared, so that events detected by two or more pixels in the same row can be encoded and streamed using the same row address MSB packets 202 and / or the same row address LSB packets 204. In scenario 223, event information from two rows is encoded and streamed using shared row address packets and shared column address MSB packets 208. Specifically, similar to scenario 222, events detected by two or more pixels in the same row can be encoded and streamed using the same row address MSB packets 202 and / or the same row address LSB packets 204. Additionally, events detected by two or more pixels with the same column address MSB can be encoded and streamed using the shared column address MSB packets 208. In scenario 224, similar to scenario 223, event information from two rows is encoded and streamed using shared row address packets and shared column address MSB packets 208. Compared to scenario 223, events detected by two or more pixels in a row with the same row address MSB can be encoded and streamed using the same / shared row address MSB packets in scenario 224.

[0075] Other possible packet-sharing techniques are possible and within the scope of this technology. For example, scenarios 225 to 227 illustrate this. Figure 2 The encoding format 220 can be used to encode event timestamps using shared timestamp data packets. In scenario 225, the timestamp information for each event is encoded and transmitted using column appended data packets 212. In scenario 226, the timestamp information for events detected by pixels in the same row is encoded and transmitted using shared row appended data packets 206. In scenario 224, the timestamp information is encoded and transmitted using shared embedded event data packets (e.g., containing embedded group identifier data packets 214), such that all events with the same timestamp share the same embedded event data packet. As yet another example, events detected by two or more pixels located in different rows but having the same column address MSB can share the same column address MSB data packet 208.

[0076] Figure 2B Description of various embodiments of the present technology for use Figure 2A packet sharing method 230 uses an encoding format 200 to convey row and column address information. In some cases, packet sharing method 230 can be applied to pixel arrays with 1024 rows and 1024 columns. The row address MSB packet 202 can utilize a 4-bit flag (displayed as '0110') to allow the remaining 4 bits to encode the row address MSB information. This configuration divides the array into 16 row segments 232, each containing 64 pixels. The 4 bits in the row address MSB packet 202 can be used to address these 16 segments 232.

[0077] Within each row segment 232, individual pixels 238 can be addressed using the row address LSB packet 204. The row address LSB packet 204 can use a 2-bit flag (displayed as '00'), leaving 6 bits available for encoding a specific row 236 within the identified segment 232. This arrangement allows for precise addressing of each of the 64 pixels 238 within a given row segment 232.

[0078] For column addressing, the column address MSB packet 208 can use a 3-bit flag (displayed as '010') to provide 5 bits for encoding the column address MSB information. This configuration allows the columns of the pixel array to be divided into 32 segments 234, each containing 32 pixels 238. The 5 bits in the column address MSB packet 208 can be used to address these 32 column segments 234.

[0079] Within each column segment 234, individual pixels 238 can be addressed using column address LSB packets 210. Column address LSB packets 210 may employ a single flag (displayed as '1'), leaving 5 bits available for encoding a specific column 239 within the identified segment 234. This arrangement allows for precise addressing of each of the 32 pixels 238 within a given column segment 234. Additionally, column address LSB packets 210 may include two bits 216 for column-embedded information, which can be used to convey event polarity or other pixel-specific data. In some embodiments, one or both of these column-embedded bits 216 may be unused or may be repurposed / used as column address bits to extend the column address bit length and thereby extend the number of addressable columns.

[0080] The packet sharing method 230 reduces data overhead by allowing multiple events to reuse address information and share the same row or column segment. For example, when encoding an event, the process can begin by transmitting a row address MSB packet 202 to identify row segment 232, followed by transmitting a row address LSB packet 204 to specify the exact row 236 within said segment 232. A timestamp packet (e.g., row append packet 206) can then be transmitted to provide time information about the event in said row 236.

[0081] Next, a column address MSB packet 208 may be sent to identify column segment 234, followed by a column address LSB packet 210 to specify the exact column 239 within said segment 234. Given the row address and column address information, this packet combination uniquely identifies an individual pixel 238.

[0082] If another pixel 238 in the same row segment 232 and column segment 234 detects an event (e.g., corresponding to the same timestamp), the event information can be encoded by sending (e.g., sending only) another column address LSB data packet 210. In this case, the previously transmitted row address MSB data packet 202, row address LSB data packet 204, and column address MSB data packet 208 are not retransmitted because they can be shared with multiple pixels within the same row segment 232 and column segment 234. This sharing of address information across multiple events significantly reduces the amount of data that needs to be transmitted.

[0083] When a new row segment 232 or a new column segment 234 is encountered, the new corresponding row or column MSB data packet 202 or 208 is encoded respectively. For example, if an event is detected in a different column segment 234, then a new column address MSB data packet 208 is transmitted, followed by one or more corresponding column address LSB data packets 210 for the event in the new segment 234.

[0084] Although Figure 2B The example demonstrates the sharing of row address packets, but the concept can also be extended to sharing column address packets. This flexibility allows for efficient encoding of event data regardless of the distribution pattern of events across the pixel array.

[0085] In the context of single-pixel encoding, packet sharing method 230 provides additional efficiency. Instead of encoding the event polarity separately, this information can be included in the column-embedded bit 216 of the column address LSB packet 210. Because the column address LSB packet 210 is transmitted for each detected event in single-pixel encoding, incorporating polarity information into this packet 210 eliminates the need for a separate polarity packet, further reducing the total number of packets transmitted. As a specific example, one of the column-embedded bits 216 in the column address LSB packet 210 (e.g., bit 1) can be set to "1" to indicate that a pixel detected a positive event, and can be set to "0" to indicate that a pixel detected a negative event, or vice versa.

[0086] Timestamp information can also be efficiently encoded using packet sharing method 230. In some implementations, timestamp packets (e.g., line appending packets 206) can be inserted at the beginning of each line. This method allows time information to be associated with events while minimizing the number of timestamp packets that need to be transmitted.

[0087] Figure 2A and 2B The packet sharing methods 220 and 230 described herein provide flexible and efficient methods for encoding event data from event-based vision sensors. By segmenting address information and sharing packets across multiple events, this method can significantly reduce data overhead while maintaining accurate spatial and temporal information for each detected event.

[0088] Figure 2C Description of various embodiments according to the present technology Figure 2 The pixel-level encoding method 240 is based on encoding format 200. In pixel-level encoding, event information of individual pixels can be efficiently encoded. Figure 2 Within the existing encoding format 200. In some embodiments, events can be encoded into blocks, where each event is assigned a row address and a column address. The timestamp 218 of each row or each EVS frame can be transmitted in a row appendage packet 206 following the row address LSB packet 204. In other words, events detected by pixels in the same row or the same EVS frame can share the same timestamp and / or row appendage packet 206. Additionally, the event polarity information for each event can be incorporated into each column address LSB packet 210, specifically within the column embedded bit 216. As discussed above, this method eliminates the need for separate packets to convey event polarity, potentially reducing overall data overhead. For example, one of the column embedded bits 216 in the column address LSB packet 210 can be specified to indicate event polarity, where a value '0' indicates a negative event and a value '1' indicates a positive event, or vice versa.

[0089] Figure 2D Various embodiments according to the present technology are depicted. Figure 2 The encoding format 200 uses a group-based encoding method 250 (also referred to herein as "pixel group vector encoding"). Group-based encoding differs from pixel-level encoding in that it can potentially further reduce data overhead by encoding events of multiple pixels together.

[0090] In some implementations of group-based encoding, events from group 252 of pixels 238 can be encoded together. For example, events from a 2×4 pixel block 252 (two rows and four columns of pixels, also referred to as event group 252) can be encoded together. Each event group 252 can be assigned x, y addresses for identification.

[0091] In the example of a 1536×2048 array, for event group 252, the pixel array can be divided into 768 y addresses (addressable using 10-bit y addresses) and 512 x addresses (addressable using 9-bit x addresses). The event information for each pixel 238 within group 252 can be represented using two bits, allowing encoding of no event, positive polarity events, or negative polarity events.

[0092] Event information for pixel group 252 can be conveyed using two 8-bit column appended data packets 212a and 212b. For example, event group 1 (in Figure 2D The event group (marked as event group 252a and displayed using x address 0 and y address 0) can have its event information encoded as: 00 (no event), 11 (positive event), 00, 10 (negative event) in the first column of supplementary data packet 212a, and 00, 00, 11, and 00 in the second column of supplementary data packet 212b. These two columns of supplementary data packets 212a and 212b can be followed by the corresponding column address LSB data packet 210 that identifies event group 252a.

[0093] In some implementations, each column of additional data packet 212 may contain an 8-bit pixel group vector containing event information for four pixels 238 corresponding to the same event group 252 (e.g., the same row 236 of the same event group 252). This method allows for efficient communication of event information for multiple pixels 238 while maintaining the ability to identify individual pixel events within group 252. In these and other embodiments, event groups 252 in which no events were detected (e.g., ...) can be skipped. Figure 2D Event groups (252x) are used to reduce the amount of data encoded and transmitted.

[0094] Using column appended data packets 212 for pixel group vector encoding provides flexibility in encoding format. In addition to event information, column appended data packets 212 can be used to convey other types of information, such as timestamp data, potentially allowing for further customization of the encoding scheme based on specific application requirements.

[0095] Figure 2E Description of various embodiments according to the present technology Figure 2 The event run length encoding method is 260, which is format 200. The event run length encoding method 260 can be referenced in conjunction with the above. Figure 2C The described pixel-level encoding method 260 provides an efficient method for encoding consecutive events, potentially further reducing data overhead in scenarios where multiple neighboring pixels 238 detect the event.

[0096] In some implementations of the event run-length representation, run-length information 262 (also referred to herein as the “isolation flag”) may be inserted into column-embedded bit 216 of the column address LSB packet 210. For example, one bit of column-embedded bit 216 (e.g., bit 1) may be used to convey the event polarity, while another bit (referred to as the “isolation (ISO) flag bit”) (e.g., bit 0) may indicate whether the adjacent pixel 238 has also detected the event.

[0097] In some cases, isolated events may be encoded differently from consecutive events. For example, consider isolated events that have been detected. Figure 2E Pixel 238 at row 36, column 66 (e.g., because neither pixel 238 at row 36, column 65 nor pixel 238 at row 36, column 67 detected an event). In this example, column address LSB packet 210 can be used to encode the event detected by pixel 238 at row 36, column 66. In this column address LSB packet 210, one column embedded bit 216 (e.g., bit 1) can convey the event polarity, while another column embedded bit 216 (e.g., bit 0) can indicate (e.g., via bit '1') that the event is isolated.

[0098] In contrast, a different encoding method can be used when consecutive pixels 238 detect events of the same polarity. For example, consider pixels 238 at row 36, columns 70-73. Each of these pixels 238 has detected an event of the same polarity. Therefore, in the event run-length representation, only the first and last events in the sequence can be encoded using the column address LSB packet 210. More specifically, the first column address LSB packet 210 can be used to encode the event detected by pixel 238 at row 36, column 70 (corresponding to the beginning of the event sequence). One column embedded bit 216 of this packet 210 can be used to convey the event polarity, and another column embedded bit 216 of this packet can be used to indicate (e.g., via bit '0') that the event is not isolated. The second column address LSB packet 210, immediately following the first column address LSB packet 210, can be used to encode the event detected by pixel 238 at row 36, column 73 (corresponding to the end of the event sequence). One column of embedded bits 216 of this second data packet 210 can be used to convey the event polarity, and another column of embedded bits 216 of this second data packet 210 can be used to indicate (e.g. via bit '0') that the event is not isolated.

[0099] In this scenario, column address LSB packets 210 are not streamed for "run-length pixels" (e.g., pixels 238 at rows 36, columns 71, and 72). Instead, the decoder can identify these as run-length pixels based on the first and second column address LSB packets 210. More specifically, the column address LSB bits included in the first and second column address LSB packets 210 can identify the start and end pixels 238 of the event sequence (thereby identifying pixels 238 between the start and end pixels 238 as "run-length pixels"), the polarity indicated in the column embedded bit 216 can indicate that each pixel 238 in the sequence detects an event of the same polarity, and the isolation flag bit 262 in the column embedded bit 216 can indicate that these events are not isolated events. This method can significantly reduce the number of packets streamed for consecutive events, potentially improving coding efficiency.

[0100] Event run length representation is particularly effective in scenarios with spatially related events, such as when there are moving edges or objects in the scene. By encoding only the start and end of the event run, this method achieves significant data compression while preserving the spatial and temporal information of the event.

[0101] Figure 2F Description of various embodiments of the present technology for use Figure 2 The encoding format 200 is used to convey various methods of timestamp information. The encoding format can support different modes for incorporating timestamp information, including out-of-pixel timestamp mode 270 and in-pixel timestamp mode 275.

[0102] In some cases, the intra-pixel timestamp pattern 275 can convey timestamp information 218 for each event detected by each pixel. This timestamp information 218 can be encoded in a column appendage packet 212 following the column address LSB packet 210. After transmitting the column appendage packet 212 containing timestamp information, the encoding format allows four possible packet types to follow: column address MSB packet 208, column address LSB packet 210, row address MSB packet 202, or row address LSB packet 204. Flags associated with these packets can be used by the decoder to identify which of the four packet types follows the column appendage packet 212.

[0103] The off-pixel timestamp mode 270 can convey a single timestamp 218 for a group of pixels (e.g., each row of pixels or each EVS frame). A frame can represent a complete scan of all rows of an event-based visual sensor pixel array from top to bottom.

[0104] In some implementations, a row-based or row-by-row out-of-pixel timestamp mode 270a may be used. In this mode 270a, a single timestamp 218 may be output for each row. At the beginning of each row, the timestamp information 218 may be conveyed in a row append packet 206 following a row address LSB packet 204. The row append packet 206 may be followed by a column address MSB packet 208 or a column address LSB packet 210.

[0105] When a row is not at the beginning of a row in the out-of-pixel timestamp pattern 270a, the timestamp information 218 may not be output in the row append packet 206. Instead, column address LSB packets 210 can be used to convey each event. Each of these column address LSB packets 210 may be followed by a column address MSB packet 208, a column address LSB packet 210, a row address MSB packet 202, or a row address LSB packet 204. If a row address LSB packet 204 follows, then the next row append packet 206 can be used to convey the timestamp information 218.

[0106] In some cases, a frame-based or frame-by-frame out-of-pixel timestamp mode 270b can be implemented. In this mode 270b, a single timestamp 218 can be output for each frame of event-based visual sensor data. At the beginning of each frame (e.g., at the beginning of the first row of the event-based visual sensor pixel array), a row address LSB packet 204 can be output, followed by a row append packet 206 containing the timestamp information 218.

[0107] When a frame is not at the beginning of a frame in frame-by-frame out-of-pixel timestamp mode 270b, the row address LSB packet 204 may be followed by either the column address MSB packet 208 or the column address LSB packet 210. Each column address LSB packet 210 may be followed by either the column address MSB packet 208, the column address LSB packet 210, the row address MSB packet 202, or the row address LSB packet 204.

[0108] The flexibility of the encoding format in implementing different timestamp representations (pixel-by-pixel, row-by-row, frame-by-frame) in row and column appended data packets 206 and 212 allows for efficient encoding of temporal information while maintaining compatibility with the overall data packet structure. This approach enables the encoding format to adapt to different event-based vision sensor configurations and application requirements, potentially optimizing the balance between temporal resolution and data efficiency. If a higher precision timestamp 218 is required, an embedded event group identifier data packet 214 can be used, as described below. Figure 2G More detailed description.

[0109] Figure 2G Description of various embodiments according to the present technology Figure 2The embedded event packet structure 280 (also referred to herein as "embedded event packet 280") is an encoding format 200. As shown, the embedded packet data structure 280 includes an embedded packet identifier data packet 214, an embedded event packet header data packet 282, and one or more embedded event packet data packets 284. The embedded event packet 280 can be... Figure 2 Optional blocks of the encoding format 200 can be inserted at any point in the packet sequence (e.g., the start of embedded event packet 280 is identified by using a flag (displayed as '0111') of embedded event packet identifier packet 214). Therefore, the presence of embedded event packet identifier packet 214 in the data stream conveys to the decoder additional information that will follow embedded event packet 280 in subsequent packets. This feature allows for the flexible inclusion of various types of data within the event stream without disrupting the main encoding structure. In some embodiments, the 4-bit extension field of embedded event packet identifier packet 214 may be unused (e.g., bits of embedded event packet identifier packet 214 that are not part of the flag). In other embodiments, the 4-bit extension field of embedded event packet identifier packet 214 can be used to encode additional information in the data stream, such as data type (DT) information of the data included in the corresponding embedded event packet data packet 284.

[0110] As shown, an embedded event packet identifier data packet 214 may be immediately followed by an embedded event packet header data packet 282. The embedded event packet header data packet 282 may contain information about the type and quantity of data contained in subsequent embedded event packet data packets 284. For example, the embedded event packet header data packet 282 may include a data type (DT) field and a word count (WC) field. In the illustrated embodiment, the data type field may use 5 bits to specify the nature of the information contained in subsequent embedded event packet data packets 284. The word count field may use 3 bits to indicate the number of embedded event packet data packets 284 that will follow the embedded event packet header data packet 282.

[0111] The data type field in the embedded event packet header data packet 282 can be used to specify various information. In some cases, the data type can indicate CIS-Sync information, which may include details such as frame number or exposure settings. This capability is particularly useful in hybrid image sensor systems that combine event-based vision sensors with CMOS image sensors, allowing synchronization between the two types of data (Event Vision Sensor (EVS) data and CMOS Image Sensor (CIS) data).

[0112] In some implementations, the data type can specify that the embedded event packet data packet 284 contains timestamp information. For example, the data type field can be used to specify a free-running (or asynchronously read-out event) timestamp. This allows for the inclusion of more precise time information in the encoded data stream when needed, potentially enhancing the time resolution of event data beyond that provided by standard timestamp packets. For example, this capability can allow for enhanced time accuracy when required by a specific application or use case.

[0113] Figure 2G Two examples of embedded event packets 280a and 280b that can be used to transmit timestamp information in an encoded data stream are further illustrated. In embedded event packet 280a, relative timestamp information can be shared by transmitting (1) an embedded event packet identifier packet 214 including (i) a packet identifier (PI) or a flag and (ii) an extension field (PX) and (2) an embedded event packet header packet 282 specifying data type 1 (DT=1) to indicate that the relative timestamp information is included in subsequent embedded event packet data packets 284. In this example, the word count field of the embedded event packet header packet 282 is set to 1 (WC=1), indicating that a single embedded event packet data packet 284 containing relative timestamp information (relative timestamp [7:0]) will follow the embedded event packet header packet 282.

[0114] Absolute timestamp information can be conveyed in the embedded event packet 280b. This is achieved by transmitting the embedded event packet identifier data packet 214, followed by the embedded event packet header data packet 282, wherein (a) the data field indicates data type 2 (DT=2) to indicate absolute timestamp information and (b) the word count field is set to 3 (WC=3) to convey three embedded event packet data packets 284 containing absolute timestamp information (absolute timestamp [23:16], absolute timestamp [15:8], absolute timestamp [7:0]). Figure 2G The embedded event packet data packets (284a to 284c) that are individually identified will be followed by the embedded event packet header data packet 282.

[0115] The data type field of the embedded event packet header data packet 282 can also be used to indicate that the embedded event packet data packet 284 contains statistical information or a time slice order / scan order identifier. Alternatively, the embedded event packet 280 can also be used to insert general-purpose input / output (GPIO) information into the data stream. In some cases, this can be used for multi-camera synchronization, allowing multiple event-based vision sensors to coordinate their data streams. These and other example data types (and associated embedded event packets 280) are referenced below. Figures 13 to 14FMore detailed description.

[0116] By providing a flexible mechanism for including additional information within the event data stream, the embedded event packet 280 enhances the versatility and functionality of the encoding format. This approach allows adaptation to various application requirements and sensor configurations without requiring changes to the core encoding structure. In other words, the embedded event packet 280 provides a flexible mechanism for including various types of additional information within the event data stream and / or allows... Figure 2 The encoding format 200 adapts to different application requirements or sensor configurations without changing the underlying structure of event data encoding.

[0117] Figure 3 Examples of various embodiments of the present technology, illustrating compression efficiencies achieved through various encoding techniques, are shown in several block sequences 391 to 397. More specifically, Figure 3 It provides a comparison of different encoding methods and their impact on reducing the number of packets transmitted.

[0118] Figure 3 The topmost block sequence 391 in the diagram represents the direct encoding method, which has the lowest block efficiency. This method encodes each event individually without sharing any address information between events.

[0119] Figure 3 The second packet sequence 392 illustrates line-by-line scanning encoding. This method reduces the number of packets by sharing the row address MSB packet 202, the row address LSB packet 204, and the row appended packet 206 containing pixel timestamp information across the same row of the event-based visual sensor pixel array. By sharing this information among multiple events in the same row, the total number of data packets transmitted can be reduced compared to the direct encoding method (corresponding to the first packet sequence 391).

[0120] Figure 3 The third group sequence 393 demonstrates adding column address MSB packet sharing to progressive scan encoding (corresponding to the second group sequence 392). This technique (corresponding to the third group sequence 393) can further reduce the number of groups by sharing column address MSB packets 208 for all event-based visual sensor pixels within the same column MSB segment. This method allows for more efficient encoding of events occurring in the same or nearby columns.

[0121] Figure 3 The fourth packet sequence 394 illustrates an implementation scheme for sharing row address MSB packets. This technique reduces the number of packets by sharing row address MSB packets 202 among all event-based visual sensor pixels within the same row MSB segment. This method can provide additional compression for events occurring in the same or nearby rows.

[0122] Figure 3 The fifth group sequence 395 illustrates an implementation scheme for pixel group vector encoding. In this method, pixel group vector data packets can be used to convey event information of pixel groups. Intra-pixel timestamp data packets can be skipped (e.g., implemented using column appending data packets 212), and timestamp information can be shared among pixels within a group (e.g., using row appending data packets 206). This technique can further reduce the number of packets by encoding multiple events within a single data packet.

[0123] Figure 3 The sixth block sequence 396 in the example demonstrates pixel-by-pixel or pixel-level encoding. In this method, event information can be conveyed in each column address LSB packet 210 without using pixel group vector encoding. This method provides a balance between compression efficiency and individual pixel event representation.

[0124] Figure 3 The seventh block sequence 397 in the example demonstrates an implementation scheme for run-length pixel encoding. This technique can further reduce the number of streaming packets by encoding only the first and last events in a series (or non-isolated) of events (e.g., using one or more column-embedded bits in the column address LSB packet 210, such as one to convey the event polarity and one as an isolation flag to identify isolation and run-length events). By representing the run-length of events more efficiently, this method can achieve significant compression for scenarios with spatially correlated events.

[0125] Figure 3 The advancements in encoding techniques illustrated demonstrate how various methods can be combined to achieve increasingly efficient compression of event data. Each technique can build upon previous ones, potentially leading to a reduction in the total number of packets required to represent the same event information. This compression can result in reduced bandwidth requirements and improved transmission efficiency for event-based visual sensor data.

[0126] Figure 4 Description of various embodiments according to the present technology Figure 2 The possible data packet order of the encoding format 200. Figure 4 It also provides an overview of how different packet sequences can be used in various timestamp patterns and encoding scenarios. In some embodiments, Figure 4The possible packet order described herein can be used by the decoder to aid in decoding the encoded data stream. For example, given a specific packet type and operating mode, the decoder can update a candidate pool reflecting all possible packet types that can immediately follow the given packet type. The decoder can then parse and traverse the candidate pool to identify the type of the next packet. This process can be repeated (e.g., updating the candidate pool based on the current packet type and parse and traverse the updated candidate pool to identify the type of the next received packet), for example, until the encoded data stream has been decoded.

[0127] Figure 4 This describes three possible packet sequence scenarios 478 when transmitting non-timestamp-related packets. For example, when the row address MSB packet 202 is streamed, it may be followed by the row address LSB packet 204 or an embedded event packet 280. Figure 4 The embedded event block identifier packet 214 (not shown in the image) begins. As another example, when the column address MSB packet 208 is streamed, the column address MSB packet 208 may be followed by either the column address LSB packet 210 or the embedded block identifier packet 214.

[0128] As discussed above, the embedded event packet identifier data packet 214 can appear anywhere in the encoding format sequence to indicate the start of the embedded event packet 280. When the embedded event packet identifier data packet 214 is streamed, the embedded event packet data packet 284 corresponding to the embedded event packet 280 is... Figure 4 (Not shown in the text) can be followed by various packet types, including row address MSB packet 202, row address LSB packet 204, row append packet 206, column address MSB packet 208, column address LSB packet 210, column append packet 212, or another embedded event packet identifier packet 214 to indicate the start of the next embedded event packet 280. This flexibility allows additional information to be inserted at any point in the data stream.

[0129] Figure 4 It also explains how packet sequences can differ based on the timestamp pattern used. (See reference above.) Figure 2F In the discussion, in the off-pixel timestamp mode 270 in which timestamp information is shared (e.g., shared by each row or by each EVS frame), the column address LSB packet 210 may be followed by the column address MSB packet 208, another column address LSB packet 210, the row address MSB packet 202, the row address LSB packet 204, or the embedded packet identifier packet 214.

[0130] In frame-by-frame out-of-pixel timestamp mode 270a, a row append data packet 206 containing timestamp information for the entire frame may follow the row address LSB data packet 204. After the row append data packet 206, the sequence may continue with the column address MSB data packet 208, the column address LSB data packet 210, or the embedded packet identifier data packet 214.

[0131] In the row-by-row out-of-pixel timestamp mode 270b, a row append data packet 206 containing timestamp information for the specific row may follow the row address LSB data packet 204. After the row append data packet 206, the sequence may continue with the column address MSB data packet 208, the column address LSB data packet 210, or the embedded packet identifier data packet 214.

[0132] In the pixel-level timestamp pattern 275, which streams timestamp information for each pixel of a detected event, a column append data packet 212 containing the timestamp information for that specific pixel may follow the column address LSB data packet 210. After the column append data packet 212, the sequence may continue with a column address MSB data packet 208, another column address LSB data packet 210, a row address MSB data packet 202, a row address LSB data packet 204, or an embedded packet identifier data packet 214.

[0133] These various packet sequences and timestamp patterns provide flexibility in encoding event data from event-based vision sensors, allowing for efficient data transmission while maintaining temporal and spatial information about detected events.

[0134] Figure 5 This describes another encoding format 500 for pixel-by-pixel encoding in event-based vision sensors, which is configured according to various embodiments of the present technology. As shown, see above reference. Figures 2 to 4 Compared to the described encoding format 200, encoding format 500 utilizes different flag identifiers (IDs) for various data packet types.

[0135] For example, row address MSB packet 502 can use flag ID '11' instead of '0110' (as used in encoding format 200). This modification allows two extra bits in row address MSB packet 502 to be used for row address MSB information. Similarly, row address LSB packet 504 can use flag ID '110' instead of '00' (as used in encoding format 200). These changes enable a total of 11 bits of encoding for row address information, potentially allowing addressing of up to 2048 rows.

[0136] Figure 5Encoding format 500 can also modify other packet flag IDs. For example, row append packet 506 can use flag ID '0' instead of '1' (as is optionally used in encoding format 200), column address MSB packet 508 can use '10' instead of '010' (as is used in encoding format 200), column address LSB packet 510 can use '0' instead of '1' (as is used in encoding format 200), and embedded event group identifier packet 514 can use '1111' instead of '0111' (as is used in encoding format 200).

[0137] In some cases, this alternative flag ID scheme can introduce potential ambiguity between the row address MSB packet 504 and the embedded block identifier packet 514, since both can begin with '11'. To resolve this, the encoding format can be limited to addressing a total of 1536 rows (e.g., rows 0 to 1535), which ensures that the highest value of bits 7:4 of the row address MSB packet 502 is '1110' and never reaches '1111', thereby allowing the decoder to distinguish it from the embedded block identifier packet 514. In other words, the illustrated encoding format 500 can use 11 row address bits and 11 column address bits (instead of the above reference). Figures 2 to 4 The encoding format 200 described uses 10 row address bits and 10 column address bits to support pixel arrays of up to 1536×2048 pixels.

[0138] As shown, Figure 5 The encoding format described herein can use line appending data packet 506 to insert timestamps on a frame or line basis. This method provides efficient encoding of time information while maintaining compatibility with the modified flag ID scheme.

[0139] Figure 5It also includes a flowchart 599 illustrating the decoding process for encoding format 500. The decoding process can begin by checking the embedded packet identifier packet flag ('1111'). If present, the decoder can read the packet as the embedded packet identifier packet 514. If absent, the decoder can attempt to determine whether the row address MSB packet 502 and the row address LSB packet 504 have been transmitted consecutively together. This determination can be made by checking two consecutive packets and looking for the flag '11' in the first packet, followed by the flag '110' in the second packet. If this pattern is detected, the decoder can interpret the first packet as the row address MSB packet 502 and the second as the row address LSB packet 504. If this pattern is not detected, the decoder can check whether the row address LSB packet 504 has been transmitted separately from the row address MSB packet 502, and if so, read it accordingly. The decoder can then check the row appended packet 506, and if present, read it accordingly. The decoder can continue to sequentially examine column address MSB packet 508 and column address LSB packet 510. If column address MSB packet 508 or column address LSB packet 510 is present, the decoder can read the identified packet and return to sequentially examining column address MSB packet 508 and column address LSB packet 510. If neither column address MSB packet 508 nor column address LSB packet 510 is present, the decoder can return to examining embedded packet identifier packet 514.

[0140] Figure 6 Another encoding format 600 for pixel-by-pixel encoding is depicted according to various embodiments of the present technology. This encoding format 600 can focus on inserting a timestamp for each individual event. More specifically, the timestamp of each event can be encoded in a column append packet 612 following each column address LSB packet 610. This method can provide precise timing information for each detected event, potentially allowing for more accurate reconstruction of event sequences or improved timing analysis in applications requiring high temporal resolution.

[0141] Figure 6 It also includes flowchart 699 illustrating the decoding process for encoding format 600. As shown, the flowchart is relatively similar to... Figure 5 The above explanation is also referenced. Figure 5 The flowchart described is 599.

[0142] Figure 7Another encoding format 700 for pixel-by-pixel encoding according to various embodiments of the present technology is described. Encoding format 700 can simultaneously implement both (a) row-by-row or frame-by-frame timestamping and (b) column-by-column timestamping. In some cases, encoding format 700 can use row appending data packets 706 to encode row-by-row or frame-by-frame timestamping information. Simultaneously, column appending data packets 712 can be used to encode column-by-column timestamping information. This dual approach to timestamping encoding provides flexibility in the representation of time data, potentially allowing for efficient encoding of both global (row or frame) and local (column or pixel) time information.

[0143] Figure 7 It also includes flowchart 799 illustrating the decoding process for encoding format 700. As shown, the flowchart is relatively similar to those in... Figure 5 and 6 The above explanation is also referenced. Figure 5 and 6 The flowcharts 599 and 699 are described.

[0144] Figure 8 Another encoding format 800 for pixel-by-pixel encoding according to various embodiments of the present technology is described. Encoding format 800 may introduce a modified flag scheme for row address packets. More specifically, encoding format 800 may implement MSB flag bit 819 in row address LSB packets 804. MSB flag bit 819 can be used to indicate whether the immediately preceding packet is row address MSB packet 802. For example, when row address LSB packets 804 are transmitted with bits 7:4 as '1100', this can indicate to the decoder that the previously received packet is not row address MSB packet 802. Conversely, when row address LSB packets 804 are transmitted with bits 7:4 as '1101', this can indicate that the previously received packet is row address MSB packet 802.

[0145] Figure 9 This section describes an alternative encoding format 900 for pixel group vector encoding according to various embodiments of the present technology. This encoding format 900 is particularly suitable for sparse event scenarios, where a relatively small number of pixels in a group detect events.

[0146] In some implementation schemes, Figure 9 The encoding format 900 can replace the column embedding bits in the column address LSB packet 910 with half-valid bits 917. For example, bits 6:5 of the column address LSB packet 910 can be used as half-valid bits 917 to indicate which rows within a pixel group have detected an event.

[0147] Encoding format 900 can organize pixels into groups. One of these 2×4 pixel groups is... Figure 9The group is marked as "Group 0". In some cases, two bytes may be needed to encode the event for each pixel in this group. However, using half a significant bit (917) allows for more efficient encoding when only some pixels in the group detect the event.

[0148] For example, if no event is detected in the first four pixels of a group (pixels 0 to 3), then the encoding of those pixels can be skipped. The half-significant bit 917 in the column address LSB packet 910 can be used to indicate that this encoding is skipped. In some implementations, if the column address LSB packet 910 is transmitted with the half-significant bit set to '10', this can indicate to the decoder that no event was detected in the top pixel row of the group. Therefore, packets may not be sent for the top pixels (e.g., column appendage packet 912 is not sent), and only packets for the bottom pixel rows of the group (e.g., column appendage packet 912) may be transmitted.

[0149] Similarly, if column address LSB packet 910 is transmitted with half-valid bit 917 set to '01', this can indicate to the decoder that no event was detected in the bottom pixel row (pixels 4 to 7) of the group. In this case, packets may not be sent for the bottom pixels (e.g., column append packet 912 may not be sent), and only packets for the top pixel row of the group (e.g., column append packet 912) may be transmitted.

[0150] In some cases, if column address LSB packet 910 is transmitted with half-valid bit 917 set to '11', this indicates to the decoder that an event was detected in both rows of the pixel group. Therefore, packets (e.g., column append packets 912) can be sent for both rows. Optionally, if column address LSB packet 910 is transmitted with half-valid bit 917 set to '00', this indicates to the decoder that no event was detected in either row of the pixel group, and both packets (e.g., two column append packets 912) can be skipped for both rows of the group.

[0151] Figure 9 The document also describes the use of a top / bottom bit 915 in the row address LSB packet 904. In some implementations, this bit 915 is used when the upper half of a pixel array or pixel array segment is scanned simultaneously (or in parallel) with the lower half of the pixel array or pixel array segment. The top / bottom bit 915 in the row address LSB packet 904 indicates to the decoder whether a pixel row is located in the upper or lower half of the pixel array / pixel array segment. For example, a top / bottom bit value of '0' indicates that the pixel row is in the upper half of the pixel array / pixel array segment, while a value of '1' indicates that the pixel row is in the lower half.

[0152] Encoding format 900 can utilize embedded event groups to convey timestamp and / or timer information. For example... Figure 9 As shown in the example, for instance, the extended fields (bits [3:0]) of the embedded event group identifier packet 914 can be used to convey timestamp information and / or indicate when a timer is full / exhausted. Figure 9 In the specific example 931 shown, row appendage data packet 906 can be used to convey row information shared by pixels in the same pixel group (e.g., exposure flags and / or out-of-pixel timing information indicating whether an event occurred within or outside the CIS exposure period of a CIS pixel in a hybrid image sensor, for example, containing both CIS and EVS functions in a pixel array), and embedded event groups can be used to convey timer information, for example, using bits of the extended field corresponding to the embedded event group identifier data packet 914. Figure 9 In other examples 932 shown, embedded event packets (instead of line append data packets 906) are used to convey timestamp information, for example, by using bits of the extended field of the corresponding embedded event packet identifier data packet 914. Figure 9 As shown in the diagram, encoding format 900 also defines MIPI virtual data packets 913, which, for example, can resemble embedded event group identifier data packets 914, except for bit 0.

[0153] Figure 10 Another encoding format 1000 for pixel group vector encoding according to various embodiments of the present technology is described. Encoding format 1000 is particularly suitable for dense event scenarios, where a large number of pixels in a group detect events.

[0154] In some implementation schemes, Figure 10 The encoding format 1000 can modify the bit allocation in the column address LSB packet 1010 to accommodate more row address information. For example, in sparse event scenarios (such as... Figure 9 The half-valid bit 917 used in the description can be omitted from the column address LSB packet 1010 of encoding format 1000. Instead, the 7 bits of the column address LSB packet 1010 can be used to convey address information, instead of the 5 bits used in sparse event scenarios.

[0155] Using 7 bits for address information in the column address LSB packet 1010 allows for larger event groups. In some cases, this enables more events to be conveyed per segment, potentially reducing the overall number of packets that need to be transmitted. This approach is more efficient in scenarios where a high density of events is expected.

[0156] Figure 10It is also stated that an exposure flag bit 1011 is included in the column address MSB data packet 1008. In some embodiments, this exposure flag bit 1011 can be used to indicate whether an event occurs within or outside the CIS exposure period of a CIS pixel in a hybrid image sensor that includes both CIS and EVS functions in a pixel array. This method is consistent with (a) referenced above. Figures 2 to 4 The described embodiment in which the exposure flag can be conveyed in the column embedded bit 216 of the column address LSB packet 210 differs from, and / or is consistent with, (b) the above reference. Figure 9 The described embodiment differs from the one in which the exposure flag can be conveyed in the row-attached data packet 906.

[0157] Figure 9 and 10 The encoding formats 900 and 1000, described separately, demonstrate how event-based visual sensor encoding schemes can adapt to different event densities. By utilizing different bit allocations and packet structures based on the expected event density, these encoding formats 900 and 1000 can optimize data transmission efficiency in various scenarios encountered in event-based visual sensing applications.

[0158] Figure 11 This diagram illustrates an event vision sensor system 1130 configured according to various embodiments of the present technology. System 1130 may implement any or more of the encoding formats described above or any or more of the encoding formats described below.

[0159] The event vision sensor system 1130 may include an event-based vision sensor (EVS) pixel array 1142. The EVS pixel array 1142 may contain a plurality of pixels 1100 configured to asynchronously detect changes in light intensity and generate corresponding event signals.

[0160] Connected to the EVS pixel array 1142, the system 1130 may include a scanner 1153. In some cases, the scanner 1153 may be implemented as a full scanner capable of performing row and column scans in parallel. This parallel scanning capability allows for efficient readout of events from the EVS pixel array 1142.

[0161] Scanner 1153 may be coupled to event packetizer 1154. Event packetizer 1154 may receive event information from scanner 1153, including the location of the triggering event within EVS pixel array 1142. In some embodiments, event packetizer 1154 may perform encoding operations described in previous sections and / or described in more detail below to convert event data into a structured data packet format.

[0162] Following event packetizer 1154, the system may include sequencer 1157. Sequencer 1157 arranges the encoded data packets from event packetizer 1154 into an appropriate order for transmission. This ordering operation ensures that the data packets are organized according to the encoding format specification.

[0163] The final component in the described data path may be a MIPI (Mobile Industry Processor Interface) transmitter 1155. The MIPI transmitter 1155 can receive ordered data packets and potentially transmit them off-chip to other processing components or systems.

[0164] This hardware architecture enables efficient implementation of encoding schemes, allowing for rapid processing and transmission of event data from the EVS pixel array. The combination of parallel scanning capabilities and dedicated encoding and sequencing components enables high-speed, low-latency operation of the event vision sensor system 1130.

[0165] In some cases, the described hardware components can be integrated into a single chip or module, potentially reducing system complexity and power consumption. The modular nature of the architecture also allows for flexibility in system design, with the possibility of scaling or modifying individual components to meet specific application requirements.

[0166] Figure 12 This diagram illustrates a hybrid image sensor system 1230 configured according to various embodiments of the present technology. The hybrid image sensor system 1230 can integrate both event-based vision sensor (EVS) and CMOS image sensor (CIS) data paths to achieve efficient processing and transmission of hybrid sensor data.

[0167] The hybrid image sensor system 1230 may include a CIS / EVS sensor core containing an image array with row control and column control circuitry systems 1258 and 1259 for managing pixel readout, respectively. A setup control module 1260 may interface with the row and column control circuitry systems 1258 and 1259. The sensor core may include two parallel data paths: an EVS data path and a CIS data path.

[0168] The EVS data path may include an EVS pixel circuitry system 1200 connected to scan and event signal processor (ESP) blocks 1253 and 1254, respectively. Scan block 1253 performs row and / or column scans of the EVS pixel circuitry system 1200, while ESP block 1254 processes scanned event data. In some cases, ESP block 1254 may use one or more of the encoding formats described herein to encode the event data.

[0169] The CIS data path may contain an amplifier (AMP) 1261, an analog-to-digital converter (ADC) 1251, a black level calibration 1262, and a digital gain processing block 1263. The AMP 1261 amplifies the analog signal from the CIS pixels, while the ADC 1251 converts the amplified analog signal into a digital signal. The black level calibration 1262 adjusts the baseline signal level, and the digital gain block 1263 applies additional amplification to the digital signal.

[0170] The timing generator and system control logic block 1264 can provide timing and control signals to the sensor core components. The control register group 1265 can interface with the timing generator via the serial interface 1266 and can be connected to the digital signal processor (DSP) 1267 in the sensor processor section.

[0171] The sensor processor section may include a buffer 1268 connected to the DSP 1267 and a sensor control circuitry 1269. The buffer 1268 may temporarily store data from both the EVS and CIS paths, while the DSP 1267 may perform additional processing on the buffered data. The sensor control circuitry 1269 manages various aspects of sensor operation, potentially including the configuration of sensor settings and operating modes.

[0172] The output interface section may contain a MIPI (Mobile Industry Processor Interface) interface circuit system 1255 for transmitting processed sensor data. The MIPI interface circuit system 1255 enables high-speed data transmission to external devices or processors.

[0173] In some cases, the hybrid image sensor system 1230 allows for flexible operation in different modes (e.g., CIS-only mode, EVS-only mode, or hybrid CIS / EVS mode). The system architecture enables efficient processing of both CIS and EVS data, with dedicated processing paths for each type of sensor data before combination at the output interface.

[0174] Integrating CIS and EVS capabilities within a single system 1230 offers advantages in a variety of applications. For example, system 1230 can capture high-resolution image data via the CIS path while simultaneously detecting rapid changes in the scene via the EVS path. This combination can achieve enhanced performance in scenarios such as low-light imaging, high-speed motion capture, or dynamic range optimization.

[0175] Figure 12The hybrid image sensor system 1230 described herein provides a flexible and efficient architecture for capturing and processing both conventional image data and event-based visual data. By integrating these capabilities, system 1230 offers improved performance and versatility compared to traditional image sensors or standalone event-based visual sensors.

[0176] Figure 13 This describes another encoding format 1300 for transmitting event data from an event-based visual sensor, which is configured according to various embodiments of the present technology. Encoding formats similar to those described above (e.g., those similar to those referenced above) are also described. Figures 2 to 4 The encoding format described is 200). Figure 13 The encoding format 1300 defines four coordinate data packets (also referred to herein as Address Event Representation (AER) packets): Row Address MSB packet 1302, Row Address LSB packet 1304, Column Address MSB packet 1308, and Column Address LSB packet 1310. The coordinate data packets can be used to encode events generated by event sensor pixels (also referred to herein as "contrast detection" events), and more specifically, to encode (i) the address (x, y coordinate position) where each event is detected in the event sensor pixel array and (ii) the polarity of the event (e.g., up or down).

[0177] Each coordinate data packet contains (a) a different prefix (or flag) and (b) a number of content bits. Each prefix is ​​used to identify the corresponding data packet type, and the content bits are used to embed the address and / or polarity information corresponding to the event. For example, the row address MSB data packet 1302 contains a 4-bit prefix (displayed as '1110') and 4 content bits [9:6] for specifying the 4 most significant row address bits of the encoded event. The row address LSB data packet 1304 contains a 2-bit prefix (displayed as '10') and 6 content bits [5:0] for specifying the 6 least significant row address bits of the encoded event. Therefore, the 4 content bits [9:6] of the row address MSB data packet 1302 and the 6 content bits [5:0] of the row address LSB data packet 1304 can together specify the row address corresponding to the encoded event. Similarly, the column address MSB packet 1308 contains a 3-bit prefix (displayed as '110') and 5 content bits [9:5] for specifying the 5 most significant column address bits of the encoded event. The column address LSB packet 1310 contains a 1-bit prefix (displayed as '0') and 7 content bits, 5 of which can be used to specify the 5 least significant column address bits of the encoded event, and the other two can be column embedded bits 1316, which can be used to convey various information (such as the other two least significant column address bits, event polarity, isolation flag, etc.), as described in more detail below. Therefore, the 5 content bits [9:5] of the column address MSB packet 1308 and the 5 to 7 content bits [5:0] of the column address LSB packet 1304 can together specify the column address corresponding to the encoded event.

[0178] During the encoding process, packets containing higher (or most significant) address bits (e.g., row address MSB packet 1302 and / or column address MSB packet 1310) may be encoded before packets containing lower (or least significant) address bits, such that (i) packets with higher address bits can be encoded fewer times and (ii) packets with lower address bits can share packets with packets with higher address bits, consistent with the description of the technique provided above.

[0179] As shown, encoding format 1300 can further define row appended data packets 1306 and / or column appended data packets 1312. Row appended data packets 1306 and column appended data packets 1312 can be broadly similar to those in the reference above. Figures 2 to 4 The described row appended data packet 206 and column appended data packet 212. Therefore, in view of the above references Figures 2 to 4For the sake of brevity, the detailed descriptions of row appended data packets 1306 and 1312 are largely omitted here. As shown, row appended data packets 1306 and 1312 do not contain a prefix, and therefore, in use, they can be placed at fixed positions in the encoded data stream (e.g., immediately after row address LSB packet 1304 for row appended data packet 1306, and immediately after column address LSB packet 1310 for column appended data packet 1312) to help the decoder identify and decode them. As discussed in more detail above and below, row appended data packets 1306 and / or column appended data packets 1312 can be user-defined, for example, to convey timestamp information associated with an event, pixel group vectors, and / or other information.

[0180] Encoding format 1300 further defines embedded event packets 1380. Each embedded event packet 1380 is an identifiable data segment formed by a plurality of consecutive embedded event data packets: (i) an embedded event packet identifier (EPI) data packet 1314, (ii) an embedded event packet header (EPH) data packet 1382, and (iii) one or more embedded event packet data (EPD) data packets 1384. In some embodiments, embedded event packets 1380 may be used to encode events and / or other information, such as events or other information that are not frequently generated and / or transmitted within an encoded data stream. In other words, embedded event packets 1380 provide encoding format 1300 with greater flexibility in encoding and representing all kinds of events and / or information.

[0181] As shown, the embedded event packet identifier data packet 1314 includes: (i) a 4-bit prefix (or flag), which (shown as '1111') is used, for example, to indicate the start of the embedded event packet 1380 to the decoder; and (ii) a 4-bit packet extension field 1387. In other embodiments, the prefix of the embedded event packet identifier data packet 1314 may be... Figure 13 The prefix shown in the text is different. Alternatively, the prefix of the embedded event packet identifier packet 1314 may be customizable. In some embodiments, the packet extension field 1387 may not be used. Alternatively, as described in more detail below, the packet extension field 1387 may be used to (a) extend the bit width of the data type information and / or word count information conveyed at least partially by the embedded packet header packet 1382, and / or (b) further design other packets (e.g., row address extended packets, column address extended packets) based on the '1111' prefix.

[0182] The embedded event packet header data packet 1382 may contain (i) a two-bit data type (DT) field 1383 and (ii) a six-bit word count (WC) field 1385. The two-bit data type field 1383 can be used to specify the content type of information contained in the payload of the embedded event packet 1380 (e.g., the embedded event packet data packet 1384). The word count field 1385 can be used to specify the number of embedded event packet data packets 1384 (or bytes) contained in the embedded event packet 1380. In other words, the word count field 1385 can be used to specify the payload size of the embedded event packet 1380.

[0183] Embedded event packet data packets 1384 may contain the actual data payload of embedded event packets 1380. Although any number of embedded event packet data packets 1384 may be used in a single embedded event packet 1380, the data / information contained in the embedded event packet data packets 1384 may be limited to a single type of data / information and / or limited to the type specified in the data type field 1383 of the corresponding embedded event packet header data packet 1382.

[0184] Figure 13A Description of various embodiments according to the present technology Figure 13 The encoding method 1340 of encoding format 1300. In the illustrated embodiment, one of the column embedded bits 1316 (e.g., bit 1) of the column address LSB packet 1310 can be used as a column address bit, thereby increasing the addressable resolution to 1280×720. Another of the column embedded bits 1316 of the column address LSB packet 1310 can be used to convey the polarity of an event encoded and streamed using encoding format 1300. For example, a value '0' of the column embedded bit 1316 designated to indicate the event polarity can indicate a negative event, and a value '1' of the column embedded bit 1316 designated to indicate the event polarity can indicate a positive event, or vice versa. As discussed above, this method can eliminate the need for a separate packet to convey the event polarity, potentially reducing overall data overhead.

[0185] In some embodiments, timestamp information can be encoded and transmitted using embedded event blocks 1380. Figure 13AThe description covers two specific instances of embedded event packets 1380a and 1380b that can be used to transmit timestamp information in an encoded data stream using method 1340. In embedded event packet 1380a, variations in the least significant bits (e.g., bits [7:0]) of the event timestamp can be encoded and transmitted using: (1) an embedded event packet identifier packet 1314 including (i) a packet identifier (PI) or flag '1111' and (ii) an extension field (PX); (2) an embedded event packet header packet 1382 having a data type field 1383 with a specified data type 1 (DT=1) to indicate that LSB timestamp information is included in the payload of embedded event packet 1380a; and (3) an embedded event packet data packet 1384. In this example, the word count field 1385 of the embedded event packet header packet 1382 is set to 1 (WC=1), indicating that a single embedded event packet data packet 1384 containing LSB timestamp information is included in embedded event packet 1380a.

[0186] In the embedded event packet 1380b, the most significant bit (e.g., bits [31:8]) of the event timestamp can be encoded and transmitted using the following: (1) an embedded event packet identifier packet 1314 including (i) a packet identifier (PI) or flag '1111' and (ii) an extension field (PX); (2) an embedded event packet header packet 1382 having a data type field 1383 specifying data type 2 (DT=2) to indicate that the MSB timestamp information is contained in the payload of the embedded event packet 1380b; and (3) one or more embedded event packet data packets 1384. In this example, the word count field 1385 of the embedded event packet header packet 1382 is set to 3 (WC=3), indicating three embedded event packet data packets 1384 containing MSB timestamp information (in Figure 13A The data packets (1384a to 1384c) individually identified as event packets are contained in the embedded event packet 1380b.

[0187] In some embodiments, row appendage data packets 1306 and / or column appendage data packets 1312 are not used, and / or are not used for encoding and transmitting timestamp information. In other embodiments, row appendage data packets 1306 and / or column appendage data packets 1312 may be used, in addition to or instead of embedded event packets 1380, to encode and transmit timecode information. As a specific example, column appendage data packets 1312 may (e.g., instead of the embedded event packets 1380a described above) be used to encode and transmit variations in the least significant bits (e.g., bits [7:0]) of the event timestamp.

[0188] Figure 13B Description of various embodiments according to the present technology Figure 13 The encoding format 1300 uses a group-based encoding method 1350. In the illustrated embodiment, two of the column embedded bits 1316 (e.g., bits 0 and 1) of the column address LSB data packet 1310 can be used as column address bits, thereby increasing the addressable resolution to 4096×4096. In this example, the pixel array can be divided into 4096×1024 pixel groups, where each pixel group contains 4 pixels arranged in 4 rows and 1 column. Each pixel group can be scanned, and events detected by the pixels of the pixel group can be encoded together in the column append data packet 1312. For example, in Figure 13B In the example described, bits [1:0] of column append data packet 1312 may correspond to the first pixel of the pixel group, bits [3:2] of column append data packet 1312 may correspond to the second pixel of the pixel group, bits [5:4] of column append data packet 1312 may correspond to the third pixel of the pixel group, and bits [7:6] of column append data packet 1312 may correspond to the fourth pixel of the pixel group. Continuing this example, bits 7, 5, 3, and 1 of column append data packet 1312 may each indicate whether the corresponding of the four pixels in the pixel group has detected an event (e.g., a value '0' may indicate that the event was not detected by the corresponding pixel, and a value '1' may indicate that the event was detected by the corresponding pixel), and bits 6, 4, 2, and 0 of column append data packet 1312 may each indicate the polarity of the corresponding event detected by the corresponding pixel. Therefore, events can be encoded using bit pairs in column supplementary data packet 1312, for example, where {1,1} indicates that the corresponding pixel detected an event with positive polarity, {1,0} indicates that the corresponding pixel detected an event with negative polarity, and {0,0} indicates that the corresponding pixel did not detect an event. Similar to the reference above. Figure 13A The described method 1340, embedded event grouping 1380 can be used according to Figure 13B The method described in section 1350 is used to encode and transmit timestamp information of events.

[0189] Figure 13C Description of various embodiments of the present technology for use Figure 13 The event run-length encoding method 1360 is based on encoding format 1300. In the illustrated embodiment, one of the column-embedded bits (e.g., bit 0) of the column address LSB data packet 1310 is used to convey the polarity information of the detected event, and the other of the column-embedded bits (e.g., bit 1) of the column address LSB data packet 1310 is used as a reference above. Figure 2EThe isolation flag 1362 described is similar to the isolation flag bit 262. Therefore, the addressable resolution of AER packets (e.g., row address MSB packet 1302, row address LSB packet 1304, column address MSB packet 1308, and column address LSB packet 1310) is 1024 × 1024. In some embodiments, row appended packets 1306 can be used to convey LSB timestamp information of detected events. Row appended packets 1306 can be shared among events detected by pixels in the same row. Similar to the reference above. Figure 13A and 13B The methods described in 1340 and 1350, the embedded event group 1380 (e.g., similar to embedded event group 1380b) can be used according to Figure 13C The method described in section 1360 is used to encode and transmit the MSB timestamp information of the event.

[0190] Figure 14 This describes another encoding format 1400 for transmitting event data from an event-based visual sensor, which is configured according to various embodiments of the present technology. As shown, encoding format 1400 is generally similar to... Figure 13 The encoding format is 1300. Therefore, in view of the above reference... Figure 13 For the sake of brevity, detailed descriptions of several aspects of encoding format 1400 are omitted here. Compared to encoding format 1300, in addition to the four AER packets described above (e.g., row address MSB packet 1402, row address LSB packet 1404, column address MSB packet 1408, and column address LSB packet 1410), encoding format 1400 also defines two AER packets (e.g., row address extended packet 1492 and column address extended packet 1494). As shown, row address extended packet 1492 contains a 5-bit prefix or flag (displayed as '11110') and three content bits [2:0] for specifying the three most significant row address bits of the encoded event. Additionally, column address extended packet 1494 contains a 6-bit prefix or flag (displayed as '111110') and two content bits [1:0] for specifying the two most significant column address bits of the encoded event. The prefixes of row address extended data packet 1492 and column address extended data packet 1494 each begin with '1111', and their tracing... Figure 13 The prefix of the embedded event packet identifier data packet 1314 in the encoding format 1300. Therefore, in some embodiments, Figure 14The row address extension data packet 1492 and column address extension data packet 1494 of the encoding format 1400 can be defined according to the embedded event packet 1480 definition. As described in more detail below, the embedded event packet identifier data packet 1414 may include a prefix (displayed as '111111') to distinguish it from other types of embedded event packets 1480 and row address extension data packets 1492 and 1494. Because row address extension data packets 1492 and 1494 provide additional content bits [2:0] and [1:0] respectively that can be used to specify additional row and column address bits, the addressable resolution can be increased from 1024×2048 to 8192×8192 by using row address extension data packets 1492 and 1494.

[0191] Defining six AER packets in encoding format 1400 is intended to provide greater flexibility to adapt to a given resolution of the image sensor. For example, for an image sensor with a resolution of 64×64 or less, two of the six AER packets defined in encoding format 1400 can be used to fully address every pixel of the image sensor. Similarly, for an image sensor with a resolution of 1024×2048 or less, four of the six AER packets defined in encoding format 1400 can be used to fully address every pixel of the image sensor. As yet another example, for an image sensor with a resolution of 8192×8192 or less, all six AER packets defined in encoding format 1400 can be used to fully address every pixel of the image sensor.

[0192] Similar to Figure 13 The encoding format is 1300. Figure 14 The encoding format 1400 also defines embedded event packets 1480. Each embedded event packet 1480 is a identifiable data segment formed by several consecutive embedded event data packets: (i) Embedded Event Packet Identifier (EPI) data packet 1414, (ii) Embedded Event Packet Header (EPH) data packet 1482, and (iii) one or more Embedded Event Packet Data (EPD) data packets 1484. This is in contrast to the use of a 4-bit prefix. Figure 13 Compared to the embedded event group identifier data packet 1314, Figure 14 The embedded event block identifier packet 1414 utilizes a 6-bit prefix (displayed as '111111'), which allows the decoder to identify the embedded event block identifier packet as the beginning of embedded event block 1480 and distinguish it from row address extended packet 1492 and column address extended packet 1494. The two remaining bits [1:0] of the embedded event block identifier packet 1414 form the packet extension field 1487.

[0193] With a two-digit data type field 1383 and a six-digit word count field 1385 Figure 13 Compared to the embedded event packet header data packet 1382, Figure 14 The embedded event packet header data packet 1482 may include a 4-bit data type field 1483 and a 4-bit word count field 1485. In some embodiments, the packet extension field 1487 of the embedded packet identifier data packet 1414 may be combined with the data type field 1483 of the embedded event packet header data packet 1482 to specify the 6-bit data type of the content in the payload of the corresponding embedded event packet 1480. For example, the packet extension field 1487 of the embedded packet identifier data packet 1414 may be used to specify the most significant bit of the data type of the content, and the data type field 1483 of the embedded event packet header data packet 1482 may be used to specify the least significant bit of the data type.

[0194] As discussed above, the packet extension field 1487 of the embedded packet identifier packet 1414 and the data type field 1483 of the embedded event packet header packet 1482 can be used to specify the 6-bit data type of the content in the payload of the corresponding embedded event packet 1480. Therefore, up to 64 data types can be supported by the embedded event packet 1480 of encoding format 1400. Data types can be arranged according to data type classes. Table 1 below lists four different instances of data type classes and the corresponding range of 6-bit data type values:

[0195]

[0196] Table 1

[0197] Within each of the above data type classes, up to 16 different data type definitions may exist. Data types belonging to the functional event type class may include data types of events generated by system modules (e.g., not locally from event sensor pixels). For example, data belonging to the functional event type class may include data useful for various system functions, such as internal timeline construction (timestamp change events), multi-system synchronization (externally triggered events), data segmentation and identification (frame start / end events), etc. Table 2 below lists several instance data types (and corresponding instance data type values ​​and / or corresponding word count values) that may be included in the functional event type class:

[0198]

[0199]

[0200] Table 2

[0201] Figures 14A to 14EThe various embodiments of this technology can correspond to the data types listed in Table 2 above and can be described as follows: Figure 14 The example of embedded event grouping 1480 used in encoding format 1400 for encoding and conveying information. Figure 14A This section describes (i) embedded event packet 1480a, which can be used to convey LSB timestamp changes, and (ii) embedded event packet 1480b, which can be used to convey MSB timestamp changes. Referring to embedded event packet 1480a, the packet extension field 1487 of the embedded event packet identifier packet 1414 and the data type field 1483 of the embedded event packet header packet 1482 specify data type DT=0, indicating that the content of the embedded event packet data packet 1484 of embedded event packet 1480a is timestamp LSB change information. The word count field 1483 of embedded event packet 1480a indicates that embedded event packet 1480a contains a single byte of embedded event packet data packet 1484.

[0202] Referring to embedded event packet 1480b, the packet extension field 1487 of the embedded event packet identifier packet 1414 and the data type field 1483 of the embedded event packet header packet 1482 specify data type DT=1, indicating that the content of the embedded event packet data packet 1484 of embedded event packet 1480b is timestamp MSB change information. The word count field 1483 of embedded event packet 1480b indicates that embedded event packet 1480b contains 3 bytes of embedded event packet data packet 1484.

[0203] As discussed above, timestamps can be a common characterization parameter for contrast detection events and any other type of event. For a given event stream, it is desirable to use a single global timeline shared by multiple events, so that the timestamps of different events can be compared relative to that timeline.

[0204] In many event streams, events can be generated at a frequency higher than timestamp changes. Therefore, multiple events can share the same timestamp value. This concept can be used to achieve better encoding efficiency by defining timestamp-changing events separately and encoding the timestamp of each event relative to those timestamp-changing events (e.g., instead of embedding / encoding timestamp values ​​for each event).

[0205] For example, two types of "timestamp change" events can be defined to describe different degrees of time change: (1) a "timestamp LSB change event" can be defined to describe the case when the 8 least significant bits [7:0] of the timestamp change change, and (2) a "timestamp MSB change event" can be defined to describe the case when the 24 most significant bits [31:8] of the timestamp change change. When a "timestamp change" event occurs, subsequent events detected by the EVS pixel can share a timestamp value different from the EVS event that occurred before the timestamp change event. Therefore, the event encoding order should be: "timestamp MSB change" event, "timestamp LSB change" event, and subsequent EVS events.

[0206] Figure 14B This describes embedded event packet 1480c, which can be used to convey external trigger data type information (such as trigger ID, trigger polarity, and trigger count). External trigger events can specify value changes on the monitored input (trigger line). The trigger line is typically external to the event sensor and can be used to synchronize an external device with the event sensor, thereby allowing the user to correlate the event sensor's timeline with an external device. As shown, the packet extension field 1487 of the embedded event packet identifier packet 1414 and the data type field 1483 of the embedded event packet header packet 1482 specify data type DT=2, indicating that the contents of embedded event packet data packets 1484a and 1484b of embedded event packet 1480c are external trigger information. The word count field 1483 of embedded event packet 1480c indicates that embedded event packet 1480a contains two bytes of embedded event packet data packet 1484. The first embedded event packet data packet 1484a can be used to encode and transmit the trigger ID [7:0] / device ID [7:0] and a 1-bit indication of the trigger polarity (e.g., '0' for falling edge changes and '1' for rising edge changes). The second embedded event packet data packet 1484b can be used to encode and transmit the trigger count.

[0207] Figure 14C The description refers to embedded event packets 1480d to 1480g, which can be used to convey MIPI frame start information, MIPI frame end information, event frame start information, and event frame end information, respectively. MIPI frame start and MIPI frame end events can be used to identify the start and end of a MIPI frame, which is typically useful for platforms that receive event data from event sensors via a MIPI interface. The payload of these embedded event packets 1480d and 1480e can be a MIPI frame ID (or frame count), which typically starts from 0 at the first MIPI transmission frame and increments at the beginning or end of each subsequent MIPI frame.

[0208] For event sensors, an event frame typically refers to a scan of the entire EVS pixel array for an event. This scan can be synchronous or asynchronous. Event frame start and event frame end events can be used to identify the start and end of an event frame, which is typically useful for identifying event frames within a given MIPI frame. The payload of such embedded event groups 1480f and 1480g can be an event frame ID (or frame count), which typically starts at 0 for each new MIPI frame and thereafter increments at the start or end of each new event frame / MIPI subframe within the same MIPI frame.

[0209] Referring to embedded event packets 1480d to 1480g, each has a similar structure, wherein the packet extension field 1487 corresponding to the embedded event packet identifier packet 1414 and the data type field 1483 corresponding to the embedded event packet header packet 1482 specify the corresponding data type (for example, for embedded event packet 1480d, DT=3, indicating that the content in embedded event packet data packets 1484a and 1484b is MIPI frame start information; for embedded event packet 1480e, DT=4, indicating that the content in embedded event packet data packets 1484a and 1484b is MIPI frame end information; for embedded event packet 1480f, DT=5, indicating that the content in embedded event packet data packets 1484a and 1484b is event frame start information; and for embedded event packet 1480g, DT=6, indicating that the content in embedded event packet data packets 1484a and 1484b is event frame end information). The word count field 1483 of embedded event packets 1480d to 1480g indicates that each of the embedded event packets 1480d to 1480g contains two bytes of embedded event packet data packet 1484.

[0210] Figure 14D The description refers to embedded event groups 1480h and 1480i, which can be used to convey external metadata insertion information and internal metadata reporting information, respectively. External metadata insertion event information specifies an external data insertion request received on the monitored input (insertion line). This is typically received from an external sensor (e.g., an inertial measurement unit (IMU)) that works in conjunction with the event sensor. Metadata information can be device-dependent.

[0211] Internal metadata reporting of event information can specify internal state update reports. This is typically useful when an event sensor adjusts internal operating parameters of an event pixel or functional core and notifies external systems of the validity period of the new parameters. For example, an event sensor might change the contrast threshold used for an event pixel. Subsequent events generated using the updated contrast threshold can be identified in a continuous event stream using embedded event groups in the internal metadata report.

[0212] Referring to embedded event packets 1480h and 1480i, each has a similar structure, wherein the packet extension field 1487 corresponding to the embedded event packet identifier packet 1414 and the data type field 1483 corresponding to the embedded event packet header packet 1482 specify the corresponding data type (for example, for embedded event packet 1480h, DT=7, indicating that the content of embedded event packet data packets 1484a to 1484c is external metadata insertion information; and for embedded event packet 1480i, DT=8, indicating that the content of embedded event packet data packets 1484a to 1484c is internal metadata reporting information). The word count field 1483 of embedded event packets 1480h and 1480i indicates that each of embedded event packets 1480h and 1480i contains 3 bytes of embedded event packet data packet 1484. The first byte 1484a of embedded event packet 1480h can specify the device ID of the device supplying the metadata contained in the second and third bytes 1484b and 1484c. The first byte 1484a of the embedded event group 1480i can specify the report ID (e.g., internal report type) corresponding to the metadata (e.g., updated operation parameters) contained in the second and third bytes 1484b and 1484c.

[0213] Figure 14E The description refers to embedded event packets 1480j and 1480k, which can be used to convey hybrid CIS frame start and end information, respectively. For hybrid sensors combining CMOS and EVS image sensors, embedded event packets can be used to identify the start and end of CIS image frames, enabling on-chip synchronization between the two systems (e.g., CMOS and EVS image sensors). The start of a CIS image frame may correspond to the scan time of the first line, and the end of a CIS image frame may correspond to the scan time of the last line or the exposure completion time of the image frame. The payload of such embedded event packets can be the frame ID (or frame count) of the image sensor, which typically starts at 0 at the first image frame and increments at the start or end of each subsequent image frame.

[0214] Referring to embedded event packets 1480j and 1480k, each has a similar structure, wherein the packet extension field 1487 corresponding to the embedded event packet identifier packet 1414 and the data type field 1483 corresponding to the embedded event packet header packet 1482 specify the corresponding data type (for example, for embedded event packet 1480j, DT=9, indicating that the content in embedded event packet data packets 1484a and 1484b is mixed CIS frame start information; and for embedded event packet 1480k, DT=10, indicating that the content in embedded event packet data packets 1484a and 1484b is mixed CIS frame end information). The word count field 1483 of embedded event packets 1480j and 1480k indicates that each of embedded event packets 1480j and 1480k contains two bytes of embedded event packet data packet 1484.

[0215] Referring back to Table 1 above, data types belonging to the Special CD Event Type class can include data types of events generated by EVS pixels of the EVS image sensor. The distribution of events generated by EVS pixels can be highly scene-dependent. In some specially distributed event scenarios, it may be more efficient to encode events detected by EVS pixels using embedded event packets 1480 instead of AER data packets using the encoding format 1400 described above. Special CD Event Types can be defined to achieve this. Each Special CD Event Type can refer to a specially distributed event pattern and a corresponding specially designed encoding scheme. Table 3 below lists several instance data types (and corresponding instance data type values ​​and / or corresponding word count values) that can be included in the Special CD Event Type class:

[0216]

[0217]

[0218] Table 3

[0219] Figure 14F The various embodiments of this technology can correspond to the data types listed in Table 3 above and can be described as follows: Figure 14 The encoding format 1400 is used to encode and convey event information in instances of embedded event groups 1480l to 1480o. More specifically, Figure 14FThe embedded event group 1480l, which can be used to convey in-line vector event information, (ii) the embedded event group 1480m, which can be used to convey line vector event information, (iii) the embedded event group 1480n, which can be used to convey run length event information, and (iv) the embedded event group 1480o, which can be used to convey Diff event information, are described below. Embedded event groups 1480l and 1480m can be used for group-based coding. Instance configurations of embedded event groups 1480l and 1480m are described in Tables 4 and 5 below, respectively.

[0220] Referring first to embedded event packet 1480m (which can be used to encode an entire line event into a group), two bytes of embedded event packet data packet 1484 (e.g., embedded event packet data packets 1484a and 1484b) can be used to specify a 13-bit Y address and line vector polarity. In the illustrated embodiment, embedded event packet data packet 1484a contains MSB bits with a value of '0', followed by two bits for encoding the line vector polarity, and then five bits for encoding the MSB bits [12:8] of the Y address. Additionally, embedded event packet data packet 1484b contains eight bits for encoding the LSB bits [7:0] of the Y address. The line vector polarity can be used to specify the type of event included in the corresponding group (e.g., on-only event, off-only event, on and off event). Additional embedded event packet data packets 1484c to 1484n can be used to encode the line vector of events detected by pixel groups. Events can be encoded in the line vector using one or more bits (e.g., two bits). In this way, the embedded event group 1480m can be used to encode and convey one or more events detected by one or more pixels in the same row. Figure 15 This section describes various embodiments of the present technology, including units and other multi-bit techniques for encoding event polarity information for in-line vector event encoding. Additionally, although described above in the context of line-by-line encoded events, the embedded event group 1480 can also be used to encode events detected by pixels belonging to multiple rows / lines.

[0221]

[0222] Table 4

[0223] Referring to embedded event packet 1480l (which can be used to encode events in a pixel group as part of a line), two bytes of embedded event packet data packet 1484 (e.g., embedded event packet data packets 1484a and 1484b) can be used to specify a 13-bit X address and in-line vector polarity. In the illustrated embodiment, embedded event packet data packet 1484a contains MSB bits with a value of '0', followed by two bits encoding the in-line vector polarity, followed by 5 bits for encoding the MSB bits [12:8] of the X address. The in-line vector polarity can be used to specify the type of event in the group (e.g., on-only event, off-only event, on and off event). Additionally, embedded event packet data packet 1484b contains 8 bits for encoding the LSB bits [7:0] of the X address. Additional embedded event packet data packets 1484c to 1484n can be used to encode the in-line vector of events detected by the pixel group. Events can be encoded in the line vector using one or more bits (e.g., two bits). The embedded event packet 1480l can be combined with one or more other data packets (such as row address MSB packet 1402, row address LSB packet 1404, and / or row address extended packet 1406) to encode and transmit group event information. More specifically, one or more other data packets can be used to specify a Y address or a block of Y address, and the embedded event packet 1480l can be used to encode and convey one or more events detected by one or more pixels in the same column. Figure 15 This section describes various embodiments of the present technology, including units and multi-bit techniques for encoding event polarity information for line vector event encoding. Additionally, although described above in the context of line-by-line encoded events, the embedded event group 1480 can also be used to encode events detected by pixels belonging to multiple rows / lines.

[0224]

[0225]

[0226] Table 5

[0227] Referring now to embedded event packet 1480n, embedded event packet 1480n can be used for run-length encoding of event information. Instance configurations of embedded event packet 1480n are described in Table 6 below. As shown, two bytes of embedded event packet data packet 1484 (e.g., embedded event packet data packets 1484a and 1484b) can be used to specify (i) the 13-bit X address of the pixel at the start of the run of the detected event and (ii) the polarity of the event. In the illustrated embodiment, embedded event packet data packet 1484a contains MSB bits with a value of '0', followed by two bits encoding the event polarity, and then 5 bits used to encode the MSB bits [12:8] of the starting pixel's X address. Additionally, embedded event packet data packet 1484b contains 8 bits used to encode the LSB bits [7:0] of the starting pixel's X address. Additionally, two bytes of the embedded event packet data packet 1484 (e.g., embedded event packet data packets 1484c and 1484d) can be used to specify (i) the 13-bit X address of the pixel at the end of the run of the detected event and (ii) the polarity of the event. In the illustrated embodiment, embedded event packet data packet 1484c includes an MSB bit with a value of '0', followed by two bits encoding the event polarity, and then five bits for encoding the MSB bits [12:8] of the end pixel's X address. Additionally, embedded event packet data packet 1484b includes eight bits for encoding the LSB bits [7:0] of the end pixel's X address. In some embodiments, embedded event packet 1480n can be used in conjunction with one or more other packets (e.g., row address MSB packet 1402, row address LSB packet 1404, and / or row address extension packet 1406). More specifically, one or more other packets can be used to specify the Y address, and embedded event packet 1480n can be used for run-length encoding of events within the row.

[0228]

[0229] Table 6

[0230] Referring now to embedded event packet 1480o, embedded event packet 1480o can be used for pixel-level encoding and encoding of each distinct event address. Instance configurations of embedded event packet 1480o are described in Table 7 below. In Table 7, "distinct event address" can refer to a group of events generated simultaneously by pixels spaced at most 16 pixels apart in the X direction. For example, several bytes of embedded event packet data packet 1484 (e.g., embedded event packet data packets 1484a to 1484i) can be used to encode the polarity of events detected for pixel groups (e.g., a row, a subset of a row), and a corresponding number of bytes of embedded event packet data packet 1484 (e.g., embedded event packet data packets 1484j to 1484n) can be used to encode neighboring pixel address difference information regarding the X address of the pixel corresponding to the encoded event. In each byte of embedded event packet data packet 1484 used to encode and convey X address information, the provided X address information can be relative to the starting pixel of a pixel with a known address. For example, in the embedded event packet data packet 1484i, 4 bits can be used to convey the X address information of the first pixel, and another 4 bits can be used to convey the X address of the second pixel. The first and second pixels each serve as pixels that detect events whose polarities are encoded in the embedded event packet data packet 1484a. The 4 bits encoding the X address information of the first pixel in the embedded event packet data packet 1484i can encode a first difference (or first difference quantity) between the X address of the first pixel and the starting / known pixel address. Similarly, the 4 bits encoding the X address information of the second pixel in the embedded event packet data packet 1484i can encode a second difference (or second difference quantity) between the X address of the second pixel and the starting / known pixel address. In this way, the embedded event packet 1480o can be used to convey several different events using pixel-level encoding. It is expected that Diff event encoding using embedded event packets will be particularly efficient when handling scenarios with low-density, nearly uniformly distributed events in the same line.

[0231]

[0232]

[0233] Table 7

[0234] For each of the embedded event packets 1480l to 1480p, the corresponding data type is specified in the packet extension field 1487 of the embedded event packet identifier packet 1414 and the data type field 1483 of the embedded event packet header packet 1482 (for example, for embedded event packet 1480l, DT=0x10, indicating that the content of embedded event packet data packets 1484a to 1484n is line vector event information; for embedded event packet 1480m, DT=0x11, indicating that the content of embedded event packet data packets 1484a to 1484n is line vector event information; for embedded event packet 1480n, DT=0x12, indicating that the content of embedded event packet data packets 1484a to 1484d is run length event information; and for embedded event packet 1480o, DT=0x13, indicating that the content of embedded event packet data packets 1484a to 1484n is Diff event information). The word count field 1483 of embedded event packet 1480n indicates that embedded event packet 1480n contains 4 bytes of embedded event packet data packet 1484. The word count field 1483 of embedded event packets 1480l, 1480m, and 1480o indicates that each of embedded event packets 1480l, 1480m, and 1480o contains N bytes of embedded event packet data packet 1484 (e.g., depending on the number of inline vectors, line vectors, and polarity / Diff X addresses used to encode and convey event information in these embedded event packets 1480l, 1480m, and 1480n, respectively). Alternatively, in embodiments where the word count (WC) is defined by the number of pixels in a line, since line-by-line encoding is used with embedded event packet 1480m, the word count field 1483 of embedded event packet 1480m can be set to WC=0.

[0235] In some embodiments, except Figures 14 to 14F In addition to the data packets described in the document, Figure 14 Encoding format 1400 may contain one or more other or additional data packets. For example, encoding format 1400 may contain padding packets (not shown) in which all 8 bits [7:0] are assigned the value '1'. In other words, a padding packet may be an 8-bit 0xFF packet. Padding packets can be used to pad encoded data to a specific length or as escape packets for another advanced protocol.

[0236] C. Example

[0237] Several aspects of this technology are set forth in the following examples. These examples are provided for illustrative purposes only and are not intended to limit the technology in any way. In fact, it should be noted that all or a subset of any one or more of the following examples may be combined (in any combination) with (i) all or a subset of any or more of the other examples below and / or (ii) any or more aspects of the technology described above. Furthermore, all or a subset of any one or more of the appended examples below may be incorporated into the corresponding independent examples below. Moreover, although aspects of this technology are set forth below in examples relating to specific categories (e.g., systems, devices, methods, or computer-readable media), those aspects may similarly be set forth in other examples relating to any of other categories (e.g., systems, devices, methods, and / or computer-readable media). Such other examples, though omitted below for brevity, form part of this disclosure.

[0238] Example 1. An encoding format for transmitting event data from an event-based vision sensor, the encoding format comprising: a row address most significant bit (MSB) data packet containing a first prefix code and content bits for encoding MSB row address information; a row address least significant bit (LSB) data packet containing a second prefix code and content bits for encoding LSB row address information, wherein the second prefix code is different from the first prefix code; a column address MSB data packet containing a third prefix code and content bits for encoding MSB column address information, wherein the third prefix code is different from the first and second prefix codes; and a column address LSB data packet containing a fourth prefix code and content bits for encoding LSB column address information, wherein the fourth prefix code is different from the first, second, and third prefix codes, wherein at least one of the first, second, third, and fourth prefix codes has a length different from at least one of the other of the first, second, third, and fourth prefix codes.

[0239] Example 2. According to the encoding format described in Example 1, it further includes an embedded event packet containing an embedded packet identifier data packet having a fifth prefix code that is different from the first, second, third and fourth prefix codes.

[0240] Example 3. According to the encoding format described in Example 1 or Example 2, the embedded event packet further includes an embedded packet header data packet and one or more embedded packet data packets.

[0241] Example 4. According to the encoding format described in any of Examples 1 to 3, wherein the embedded packet header data packet includes a data type field that can be used to specify the content type of the payload in the embedded event packet and a word count field that can be used to indicate the payload size of the embedded event packet.

[0242] Example 5. According to the encoding format described in any of Examples 1 to 4, the data type field can be used to specify relative timestamp information and / or absolute timestamp information, and the one or more embedded packet data packets can be used to encode the relative timestamp data and / or absolute timestamp information.

[0243] Example 6. According to the encoding format described in any of Examples 1 to 5, the data type field can be used to specify timestamp LSB change information and / or timestamp MSB change information, and the one or more embedded packet data packets can be used to encode the least significant bit and / or the most significant bit of the timestamp.

[0244] Example 7. According to the encoding format described in any of Examples 1 to 6, wherein the data type field can be used to specify external trigger information, and wherein the one or more embedded packet data packets can be used to encode the trigger identifier and / or polarity indicator.

[0245] Example 8. According to the encoding format described in any of Examples 1 to 7, wherein the one or more embedded packet data packets can be used to encode the trigger count.

[0246] Example 9. According to the encoding format described in any of Examples 1 to 8, wherein the data type field can be used to specify MIPI frame start information and / or MIPI frame end information, and wherein the one or more embedded packet data packets can be used to encode MIPI frame identifiers.

[0247] Example 10. According to the encoding format described in any of Examples 1 to 9, wherein the data type field can be used to specify event frame start information and / or event frame end information, and wherein the one or more embedded packet data packets can be used to encode the event frame identifier.

[0248] Example 11. An encoding format according to any of Examples 1 to 10, wherein the data type field can be used to specify external metadata insertion information and / or internal metadata reporting information.

[0249] Example 12. According to the encoding format described in any of Examples 1 to 11, wherein the data type field can be used to specify mixed CIS frame start information and / or mixed CIS frame end information, and wherein the one or more embedded packet data packets can be used to encode the CIS frame identifier.

[0250] Example 13. According to the encoding format described in any of Examples 1 to 12, wherein the data type field can be used to specify run length event information, and wherein the one or more embedded packet data packets can be used to encode the start X address of the pixel at the start of the run of the detected event, the end X address of the pixel at the end of the run of the detected event, and the polarity of the detected event.

[0251] Example 14. An encoding format according to any of Examples 1 to 13, wherein the data type field can be used to specify inline vector event information, and wherein the one or more embedded group data packets can be used to encode the X address, the inline vector polarity of the type of event in the specified group, and the inline vector of the event detected by the pixel group.

[0252] Example 15. According to the encoding format described in any of Examples 1 to 14, the embedded group identifier data packet includes an extended field, and the extended field is not used; or the extended field can be used to at least partially indicate the data type of the data contained in the embedded group identifier data packet, can be used to encode timestamp information, and / or can be used to indicate when a timer is full or exhausted.

[0253] Example 16. According to the encoding format described in any of Examples 1 to 15, wherein the embedded event packet does not contain an end identifier data packet, and wherein the word count value in the embedded packet header data packet can be used to determine the end of the embedded event packet.

[0254] Example 17. An encoding format according to any of Examples 1 to 16, wherein the encoding format implements a packet sharing method, wherein events detected by two or more pixels in the same row share MSB packets and LSB packets at the same row address.

[0255] Example 18. An encoding format according to any of Examples 1 to 17, wherein the encoding format implements a packet sharing method, wherein events detected by two or more pixels having the same column address MSB share packets with the same column address MSB.

[0256] Example 19. An encoding format according to any of Examples 1 to 18, wherein the encoding format implements a packet sharing method, wherein events detected by two or more pixels of a row having the same row address MSB share a packet with the same row address MSB.

[0257] Example 20. An encoding format according to any of Examples 1 to 19, wherein the column address LSB data packet contains column embedded bits for encoding event polarity information.

[0258] Example 21. An encoding format according to any of Examples 1 to 20, wherein the encoding format uses a unit representation for encoding events of a single polarity type to encode event polarity information.

[0259] Example 22. An encoding format according to any of Examples 1 to 21, wherein the encoding format uses a multi-bit representation for encoding events of multiple polarity types to encode event polarity information.

[0260] Example 23. An encoding format according to any of Examples 1 to 22, wherein the multiple bits represent using two bits per pixel to encode no event, off event, on event, or both off and on events.

[0261] Example 24. A method for encoding event data from an event-based vision sensor, the method comprising: encoding row address information using row address MSB packets and row address LSB packets, the row address MSB packets including a first prefix code and the row address LSB packets including a second prefix code different from the first prefix code; and encoding column address information using column address MSB packets and column address LSB packets, the column address MSB packets including a third prefix code and the column address LSB packets including a fourth prefix code different from the third prefix code, wherein at least one of the first, second, third, and fourth prefix codes has a length different from the other of at least one of the first, second, third, and fourth prefix codes.

[0262] Example 25. The method according to Example 24 further includes selectively transmitting embedded event packets including embedded packet identifier packets with a fifth prefix code.

[0263] Example 26. The method according to Example 24 or Example 25 further includes encoding timestamp information into the embedded event group.

[0264] Example 27. The method according to any one of Examples 24 to 26, further comprising encoding external trigger information in the embedded event group, wherein the external trigger information includes a trigger identifier, a polarity indicator, a trigger count, or any combination thereof.

[0265] Example 28. The method according to any one of Examples 24 to 27, further comprising encoding MIPI frame start information or MIPI frame end information in the embedded event packet.

[0266] Example 29. The method according to any one of Examples 24 to 28, further comprising encoding event frame start information or event frame end information in the embedded event group.

[0267] Example 30. The method according to any one of Examples 24 to 29, further comprising encoding hybrid CIS frame start information or hybrid CIS frame end information into the embedded event packet.

[0268] Example 31. The method according to any of Examples 24 to 30, further comprising sharing the row address MSB packet and the row address LSB packet in events detected by two or more pixels in the same row.

[0269] Example 32. The method according to any of Examples 24 to 31, further comprising sharing the column address MSB data packet in events detected by two or more pixels having the same column address MSB.

[0270] Example 33. The method according to any of Examples 24 to 32, further comprising sharing the row address MSB data packet in events detected by two or more pixels of a row having the same row address MSB.

[0271] Example 34. An event-based vision sensor system comprising: an event-based vision sensor pixel array configured to detect events; and an event signal processor configured to encode the detected events using an encoding format including coordinate data packets, the coordinate data packets including row address most significant bit (MSB) data packets, row address least significant bit (LSB) data packets, column address MSB data packets, and column address LSB data packets, wherein each coordinate data packet includes a variable-length prefix code for data packet identification, and at least two of the variable-length prefix codes have different lengths from each other.

[0272] Example 35. The event-based vision sensor system according to Example 34, wherein the encoding format further includes embedded event packets, each of the embedded event packets including an embedded packet identifier data packet with a prefix code, an embedded packet header data packet, and an embedded packet data data packet.

[0273] Example 36. An event-based vision sensor system according to Example 34 or Example 35, wherein the embedded packet header data packet includes a data type field specifying the content type of the payload and a word count field indicating the size of the payload.

[0274] Example 37. An event-based visual sensor system according to any of Examples 34 to 36, wherein the event signal processor is configured to implement a packet sharing method, wherein events detected by two or more pixels in the same row share a row address MSB packet and a row address LSB packet; events detected by two or more pixels with the same column address MSB share a column address MSB packet; and / or events detected by two or more pixels in a row with the same row address MSB share a row address MSB packet.

[0275] Example 38. An event-based visual sensor system according to any of Examples 34 to 37, further comprising a MIPI transmitter configured to transmit encoded event data.

[0276] Example 39. An encoding format for transmitting event data from an event-based vision sensor, the encoding format comprising: a row address most significant bit (MSB) data packet, which includes a first prefix code and content bits for encoding MSB row address information; a row address least significant bit (LSB) data packet, which includes a second prefix code and content bits for encoding LSB row address information, wherein the second prefix code is different from the first prefix code; and a column address MSB data packet, which includes a third prefix code and content bits for encoding MSB column address information, wherein the third prefix code is different from the first and second prefix codes. The encoding format further includes: a row appended data packet, which contains a fourth prefix code and content bits for encoding LSB column address information, wherein the fourth prefix code is different from the first, second, and third prefix codes, and wherein the encoding format further includes: one or more row appended data packets, which can be located in the encoded event stream after the row address LSB data packet, wherein the one or more row appended data packets can be used to encode row-by-row information; one or more column appended data packets, which can be located in the encoded event stream after the column address LSB data packet, wherein the one or more column appended data packets can be used to encode pixel-by-pixel information; or a combination thereof.

[0277] Example 40. According to the encoding format described in Example 39, wherein the first, second, third and fourth prefix codes are assigned at least in part based on packet frequency, wherein more frequently occurring packets receive shorter prefix codes.

[0278] Example 41. An encoding format according to Example 39 or Example 40, wherein the encoding format includes the one or more line-attached data packets, and wherein the line-by-line information includes timestamp information shared by events that can be detected by pixels in the same line.

[0279] Example 42. An encoding format according to any of Examples 39 to 41, wherein the encoding format includes the one or more column-attached data packets, and wherein the pixel-by-pixel information includes a pixel group vector representing event information of a pixel group.

[0280] Example 43. An encoding format according to any of Examples 39 to 42, wherein the pixel group vector encodes the event state and polarity information of each pixel within the pixel group.

[0281] Example 44. An encoding format according to any of Examples 39 to 43, wherein the column address LSB data packet contains half-valid bits that can be used to indicate which rows within the pixel group have detected an event.

[0282] Example 45. An encoding format according to any of Examples 39 to 44, wherein the column address LSB packet contains one or more column embedded bits that can be used to convey event polarity information.

[0283] Example 46. An encoding format according to any of Examples 39 to 45, wherein the column address LSB packet includes one or more column embedded bits that can be used as additional column address bits to extend the addressable range of the column address.

[0284] Example 47. An encoding format according to any of Examples 39 to 46, wherein the encoding format includes the one or more column-attached data packets, and wherein the one or more column-attached data packets can be used to encode timestamp information of each event detected by each pixel.

[0285] Example 48. An encoding format according to any of Examples 39 to 47, wherein the column address LSB packet contains an isolation flag that can be used to distinguish between isolated events and non-isolated events.

[0286] Example 49. An encoding format according to any of Examples 39 to 48, wherein the isolation flag can be used to indicate the start or end of an event run.

[0287] Example 50. An encoding format according to any of Examples 39 to 49, wherein the row address LSB packet includes an MSB flag that can be used to indicate whether the preceding packet is the row address MSB packet.

[0288] Example 51. An encoding format according to any of Examples 39 to 50, wherein the encoding format supports a line-by-line timestamp mode, wherein the one or more line appended data packets can be used to encode timestamp information shared by events detected by pixels in the same line.

[0289] Example 52. An encoding format according to any of Examples 39 to 51, wherein the encoding format supports an adaptive line-by-line timestamp output mode, wherein timestamp information is selectively encoded at least in part based on event distribution.

[0290] Example 53. An encoding format according to any of Examples 39 to 52, wherein the encoding format includes the one or more line appended data packets, and wherein the line-by-line information includes an exposure flag indicating whether the event occurred within or outside the exposure period.

[0291] Example 54. An encoding format according to any of Examples 39 to 53, wherein the row address LSB packet includes top-bottom bits indicating whether the pixel row is located in the upper or lower half of the pixel array segment.

[0292] Example 55. An encoding format according to any of Examples 39 to 54, further comprising an embedded event packet containing an embedded packet identifier data packet having a fifth prefix code different from the first, second, third, and fourth prefix codes.

[0293] Example 56. The encoding format according to any of Examples 39 to 55, wherein the embedded event packet further includes an embedded packet header data packet and one or more embedded packet data packets.

[0294] Example 57. An encoding format according to any of Examples 39 to 56, wherein the embedded event grouping is used, the encoding format supporting line vector event encoding, wherein events detected by pixels on the same line are encoded together.

[0295] Example 58. An encoding format according to any of Examples 39 to 57, wherein the embedded packet header data packet can be used to specify the data type of the line vector event information; and the one or more embedded packet data packets can be used to encode: (a) the Y address of the event line, (b) the line vector polarity indicator, and (c) the line vector encoding the event information of the pixels along the line.

[0296] Example 59. An encoding format according to any of Examples 39 to 58, wherein the line vector polarity indicator can be used to specify whether the line contains only on events, only off events, or both on and off events.

[0297] Example 60. An encoding format according to any of Examples 39 to 59, wherein the embedded event packets are used, the encoding format supporting differential event encoding, wherein the embedded packet header data packets can be used to specify the data type indicating differential event information; and the one or more embedded packet data packets can be used to encode: (a) polarity information of multiple events and (b) differential X address information of each event relative to a previous pixel address.

[0298] Example 61. A method for encoding event data from an event-based visual sensor, the method comprising: encoding row address information using row address MSB packets and row address LSB packets, the row address MSB packets containing a first prefix code and the row address LSB packets containing a second prefix code different from the first prefix code; encoding column address information using column address MSB packets and column address LSB packets, the column address MSB packets containing a third prefix code and the column address LSB packets containing a fourth prefix code different from the third prefix code, wherein the method further comprises encoding row-by-row information using one or more row append packets located after the row address LSB packets, encoding pixel-by-pixel information using one or more column append packets located after the column address LSB packets, or a combination thereof.

[0299] Example 62. The method according to Example 61, wherein the method includes encoding the line-by-line information, and wherein encoding the line-by-line information includes encoding timestamp information shared by events that can be detected by pixels in the same line.

[0300] Example 63. The method according to Example 61 or Example 62, wherein the method includes encoding the line-by-line information, and wherein encoding the line-by-line information includes encoding whether the event occurred within or outside the exposure period using an exposure flag.

[0301] Example 64. The method according to any one of Examples 61 to 63, wherein the method includes encoding the pixel-by-pixel information, and wherein encoding the pixel-by-pixel information includes encoding a pixel group vector representing event information of a group of pixels.

[0302] Example 65. The method according to any of Examples 61 to 64, further comprising encoding which rows within the pixel group have detected events using half of the valid bits of the column address LSB packet.

[0303] Example 66. The method according to any of Examples 61 to 65, further comprising encoding event polarity information using one or more column embedded bits of the column address LSB packet.

[0304] Example 67. The method according to any one of Examples 61 to 66, wherein the method includes encoding the pixel-by-pixel information, and wherein encoding the pixel-by-pixel information includes encoding timestamp information of each event detected by each pixel using the one or more column-attached data packets.

[0305] Example 68. The method according to any of Examples 61 to 67, further comprising encoding an indication of whether the corresponding event is an isolated event or a non-isolated event using the isolation flag of the column address LSB packet.

[0306] Example 69. The method according to any of Examples 61 to 68, further comprising encoding an indication of whether a preceding data packet is a row address MSB data packet using the MSB flag of the row address LSB data packet.

[0307] Example 70. The method according to any of Examples 61 to 69, further comprising encoding an indication of whether a pixel row is located in the upper or lower half of a pixel array segment using the top-bottom bits of the row address LSB packet.

[0308] Example 71. The method according to any one of Examples 61 to 70, further comprising encoding line vector event information using embedded event grouping, wherein the line vector event information includes (a) the Y address of the event line, (b) the line vector polarity indicator and (c) a line vector encoding event information of pixels along the line.

[0309] Example 72. The method according to any one of Examples 61 to 71 further includes encoding differential event information using embedded event groups, wherein the differential event information includes (a) polarity information of a plurality of events and (b) differential X address information of each event relative to a previous pixel address.

[0310] Example 73. An event-based vision sensor system comprising: an event-based vision sensor pixel array configured to detect events; and an event signal processor configured to encode the detected events using an encoding format comprising coordinate data packets, the coordinate data packets comprising row address most significant bit (MSB) data packets, row address least significant bit (LSB) data packets, column address MSB data packets, and column address LSB data packets, wherein each coordinate data packet contains a prefix code for data packet identification, and wherein the encoding format further comprises one or more row appended data packets for encoding row-by-row information, one or more column appended data packets for encoding pixel-by-pixel information, or a combination thereof.

[0311] Example 74. An event-based visual sensor system according to Example 73, wherein the encoding format includes the one or more row-attached data packets, and wherein the row-by-row information includes timestamp information shared by events that can be detected by pixels in the same row.

[0312] Example 75. An event-based visual sensor system according to Example 73 or Example 74, wherein the encoding format includes the one or more line-by-line data packets, and wherein the line-by-line information includes an exposure flag indicating whether the event occurred within or outside the exposure period.

[0313] Example 76. An event-based visual sensor system according to any of Examples 73 to 75, wherein the encoding format includes the one or more column-attached data packets, and wherein the pixel-by-pixel information includes a pixel group vector representing event information of a group of pixels.

[0314] Example 77. An event-based visual sensor system according to any of Examples 73 to 76, wherein the encoding format includes the one or more column-attached data packets, and wherein the one or more column-attached data packets can be used to encode timestamp information of each event detected by each pixel.

[0315] Example 78. An event-based visual sensor system according to any of Examples 73 to 77, wherein the column address LSB data packet contains half-valid bits that can be used to indicate which rows within a pixel group have detected an event.

[0316] Example 79. An event-based vision sensor system according to any of Examples 73 to 78, wherein the column address LSB data packet contains one or more column embedded bits that can be used to convey event polarity information.

[0317] Example 80. An event-based visual sensor system according to any of Examples 73 to 79, wherein the column address LSB packet contains an isolation flag that can be used to distinguish between isolated events and non-isolated events.

[0318] Example 81. An event-based visual sensor system according to any of Examples 73 to 80, wherein the row address LSB packet includes an MSB flag that can be used to indicate whether the preceding packet is the row address MSB packet.

[0319] Example 82. An encoding format for transmitting event data from an event-based vision sensor, the encoding format comprising: a row address most significant bit (MSB) data packet, which includes a first prefix code and content bits for encoding MSB row address information; a row address least significant bit (LSB) data packet, which includes a second prefix code and content bits for encoding LSB row address information, wherein the second prefix code is different from the first prefix code; and a column address MSB data packet, which includes a third prefix code and content bits for encoding MSB column address information, wherein the third prefix code is different from the first and second prefix codes. The prefix codes are different; and the column address LSB data packet contains a fourth prefix code and content bits for encoding LSB column address information, wherein the fourth prefix code is different from the first, second, and third prefix codes, wherein the encoding format further includes: a column append data packet, which can be located in the encoded event stream after the column address LSB data packet; a row append data packet, which can be located in the encoded event stream after the row address LSB data packet; an embedded event packet, which includes an embedded packet identifier data packet having a fifth prefix code different from the first, second, third, and fourth prefix codes; or any combination thereof.

[0320] Example 83. The encoding format according to Example 82, wherein the encoding format includes the column-attached data packet, and wherein the column-attached data packet can be used to encode the timestamp information of each event detected by each pixel.

[0321] Example 84. An encoding format according to Example 82 or Example 83, wherein the encoding format includes the column appended data packet, and wherein the encoding format supports an in-pixel timestamp mode, wherein the column appended data packet encodes timestamp information for individual events.

[0322] Example 85. An encoding format according to any of Examples 82 to 84, wherein in the intra-pixel timestamp mode, the column append data packet is followed by the column address MSB data packet, another column address LSB data packet, the row address MSB data packet, the row address LSB data packet, or an embedded packet identifier data packet.

[0323] Example 86. An encoding format according to any of Examples 82 to 85, wherein the encoding format includes the row appended data packet and the column appended data packet, wherein the row appended data packet can be used to encode row-by-row timestamp information, and the column appended data packet can be used to encode column-by-column timestamp information simultaneously.

[0324] Example 87. An encoding format according to any of Examples 82 to 86, wherein the encoding format supports a frame-by-frame timestamp mode, wherein a single timestamp is output for each frame of event-based visual sensor data.

[0325] Example 88. An encoding format according to any of Examples 82 to 87, wherein the encoding format includes the line append data packet, and wherein in the frame-by-frame timestamp mode, the line append data packet containing timestamp information of the entire frame follows the line address LSB data packet at the beginning of the entire frame.

[0326] Example 89. An encoding format according to any of Examples 82 to 88, wherein the column address LSB data packet includes column embedded bits including polarity bits, isolation flag bits, or combinations thereof.

[0327] Example 90. An encoding format according to any of Examples 82 to 89, wherein the isolation flag indicates whether an event was also detected in an adjacent pixel.

[0328] Example 91. An encoding format according to any of Examples 82 to 90, wherein the encoding format supports event run-length encoding, wherein only the first and last events of a consecutive event sequence are encoded using column address LSB packets.

[0329] Example 92. An encoding format according to any of Examples 82 to 91, wherein the encoding format is configured such that the run-length pixel between the first event and the last event is not streamed, and such that the decoder can identify the run-length pixel at least in part based on the first and last column address LSB packets.

[0330] Example 93. An encoding format according to any of Examples 82 to 92, wherein the column address LSB data packet further includes a half-valid bit, and wherein the encoding format includes one or more column append data packets that can be located after the column address LSB data packet, wherein the half-valid bit can be used to indicate which rows within a pixel group have detected an event and can be used to identify how much of the one or more column append data packets have been transmitted.

[0331] Example 94. An encoding format according to any of Examples 82 to 93, wherein the half-valid bit comprises at least two bits, and wherein a first value of the half-valid bit indicates that no event was detected in the top pixel row of the pixel group, a second value indicates that no event was detected in the bottom pixel row of the pixel group, and a third value indicates that an event was detected in both rows of the pixel group.

[0332] Example 95. An encoding format according to any of Examples 82 to 94, wherein when the half-valid bit indicates that no event was detected in the top pixel row, the encoding format is configured such that no column append data packet is sent for the top row, and only column append data packets for the bottom row are transmitted.

[0333] Example 96. An encoding format according to any of Examples 82 to 95, wherein the row address LSB packet includes top-bottom bits indicating whether the pixel row is located in the upper or lower half of the pixel array segment.

[0334] Example 97. The encoding format according to any of Examples 82 to 96, wherein the column address MSB data packet further includes an exposure flag indicating whether the event occurred within or outside the CMOS image sensor exposure period.

[0335] Example 98. An encoding format according to any of Examples 82 to 97, wherein the encoding format includes the column-attached data packet, and wherein the column-attached data packet can be used to encode a pixel group vector that encodes event information of a group of pixels arranged in multiple rows and a column.

[0336] Example 99. An encoding format according to any of Examples 82 to 98, wherein the event state of each pixel within the pixel group is represented by at least two bits, and wherein the at least two bits encode no event, positive polarity event, or negative polarity event.

[0337] Example 100. An encoding format according to any of Examples 82 to 99, wherein the column address LSB data packet includes column embedded bits that can be used as column address bits to increase addressable resolution.

[0338] Example 101. An encoding format according to any of Examples 82 to 100, wherein the encoding format is configured such that event groups in which no pixel detection event is performed are skipped to reduce the amount of data encoded and transmitted.

[0339] Example 102. An encoding format according to any of Examples 82 to 101, wherein the encoding format includes the row appended data packet and the embedded event packet, wherein the row appended data packet can be used to encode timestamp information shared by events detected by pixels in the same row, and wherein the embedded event packet can be used to encode MSB timestamp information, while the row appended data packet encodes LSB timestamp information.

[0340] Example 103. A method for encoding event data from an event-based vision sensor, the method comprising: encoding row address information using a row address most significant bit (MSB) data packet containing a first prefix code and a row address least significant bit (LSB) data packet containing a second prefix code different from the first prefix code; encoding column address information using a column address MSB data packet containing a third prefix code different from the first and second prefix codes and a column address LSB data packet containing a fourth prefix code different from the first, second, and third prefix codes; and encoding additional event information using column append data packets, row append data packets, embedded event packets, or any combination thereof.

[0341] Example 104. The method according to Example 103, wherein encoding additional event information includes encoding timestamp information for each event using the column append data packet.

[0342] Example 105. The method according to Example 103 or Example 104, wherein encoding the additional event information includes encoding the row-by-row timestamp information using the row append data packet, while simultaneously encoding the column-by-column timestamp information using the column append data packet.

[0343] Example 106. The method according to any of Examples 103 to 105, wherein encoding additional event information includes encoding a single timestamp of the frame at the beginning of each frame of event-based visual sensor data using the row appended data packet.

[0344] Example 107. The method according to any one of Examples 103 to 106, further comprising: encoding event polarity information in the column embedded bits of the column address LSB data packet; encoding an indication of whether an adjacent pixel has also detected an event in the column embedded bits of the column address LSB data packet; or a combination thereof.

[0345] Example 108. The method according to any of Examples 103 to 107, further comprising implementing event run-length encoding by using column address LSB packets to encode only the first and last events of a continuous event sequence, wherein the run-length pixels between the first event and the last event are not streamed.

[0346] Example 109. The method according to any of Examples 103 to 108, wherein the column address LSB data packet includes half-valid bits, and wherein the method further includes encoding an indication of which rows within a pixel group have detected events using the half-valid bits; and selectively transmitting one or more column append data packets based on the half-valid bits, wherein column append data packets for rows where no events were detected are skipped.

[0347] Example 110. The method according to any of Examples 103 to 109, further comprising encoding an indication of whether the event occurred within or outside the CMOS image sensor exposure period using an exposure flag in the column address MSB data packet.

[0348] Example 111. The method according to any one of Examples 103 to 110, further comprising: dividing a pixel array into pixel groups, each pixel group containing a plurality of pixels arranged in a plurality of rows and a column; encoding events detected by pixels of a pixel group together in a column-attached data packet as a pixel group vector; and skipping pixel groups in which no pixels detected events.

[0349] Example 112. The method according to any of Examples 103 to 111, wherein encoding the additional event information includes encoding the MSB timestamp information using the embedded event packet and encoding the LSB timestamp information using the line append data packet.

[0350] Example 113. An event-based vision sensor system comprising: an event-based vision sensor pixel array configured to detect events; and an event signal processor configured to encode the detected events using an encoding format comprising: a row address most significant bit (MSB) data packet containing a first prefix code and content bits for encoding MSB row address information; a row address least significant bit (LSB) data packet containing a second prefix code and content bits for encoding LSB row address information, wherein the second prefix code is different from the first prefix code; and a column address MSB data packet containing a third prefix code and content bits for encoding MSB... The following are the following: content bits for encoding column address information, wherein the third prefix code is different from the first and second prefix codes; a column address LSB data packet containing a fourth prefix code and content bits for encoding LSB column address information, wherein the fourth prefix code is different from the first, second, and third prefix codes; a column append data packet located after the column address LSB data packet in the encoded event stream; a row append data packet located after the row address LSB data packet in the encoded event stream; an embedded event packet including an embedded packet identifier data packet having a fifth prefix code different from the first, second, third, and fourth prefix codes; or any combination thereof.

[0351] Example 114. An event-based visual sensor system according to Example 113, wherein the encoding format includes the column-attached data packet, and wherein the column-attached data packet can be used to encode timestamp information of each event detected by each pixel.

[0352] Example 115. An event-based visual sensor system according to Example 113 or Example 114, wherein the encoding format includes the row-attached data packet and the column-attached data packet, wherein the row-attached data packet can be used to encode row-by-row timestamp information, and the column-attached data packet can be used to encode column-by-column timestamp information simultaneously.

[0353] Example 116. An event-based visual sensor system according to any of Examples 113 to 115, wherein the encoding format supports a frame-by-frame timestamp mode, wherein a single timestamp is output for each frame of event-based visual sensor data.

[0354] Example 117. An event-based vision sensor system according to any of Examples 113 to 116, wherein the column address LSB data packet includes column embedded bits including polarity bits, isolation flag bits, or combinations thereof.

[0355] Example 118. An event-based visual sensor system according to any of Examples 113 to 117, wherein the event signal processor is configured to implement event run-length encoding by encoding only the first and last events of a continuous event sequence.

[0356] Example 119. An event-based vision sensor system according to any of Examples 113 to 118, wherein the event signal processor is configured to transmit only the first column address LSB data packet of the start pixel and the second column address LSB data packet of the end pixel for run-length events of the same polarity.

[0357] Example 120. An event-based visual sensor system according to any of Examples 113 to 119, wherein the column address LSB data packet further includes a half-valid bit indicating which rows within a pixel group have detected an event, and wherein the event signal processor is configured to selectively transmit one or more column supplementary data packets based on the half-valid bit.

[0358] Example 121. An event-based vision sensor system according to any of Examples 113 to 120, wherein the event-based vision sensor pixel array is divided into pixel groups, each comprising a plurality of pixels arranged in a plurality of rows and a column, and wherein the encoding format includes the column-attached data packet that can be used to encode a pixel group vector representing event information of pixels within the pixel group.

[0359] Example 122. An event-based vision sensor system according to any of Examples 113 to 121, wherein the encoding format includes the row appended data packet and the embedded event packet, wherein the row appended data packet can be used to encode timestamp information shared by events detected by pixels in the same row, and wherein the embedded event packet can be used to encode MSB timestamp information, while the row appended data packet encodes LSB timestamp information.

[0360] Example 123. An encoding format for transmitting event data from an event-based vision sensor, the encoding format comprising: a row address most significant bit (MSB) data packet containing a first prefix code and content bits for encoding MSB row address information; a row address least significant bit (LSB) data packet containing a second prefix code and content bits for encoding LSB row address information, wherein the second prefix code is different from the first prefix code; a column address MSB data packet containing a third prefix code and content bits for encoding MSB column address information, wherein the third prefix code is different from the first and second prefix codes; a column address LSB data packet containing a fourth prefix code and content bits for encoding LSB column address information and column embedding bits, wherein the fourth prefix code is different from the first, second, and third prefix codes; and an embedded event packet comprising an embedded packet identifier data packet, an embedded packet header data packet, and one or more embedded packet data packets, wherein the embedded packet identifier data packet contains a fifth prefix code, and wherein the fifth prefix code is different from the first, second, third, and fourth prefix codes.

[0361] Example 124. According to the encoding format described in Example 123, wherein the first, second, third, fourth and fifth prefix codes are assigned based on the frequency of data packets, wherein more frequently occurring data packets receive shorter prefix codes.

[0362] Example 125. The encoding format according to Example 123 or Example 124 further includes a line append data packet that can be located in the encoded event stream after the line address LSB data packet, wherein the line append data packet can be used to encode line-by-line information.

[0363] Example 126. An encoding format according to any of Examples 123 to 125, wherein the line-by-line information includes timestamp information shared by events that can be detected by pixels in the same line.

[0364] Example 127. The encoding format according to any of Examples 123 to 126 further includes a column append data packet located in the encoded event stream after the column address LSB data packet, wherein the column append data packet can be used to encode pixel-by-pixel information.

[0365] Example 128. An encoding format according to any of Examples 123 to 127, wherein the pixel-by-pixel information includes a pixel group vector representing event information of a pixel group.

[0366] Example 129. An encoding format according to any of Examples 123 to 128, wherein the column embedded bits include at least one bit for encoding event polarity information and at least one bit for encoding column address information.

[0367] Example 130. A method for encoding event data from an event-based vision sensor, the method comprising: encoding row address information using row address MSB packets and row address LSB packets, the row address MSB packets containing a first prefix code and the row address LSB packets containing a second prefix code shorter than the first prefix code; encoding column address information using column address MSB packets and column address LSB packets, the column address MSB packets containing a third prefix code and the column address LSB packets containing a fourth prefix code shorter than the third prefix code; encoding event polarity information, an isolation flag, or one or more column address bits in column embedded bits of the column address LSB packets; and selectively transmitting embedded event packets including embedded packet identifier packets with a fifth prefix code, embedded packet header packets, and one or more embedded packet data packets.

[0368] Example 131. The method according to Example 130 further includes implementing a packet sharing method, wherein packets containing higher address bits are shared across multiple events within the same address segment.

[0369] Example 132. The method according to Example 130 or Example 131, wherein the packet sharing method includes sharing the row address MSB packet and the row address LSB packet of events detected by pixels in the same row.

[0370] Example 133. The method according to any of Examples 130 to 132, further comprising encoding timestamp information in a row append packet located after the row address LSB packet, wherein the timestamp information is shared by events detected by pixels in the same row.

[0371] Example 134. The method according to any of Examples 130 to 133, further comprising encoding pixel group vector information in a column append packet located after the column address LSB packet, wherein the pixel group vector represents event information of the pixel group.

[0372] Example 135. The method according to any of Examples 130 to 134, wherein the pixel group vector information includes the event polarity and event detection state of each pixel in the pixel group.

[0373] Example 136. The method according to any of Examples 130 to 135, wherein the embedded event grouping is used to encode timestamp change events, external trigger events, or frame synchronization events.

[0374] Example 137. An event-based vision sensor system comprising: an event-based vision sensor pixel array configured to detect events; and an event signal processor configured to encode the detected events using an encoding format including coordinate data packets, the coordinate data packets including row address most significant bit (MSB) data packets, row address least significant bit (LSB) data packets, column address MSB data packets, and column address LSB data packets, each coordinate data packet including a variable-length prefix code for data packet identification; and embedded event packets, each including an embedded packet identifier data packet with a prefix code, an embedded packet header data packet, and an embedded packet data data packet.

[0375] Example 138. An event-based visual sensor system according to Example 137, wherein the event signal processor implements a packet sharing method, wherein packets containing higher address bits are shared across multiple events within the same address segment.

[0376] Example 139. An event-based vision sensor system according to Example 137 or Example 138, wherein the packet sharing method includes sharing the row address MSB packet and the row address LSB packet of events detected by pixels in the same row of the event-based vision sensor pixel array.

[0377] Example 140. An event-based visual sensor system according to any of Examples 137 to 139, wherein the encoding format further includes a row append data packet located after the row address LSB data packet, the row append data packet being used to encode timestamp information shared by events detected by pixels in the same row.

[0378] Example 141. An event-based visual sensor system according to any of Examples 137 to 140, wherein the encoding format further includes a column append data packet located after the column address LSB data packet, the column append data packet being used to encode a pixel group vector representing event information of a pixel group.

[0379] Example 142. An event-based visual sensor system according to any of Examples 137 to 141, wherein the pixel group vector includes the event polarity and event detection state of each pixel in the pixel group, and wherein the pixel group comprises a 4×1 pixel arrangement.

[0380] Example 143. An encoding format for transmitting event data from an event-based vision sensor, the encoding format comprising: a row address most significant bit (MSB) data packet, which includes a first prefix code and content bits for encoding MSB row address information; a row address least significant bit (LSB) data packet, which includes a second prefix code and content bits for encoding LSB row address information, wherein the second prefix code is different from the first prefix code; a row address extended data packet, which includes a third prefix code and content bits for specifying additional row address bits of the encoded event, wherein the third prefix code is different from the first and second prefix codes; and a column address MSB data packet, which includes a fourth prefix code and content bits for encoding MSB column address information, wherein... The fourth prefix code is different from the first, second, and third prefix codes; the column address LSB data packet includes a fifth prefix code and content bits for encoding LSB column address information and column embedded bits, wherein the fifth prefix code is different from the first, second, third, and fourth prefix codes; the column address extended data packet includes a sixth prefix code and content bits for specifying additional column address bits for encoded events, wherein the sixth prefix code is different from the first, second, third, and fourth prefix codes; and the embedded event packet identifier data packet includes a seventh prefix and an extended field, wherein the embedded event packet identifier data packet is configured to identify the beginning of an embedded event packet containing an embedded event packet header data packet and one or more embedded event packet data packets.

[0381] Example 144. According to the encoding format described in Example 143, the first prefix code is a 4-bit prefix; and the second prefix code is a 2-bit prefix code.

[0382] Example 145. According to the encoding format described in Example 143 or Example 144, the fourth prefix code is a 3-bit prefix code; and the sixth prefix code is a 1-bit prefix code.

[0383] Example 146. An encoding format according to any of Examples 143 to 145, wherein the column address LSB data packet contains column embedded bits that can be used to encode event polarity information.

[0384] Example 147. An encoding format according to any of Examples 143 to 146, wherein the embedded event packet header data packet includes a data type field and a word count field, and wherein the extended field of the embedded event packet identifier data packet is configured to combine with the data type field to specify the data type of the content in the payload of the embedded event packet.

[0385] Example 148. An encoding format according to any of Examples 143 to 147, wherein the embedded event grouping is configured to encode timestamp information, external trigger information, frame start information, frame end information, metadata insertion information, metadata report information, or any combination thereof.

[0386] Example 149. An encoding format according to any of Examples 143 to 148, wherein the embedded event grouping can be used to encode timestamp least significant bit (LSB) change information or timestamp most significant bit (MSB) change information.

[0387] Example 150. A method for encoding event data from an event-based vision sensor, the method comprising: encoding event information using embedded event packets for transmission in an encoded data stream, wherein encoding the event information includes specifying the data type of the event information using (i) a data type extension field of an embedded event packet identifier packet and (ii) a data type field of an embedded event packet header packet, wherein the embedded event packet identifier packet includes a prefix code that can be used to identify the beginning of the embedded event packet; and encoding the event information in at least one embedded event packet data packet.

[0388] Example 151. The method according to Example 150, wherein specifying the data type includes specifying that the event information includes least significant bit (LSB) timestamp change information; and encoding the event information includes encoding the LSB timestamp change information in the at least one embedded event packet data packet.

[0389] Example 152. The method according to Example 150 or Example 151, wherein specifying the data type includes specifying that the event information includes most significant bit (MSB) timestamp change information; and encoding the event information includes encoding the MSB timestamp change information in the at least one embedded event packet data packet.

[0390] Example 153. The method according to any one of Examples 150 to 152, wherein specifying the data type includes specifying that the event information includes external trigger information, MIPI frame start information, MIPI frame end information, event frame start information, event frame end information, external metadata insertion information, internal metadata report information, CMOS image sensor (CIS) frame start information, CIS frame end information, or any combination thereof; and encoding the event information includes encoding the external trigger information, the MIPI frame start information, the MIPI frame end information, the event frame start information, the event frame end information, external metadata insertion information, the internal metadata report information, the CMOS image sensor (CIS) frame start information, and / or the CIS frame end information into the at least one embedded event packet data packet.

[0391] Example 154. The method according to any of Examples 150 to 153, wherein specifying the data type includes specifying that the event information includes inline vector event information, line vector event information or a combination thereof; and encoding the event information includes encoding the inline vector event information and / or the line vector event information in the at least one embedded event packet data packet.

[0392] Example 155. The method according to any one of Examples 150 to 154, wherein specifying the data type includes specifying that the event information includes run-length event information; and encoding the event information includes encoding the run-length event information in the at least one embedded event packet data packet.

[0393] Example 156. The method according to any one of Examples 150 to 155, wherein specifying the data type includes specifying that the event information includes different event information, wherein the different event information refers to a group of events generated at the same time by pixels spaced apart from each other by a preset distance in the X direction; and encoding the event information includes encoding the different event information in the at least one embedded event group data packet.

[0394] Example 157. An event-based vision sensor system comprising: an event-based vision sensor pixel array configured to detect events; and an event signal processor configured to encode the detected events using an encoding format including embedded event packets, the embedded event packets comprising: (1) an embedded event packet identifier data packet having a prefix code for identifying the start of the embedded event packet and a data type extension field; (2) an embedded event packet header data packet including a data type field and a word count field; and (3) one or more embedded event packet data packets.

[0395] Example 158. The event-based vision sensor system according to Example 157, wherein the encoding format further comprises: a row address most significant bit (MSB) data packet having a second prefix and content bits for specifying the most significant row address bit; and a row address least significant bit (LSB) data packet having a third prefix and content bits for specifying the least significant row address bit.

[0396] Example 159. An event-based vision sensor system according to Example 157 or Example 158, wherein the encoding format further includes: a row address extended data packet having a fourth prefix and content bits for specifying additional most significant row address bits.

[0397] Example 160. An event-based visual sensor system according to any of Examples 157 to 159, wherein the encoding format further comprises: a column address MSB data packet having a second prefix and content bits for specifying the most significant column address bit; and a column address LSB data packet having a third prefix and content bits for specifying the least significant column address bit and / or event polarity.

[0398] Example 161. An event-based visual sensor system according to any of Examples 157 to 160, wherein the encoding format further comprises: a column address extended data packet having a fourth prefix and content bits for specifying additional most significant column address bits.

[0399] Example 162. An event-based vision sensor system according to any of Examples 157 to 161, wherein the embedded event grouping can be used to encode timestamp least significant bit (LSB) change information, timestamp most significant bit (MSB) change information, in-line vector event information, line vector event information, run length event information, different event information, or any combination thereof; and the different event information refers to a group of events generated simultaneously by pixels spaced apart from each other by a preset distance in the X direction.

[0400] D. in conclusion

[0401] The above detailed description of embodiments of this technology is not intended to be exhaustive or to limit the technology to the precise forms disclosed above. Although specific embodiments and examples of the technology have been described above for illustrative purposes, those skilled in the art will recognize that various equivalent modifications are possible within the scope of the technology. For example, although the steps are presented in the given order above, alternative embodiments may perform the steps in a different order. Furthermore, the various embodiments described herein may also be combined to provide other embodiments.

[0402] From the foregoing, it should be understood that specific embodiments of the present technology have been described herein for illustrative purposes, but well-known structures and functions have not been shown or described in detail to avoid unnecessarily obscuring the description of embodiments of the present technology. To the extent that any material incorporated herein by reference conflicts with this disclosure, this disclosure shall prevail. Where the context permits, singular or plural terms may also contain plural or singular terms, respectively. Furthermore, unless the word “or” is explicitly limited to meaning only a single item excluded from other items when referring to a list of two or more items, its use in this list shall be interpreted as including (a) any single item in the list, (b) all items in the list, or (c) any combination of items in the list. Additionally, as used herein, the phrase “and / or” in “A and / or B” means A alone, B alone, and both A and B. Furthermore, the use of the terms “comprising,” “including,” “having,” and “having” throughout the text means including at least the stated features, such that no larger number of the same features and / or other features of additional types are excluded. Furthermore, as used herein, the phrases “based on,” “depending on,” “as a result of,” and “in response to” should not be construed as referring to a closed set of conditions. For example, an exemplary step described as “based on condition A” may be based on both condition A and condition B without departing from the scope of this disclosure. In other words, as used herein, the phrase “based on” should be interpreted in the same manner as the phrase “at least partially based on” or the phrase “at least partially based on.” Moreover, the terms “connected” and “coupled” are used interchangeably herein and refer to both direct and indirect connection or coupling. For example, where the context permits, element A being “connected” or “coupled” to element B may mean (i) A being directly “connected” or directly “coupled” to B and / or (ii) A being indirectly “connected” or indirectly “coupled” to B.

[0403] It should also be understood from the foregoing that various modifications can be made without departing from this disclosure or the present technology. For example, those skilled in the art will understand that the various components of the present technology can be further divided into sub-components, or the various components and functions of the present technology can be combined and integrated. Furthermore, certain aspects of the present technology described in the context of a particular embodiment may be combined or omitted in other embodiments. Moreover, although advantages associated with certain embodiments of the present technology have been described in the context of those embodiments, other embodiments may also exhibit such advantages, and not all embodiments necessarily need to exhibit such advantages to fall within the scope of the present technology. Therefore, this disclosure and related technologies may cover other embodiments not explicitly shown or described herein.

Claims

1. A method for encoding event data from an event-based visual sensor, the method comprising: Row address information is encoded using a row address most significant bit data packet and a row address least significant bit data packet, wherein the row address most significant bit data packet contains a first prefix code and the row address least significant bit data packet contains a second prefix code different from the first prefix code; and Column address information is encoded using a most significant bit (LSB) data packet and a least significant bit (LCB) data packet. The LSB data packet contains a third prefix code, and the LCB data packet contains a fourth prefix code that is different from the third prefix code. The method further includes: The line-by-line information is encoded using one or more line append data packets located after the least significant bit of the data packet at the line address. The pixel-by-pixel information is encoded using one or more column append data packets located after the least significant bit of the column address data packet, or Its combination.

2. The method of claim 1, wherein the method includes encoding the line-by-line information, and wherein encoding the line-by-line information includes encoding timestamp information shared by events that can be detected by pixels in the same line.

3. The method of claim 1, wherein the method includes encoding the line-by-line information, and wherein encoding the line-by-line information includes encoding whether the event occurred within or outside an exposure period using an exposure flag.

4. The method of claim 1, wherein the method includes encoding the pixel-by-pixel information, and wherein encoding the pixel-by-pixel information includes encoding a pixel group vector representing event information of a group of pixels.

5. The method of claim 1, further comprising encoding which rows within the pixel group have detected events using half of the least significant bits of the column address data packet.

6. The method of claim 1, further comprising encoding event polarity information using one or more column embedded bits of the least significant bit data packet of the column address.

7. The method of claim 1, wherein the method includes encoding the pixel-by-pixel information, and wherein encoding the pixel-by-pixel information includes encoding timestamp information of each event detected by each pixel using the one or more column-attached data packets.

8. The method of claim 1, further comprising encoding an indication of whether the corresponding event is an isolated event or a non-isolated event using an isolation flag of the least significant bit of the column address data packet.

9. The method of claim 1, further comprising encoding an indication of whether a preceding data packet is a data packet with the most significant bit flag of the least significant bit data packet of the row address.

10. The method of claim 1, further comprising encoding an indication of whether a pixel row is located in the upper or lower half of a pixel array segment using the top-bottom bits of the least significant bit data packet of the row address.

11. The method of claim 1, further comprising encoding line vector event information using embedded event groups, wherein the line vector event information includes (a) the Y address of an event line, (b) a line vector polarity indicator, and (c) a line vector encoding event information for pixels along the line.

12. The method of claim 1, further comprising encoding differential event information using embedded event groups, wherein the differential event information includes (a) polarity information of a plurality of events and (b) differential X address information of each event relative to a previous pixel address.

13. An event-based visual sensor system, comprising: An event-based visual sensor pixel array configured to detect events; and An event signal processor is configured to encode detected events using an encoding format that includes coordinate data packets, the coordinate data packets comprising row address most significant bit packets, row address least significant bit packets, column address most significant bit packets, and column address least significant bit packets, wherein each coordinate data packet includes a prefix code for packet identification, and wherein the encoding format further comprises: One or more line appended data packets can be used to encode line-by-line information. One or more column-attached data packets, which can be used to encode pixel-by-pixel information, or Its combination.

14. The event-based visual sensor system of claim 13, wherein the encoding format includes the one or more row-attached data packets, and wherein the row-by-row information includes timestamp information shared by events detectable by pixels in the same row.

15. The event-based visual sensor system of claim 13, wherein the encoding format includes the one or more line-by-line data packets, and wherein the line-by-line information includes an exposure flag indicating whether the event occurred within or outside the exposure period.

16. The event-based visual sensor system of claim 13, wherein the encoding format includes the one or more column-attached data packets, and wherein the pixel-by-pixel information includes a pixel group vector representing event information of a pixel group.

17. The event-based visual sensor system of claim 13, wherein the encoding format includes the one or more column-attached data packets, and wherein the one or more column-attached data packets can be used to encode timestamp information of each event detected by each pixel.

18. The event-based vision sensor system of claim 13, wherein the least significant bit data packet of the column address includes half significant bits that can be used to indicate which rows within a pixel group have detected an event.

19. The event-based vision sensor system of claim 13, wherein the least significant bit data packet of the column address includes one or more column embedded bits that can be used to convey event polarity information.

20. The event-based vision sensor system of claim 13, wherein the least significant bit data packet of the column address includes an isolation flag that can be used to distinguish between isolated events and non-isolated events.

21. The event-based visual sensor system of claim 13, wherein the least significant bit data packet of the row address includes a flag that can be used to indicate whether the immediately preceding data packet is the most significant bit data packet of the row address.

Images