Encoding sensor data using synchronous readout

Abstract

Various embodiments disclosed herein include devices, systems, and methods for determining a control signal before a synchronous readout of a frame (or a portion of a frame) is performed by a sensor. Various embodiments disclosed herein include devices, systems, and methods for determining an event density (e.g., a dense row signal or a dense frame signal) before a synchronous readout is performed by a sensor. In some embodiments, an event camera can operate in a first readout mode and a second readout mode based on the number of events. In the second readout mode, less data per pixel can be read out.

Description

Technical Field

[0001] The present disclosure generally relates to systems, methods, and apparatus for using sensors, for example, in high data rate environments. Background Art

[0002] In various situations, sensors may be overloaded with data. Therefore, there is a need for improved sensor input / output methods and systems. Summary of the invention

[0003] Various embodiments disclosed herein include devices, systems, and methods for determining a control signal before a synchronous readout of a frame (or a portion of a frame) is performed by a sensor. Various embodiments disclosed herein include devices, systems, and methods for determining a dense row signal or a dense frame signal before a synchronous readout is performed by a sensor. In some embodiments, a sensor using synchronous readout can select a readout mode for the frame (or a portion of a frame) based on how many pixels are detecting an event. For example, when less than a threshold percentage (e.g., 20%) of pixels are detecting an event, a first mode can be selected to read out only the data of the pixel where the event occurred, and when more than a threshold percentage of pixels are detecting an event, a second mode can be selected to read out the data of each pixel. In the second mode, less data can be read out for each pixel. In some embodiments, flexible event encoding / readout can be more efficient, reduce power, improve image quality, or avoid overloading the readout circuit of the sensor. Density determination (e.g., dense row or dense frame) is performed using analog or digital circuits. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] So that the present disclosure may be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0005] Figure 1 is a block diagram of an example system according to some implementations.

[0006] Figure 2 is a block diagram of an example controller according to some implementations.

[0007] Figure 3 is a block diagram of an exemplary electronic device according to some implementations.

[0008] Figure 4 is a graph illustrating an exemplary relationship between the number of bits per pixel encoded and the number of events per frame for an event camera according to some specific implementations.

[0009] Figure 5 is a block diagram illustrating an exemplary detection of sparse / dense conditions for an event camera according to some specific implementations.

[0010] Figure 6 is a block diagram illustrating another exemplary detection of sparse / dense conditions for an event camera according to some implementations.

[0011] Figure 7 is a block diagram illustrating an exemplary detection of a sparse / dense row condition for an event camera according to some specific implementations.

[0012] Figure 8 is a block diagram illustrating an exemplary detection of sparse / dense frame conditions for an event camera according to some implementations.

[0013] Fig. 9 is a diagram illustrating an exemplary event camera using low power synchronous readout according to some implementations.

[0014] Fig.10 is a block diagram of a pixel sensor for an event camera and an exemplary circuit diagram of a pixel sensor according to some specific implementations.

[0015] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. DETAILED DESCRIPTION

[0016] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the exemplary implementations described herein.

[0017] Various embodiments disclosed herein include devices, systems, and methods for detecting event density prior to performing a synchronous readout of a frame (or a portion of a frame) by an event camera. In some embodiments, the system includes a matrix arrangement of multiple rows of pixels, wherein each of the pixels includes a photodetector and an event detector coupled to the photodetector. The event detector is configured to detect an event based on detecting a change in light intensity exceeding a threshold value of a change in light received at the photodetector. In some embodiments, the event density detector is configured to determine an event density of events detected at a subset of pixels arranged in the matrix based on inputs received from the subset of pixels and a density threshold. The readout circuit is then configured to determine a readout mode based on the event density detector, and to read out event data based on the readout mode.

[0018] Various embodiments disclosed herein include temporary storage of sparse event data before transmitting it to an output link, and devices, systems, and methods for directly transmitting dense event data to an output link during the readout of a frame (or a portion of a frame) performed by an event camera. In some embodiments, the system includes a matrix arrangement of pixels, each of which is configured to detect an event based on a detected change in light intensity exceeding a threshold of light received at a photodetector. In some embodiments, an event buffer is configured to accumulate event data from pixels at a matrix arrangement for an event that occurs based on an accumulation criterion, and a transmission circuit is configured to read out event data from the event buffer and transmit event data. In some embodiments, the transmission circuit includes a communication link that is disabled between transmissions. In some embodiments, the transmission circuit can read out or send event data based on the degree to which the event buffer is occupied by the accumulated event data. In some embodiments, the transmission circuit bypasses the event buffer based on an event occurring at a matrix arrangement of pixels that meets an event density criterion.

[0019] Figure 1 1 is a block diagram of an exemplary operating environment 100 according to some implementations. As a non-limiting example, the operating environment 100 includes a controller 110 and an electronic device (eg, a laptop) 120, one or both of which may be in a physical set 105.

[0020] In some implementations, the controller 110 can be configured to detect intensity and contrast changes. In some implementations, the controller 110 includes a suitable combination of software, firmware, or hardware. Figure 2 The controller 110 is described in more detail. In some implementations, the controller 110 is a computing device that is located locally or remotely relative to the physical set 105.

[0021] In one example, the controller 110 is a local server located within the physical set 105. In another example, the controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside the physical environment 105. In some implementations, the controller 110 is communicatively coupled to the corresponding electronic device 120 via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.).

[0022] In some implementations, the controller 110 and a corresponding electronic device (eg, 120 ) are configured to detect both intensity and contrast changes together.

[0023] In some implementations, the electronic device 120 is configured to detect intensity and contrast changes. In some implementations, the electronic device 120 includes a suitable combination of software, firmware, or hardware. Figure 3 The electronic device 120 is described in more detail. In some implementations, the functionality of the corresponding controller 110 is provided by or combined with the electronic device 120, for example, in the case of an electronic device used as a stand-alone unit.

[0024] Figure 2 is a block diagram of an example of a controller 110 according to some implementations. While some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., a universal serial bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), code division multiple access (CDMA), time division multiple access (TDMA), global positioning system (GPS), infrared (IR), Bluetooth, ZIGBEE, or similar type interface), one or more programming (e.g., I / O) interfaces 210, a memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0025] In some implementations, the one or more communication buses 204 include circuits that interconnect system components and control communications between system components. In some implementations, the one or more I / O devices 206 include at least one of a keyboard, a mouse, a trackpad, a joystick, one or more microphones, one or more speakers, one or more image capture devices or other sensors, one or more displays, etc.

[0026] The memory 220 includes a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some specific implementations, the memory 220 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 220 optionally includes one or more storage devices located away from the one or more processing units 202. The memory 220 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 220 or the non-transitory computer-readable storage medium of the memory 220 stores the following programs, modules, and data structures or a subset thereof, including an optional operating system 230 and a detection module 240.

[0027] The operating system 230 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the detection module 240 is configured to detect contrast changes, for example, using an event camera. In addition, Figure 2 It is more of a functional description of various features present in a particular implementation, as opposed to a schematic diagram of the implementations described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 2 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are allocated among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software, or firmware selected for a specific specific implementation.

[0028] Figure 3is a block diagram of an example of an electronic device 120 according to some implementations. While some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, the electronic device 120 includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more displays 312, one or more internal or external-facing sensor systems 314, a memory 320, and one or more communication buses 304 for interconnecting these components and various other components.

[0029] In some implementations, one or more communication buses 304 include circuits that interconnect and control communications between system components. In some implementations, one or more I / O devices and sensors 306 include an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0030] In some implementations, one or more displays 312 are configured to present content to a user. In some implementations, one or more displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS), or similar display types. In some implementations, one or more displays 312 correspond to diffraction, reflection, polarization, holographic, etc. waveguide displays. For example, an electronic device may include a single display. As another example, an electronic device may include a display for each eye of a user.

[0031] In some implementations, one or more of the internally facing or externally facing image sensor systems 314 include an image capture device or array (e.g., a frame camera or an event camera) that captures image data, or an audio capture device or array (e.g., a microphone) that captures audio data.

[0032] The memory 320 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some specific implementations, the memory 320 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 320 optionally includes one or more storage devices located away from the one or more processing units 302. The memory 320 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 320 or the non-transitory computer-readable storage medium of the memory 320 stores the following programs, modules and data structures or their subsets, including an optional operating system 330 and a detection module 340.

[0033] The operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the detection module 340 is configured to detect contrast changes, for example, using an event camera. In addition, Figure 3 It is more of a functional description of various features present in a particular implementation, as opposed to a schematic diagram of the implementations described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 3 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are allocated among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software, or firmware selected for a specific specific implementation.

[0034] An event camera generates events in response to temporal contrast changes in a physical setup. Generally speaking, event camera readout is asynchronous, meaning that whenever a contrast change is detected in a pixel, an event can be generated for that particular pixel, and the event camera readout provides a fast event-driven response. In an asynchronous event camera readout, the asynchronous event encoding includes x and y coordinates, as well as a timestamp. Events are typically sparse, so the asynchronous output data rate of an event camera is extremely low when compared to a frame-based image sensor. In this case of sparse events, asynchronous encoding does not require a high-bandwidth readout of the event camera. However, when a large number of events are generated, the asynchronous event camera readout may be overloaded.

[0035] In an asynchronous event camera readout, a typical pixel grouping for an event camera may include a polarity bit (e.g., positive or negative contrast change), X and Y pixel coordinates, and a timestamp. In some implementations, the timestamp may be 16 bits. In some implementations, the X and Y pixel coordinates may be 9 / 10 bits (e.g., for a video graphics array (VGA) sensor). However, sudden background light intensity changes or fast-moving scenes may cause a large number of pixels to be triggered, which may overload the asynchronous event camera readout pipeline.

[0036] Various embodiments disclosed herein include devices, systems, and methods implemented by electronic devices using synchronous readout of event cameras. Various embodiments disclosed herein include devices, systems, and methods implemented by electronic devices that detect multiple events (e.g., sparse or dense) in event cameras. Various embodiments disclosed herein include devices, systems, and methods for changing the synchronous readout of a frame (or a portion of a frame) based on the number of detected events in the event camera (e.g., a dense row signal or a dense frame signal). In some embodiments, a dense row signal or a dense frame signal is generated before the synchronous readout of the frame (or a portion of a frame) is performed by the event camera. In some embodiments, the event camera operates in a first readout mode or a second readout mode based on a dense / sparse control signal (e.g., row or frame) generated by the pixel array before reading from the pixel array.

[0037] In various implementations, an event camera includes a plurality of pixels (eg, a matrix of pixels) capable of detecting events (eg, contrast changes).

[0038] In some implementations, the event camera can be read out in a synchronized raster mode, similar to a traditional rolling shutter camera, where each row is scanned sequentially once per frame. In some implementations employing synchronized mode, a timestamp can be transmitted once per frame as a frame header, the row number can be encoded as a read row header, and the pixel encoding can include a polarity bit and an X coordinate (e.g., 1 bit + 10 bits = 11 bits). In this case, only pixels where an event occurs are transmitted.

[0039] In some implementations, the event camera can be read out in synchronized raw raster mode, where each pixel is encoded with 2 bits (eg, positive event, negative event, or no event).

[0040] In some implementations, the synchronized readout mode and the synchronized raw grating readout mode can be used to read out a portion of an event camera. Thus, the readout mode can be used for each row, each frame, or any subset of the pixel array in the event camera.

[0041] In some implementations, data can be compressed in event camera readout mode (eg, 11-bit packets or 2-bit packets) using various known techniques. Additionally, in some implementations, the bit size of the packets can be varied in readout mode.

[0042] Figure 4 is a graph showing an exemplary relationship between the number of bits per pixel encoded and the number of events per frame for an event camera according to some implementations. In some implementations, the tradeoff between the number of bits per pixel encoded and the number of events per frame for grouped readout can be evaluated or balanced for an event camera. In some implementations, a higher bit per pixel encoding (e.g., 11 bits per pixel encoding) is effective for event occupancy below a threshold, and a lower bit per pixel encoding (e.g., 2 bits per pixel encoding) is more effective for event occupancy above a threshold. In some implementations, the threshold is set based on the encoding bit size, pixel array size, readout circuitry, event camera output circuitry, etc. In some implementations, 11 bits per pixel encoding is effective for event occupancy below 20% (e.g., less than 20% of the pixels generate an event for each frame), and 2 bits per pixel encoding is used for event occupancy above 20% (e.g., see Figure 4 ). In some implementations, using two or more thresholds enables the bits-per-pixel encoding to variably use three or more different encodings (eg, different bits or data per pixel).

[0043] In various implementations, an event camera sensor is enabled to determine whether incoming data is dense or sparse (e.g., relative to a preset threshold). In some implementations, the event camera includes a control signal that indicates whether an incoming data frame is sparse or dense before readout of the event camera begins. In some implementations, the event camera includes a control logic component or a control circuit system that is configured to determine whether an incoming data frame is sparse or dense before readout of the event camera begins. In some implementations, the control signal or control logic component drives the event camera to decide whether to transmit information packets with higher bit encoding (e.g., polarity, X coordinate; 11bpp) or lower bit encoding (e.g., positive event, negative event, or no event; 2bpp). In some implementations, the control signal or control logic component is capable of detecting sparse events or dense events on a frame-by-frame, row-by-row, or based on any subset of the event camera pixel array.

[0044] Figure 5 is a block diagram illustrating an exemplary detection of sparse / dense conditions for an event camera according to some specific implementations. Figure 5As shown, the event camera 500 includes a pixel array 510, which includes a plurality of pixels 520a, 520b, ..., 520n. In some implementations, each of the plurality of pixels 520a, 520b, ..., 520n includes a detector 550a, 550b, ..., 550n, which can be enabled when an event is present in the corresponding pixel. In some implementations, one detector is used for two or more pixels (e.g., detector 550a is used with pixels 520a and 520b). The detection of each pixel in a frame, row, or readout area is then compared to a reference to determine the value of a sparse / dense control signal (S / D flag) 540 for the frame, row, or readout area. As shown Figure 5 As shown, detections from detectors 550a, 550b, ..., 550n for each pixel in a frame, row, or readout area are combined as input 560 and compared to a threshold 532 by an event density detector 530 to determine a sparse / dense control signal (S / D flag) 540.

[0045] Figure 6 is a block diagram illustrating another exemplary detection of sparse / dense conditions for an event camera according to some specific implementations. Figure 6 As shown, the event camera 600 includes a pixel array 510, which includes a plurality of pixels 520a, 520b, ..., 520n having current sources 650a, 650b, ..., 650n, which can be enabled when an event is present in the corresponding pixel. In some implementations, one current source is used for two or more pixels (e.g., current source 650a is used with pixels 520a-520d). The currents of the pixels in a frame, row, or readout area are combined as an input 660 to a current integrator 630. In some implementations, the current integrator 630 is reset and then the input current 660 is integrated for a determined amount of time. In some implementations, the current integrator 630 outputs a voltage 634 proportional to the incoming current, which can be compared with a voltage threshold 632 in a comparator 635 to determine a sparse / dense control signal (S / D flag) 540 for a frame, row, or readout area. In some implementations, the event density detector 630 is a current comparator.

[0046] Figure 7 is a block diagram illustrating an exemplary detection of a sparse / dense row condition for an event camera according to some specific implementations. Figure 7 As shown, the event camera 700 includes a row of pixels 720a, 720b, ..., 720 of the event camera 2D pixel array. N , where current sources I1, I2, ... I NOne is placed per pixel in the row. In some implementations, each current source I1, I2, ... I N is a single transistor switch.

[0047] like Figure 7 As shown, transistor T2 (e.g., a source follower transistor) includes a second electrode (e.g., a drain) coupled to the second electrode of transistor T1, a first electrode (e.g., a source) coupled to the current source pixel memory output or the Vin input of comparator A1, and a gate coupled to a second reference voltage 704 (e.g., a supply voltage or a higher voltage). Figure 7 As shown, transistor T1 has a first electrode coupled to a third reference voltage 706 and is coupled to receive a control signal to enable (or disable) the row of pixels 720a, 720b, . . . , 720 N The current sources I1, I2, ... I N In some implementations, the second reference voltage 704 is less than the third second reference voltage 706.

[0048] In some implementations, when any particular pixel 720a, 720b, ..., 720 N When an event occurs, the corresponding current sources I1, I2, ... I N (e.g., a single transistor switch) is turned on, and current flows through the corresponding current sources I1, I2, ... I N (e.g., a single transistor switch). Figure 7 As shown, the rows of pixels 720a, 720b, ..., 720 N More events in the circuit create more load for transistor T2, and V IN The potential changes with the pixels 720a, 720b, ..., 720 N The number of events in the IN The potential reaches the threshold voltage V TH When the comparator A1 flips to generate or enable the dense row flag DR. Set the threshold voltage V for comparator A1 TH , so that when the number of events in a row of event camera 700 reaches an arbitrary critical number, it can flip. In some implementations, the arbitrary critical number is the number of pixels 720a, 720b, ..., 720 N In some implementations, each current source I1, I2, ... I NA single transistor switch having a first electrode coupled to a ground reference voltage, a gate coupled to a pixel memory output or a comparator in a pixel, and a second electrode coupled to the Vin input of comparator A1. In some implementations, one current source is used for two or more pixels (e.g., current source I2 is used with pixels 720a and 720b).

[0049] Figure 8 is a block diagram illustrating an exemplary detection of sparse / dense frame conditions for an event camera according to some specific implementations. Figure 8 As shown, the event camera 800 includes a pixel array 810, which includes a plurality of rows 8501, 8502, ..., 8503 of pixels 820a, 820b, ..., 820n of the event camera 2D pixel array. N , where current sources I1, I2, ... I can be enabled when an event exists in the corresponding pixel N In rows 8501, 8502, ..., 850 N In some implementations, each current source I1, I2, ... I N is a single transistor switch. In some implementations, increasing the number of events in the pixel array 810 involves multiple rows 8501, 8502, ..., 850 N The corresponding current sources I1, I2, ... I in the corresponding pixels in N All enabled current I n The sum flows through the common-gate load transistor T2 and is converted into a voltage V IN-F When V IN-F The potential drops below the threshold voltage V TH-F When , comparator A2 flips to generate or enable the dense frame flag DF. In some implementations, when 10%, 20%, 30%, or 40% of the pixels in the pixel array 810 have an event, the threshold voltage V is reached. TH-F .like Figure 8 As shown, according to some implementations, the gate of transistor T1 is coupled to receive a control signal to enable (or disable) frame event density detection of pixel array 810 before frame readout is performed by event camera 800 (e.g., via current sources I1, I2, ... I N ).

[0050] although Figures 7 and 8Exemplary analog circuits are shown, but other methods or implementations, such as for row-by-row / frame-by-frame readout modes, may use digital circuits. In some implementations, the number of event presence bits may be summed (e.g., before being transmitted off-chip) and compared to a digital threshold to determine the sparse / dense state (e.g., DR, DF) of any subset of the event camera pixel array.

[0051] In some implementations, the system includes a matrix arrangement of multiple rows of pixels, an event density detector, and a readout circuit. Each of the pixels includes a photodetector, and an event detector coupled to the photodetector, the event detector being configured to detect an event based on detecting a change in light intensity exceeding a threshold value of a change in light received at the photodetector. In some implementations, each pixel is a dynamic visual sensor that detects events based on a temporal contrast change in light received by each pixel. In some implementations, the event density detector is configured to determine an event density of events detected at a subset of pixels arranged in the matrix based on inputs received from the subset of pixels and a density threshold. For example, each event detector provides a signal to an event density detector that determines whether a combination of the signals exceeds a density threshold, for example, more than 20% of the pixels in a row, frame, or other subset of pixels that have received an event after the last readout. The readout circuit is then configured to determine a readout mode (e.g., low density or high density) based on the event density detector, and read out event data based on the readout mode. In one example for low-density event mode, event data is sent for each pixel where a pixel event occurs (e.g., 11 bits per pixel), and the event data includes only the polarity and X coordinate (e.g., but not the Y coordinate known from the row number). In another example for high-density event mode, event data is sent for each pixel (e.g., 2 bits per pixel), regardless of whether an event has occurred at that pixel, and the event data includes only positive events, negative events, or no events, because the X and Y coordinates are known based on the sequential data and the fact that data is sent for each pixel. In some implementations, the readout circuitry can read / send a block of pixel data at a time (e.g., one row at a time), and because the row is known (e.g., based on the row number encoded in the row header) or a timestamp is used for the frame, the event data transmitted for each pixel where an event occurs can be less than the event data that would have been transmitted for each pixel.

[0052] In some implementations, temporal contrast changes in the physical environment captured by the event camera generate a corresponding event stream. In some implementations, smaller pixels used for the event camera pixel array result in a larger pixel array, and therefore an increased number of events. In some implementations, the event camera pixel array with smaller pixels uses synchronous readout. In some implementations, the synchronous readout of the event camera pixel array is performed row by row. In some implementations, the smaller pixels of the event camera pixel array generate timing or handshake synchronization to maintain accurate readout of events. In some implementations, in order to improve the accuracy of event detection and output, the synchronous readout speed is increased, and the event camera output link must be synchronized with the increased readout speed (e.g., fast row readout).

[0053] However, the event camera output link (e.g., 980PHY) is a significant source of power consumption. In some implementations, the event camera output link is a low power display port, camera communication link, etc. In some implementations, the event camera output link power consumption creates a dilemma in how to transmit (e.g., how to package and send data) over the output link (e.g., event camera interface) with reduced or minimal power consumption.

[0054] In some implementations, sparse events are accumulated or stored (e.g., memorized) in the event camera device until data from the sparse event population reaches a given size or a particular level. In some implementations, sparse events from the synchronous readout are accumulated or stored in a memory (e.g., storage buffer 960) in the event camera device until a burst of sparse event data (e.g., temporarily stored in storage buffer 960) is transmitted. In some implementations, the synchronous sparse event pixel encoding (e.g., sparse event data) includes coordinates and polarity of the event with a timestamp (e.g., fine timestamp 992) corresponding to each row. In some implementations, accumulation of sparse event data in the synchronous readout of the event camera pixel array allows disabling of the event camera output link (e.g., sensor link) between bursts of sparse event data, which reduces power consumption of the event camera.

[0055] In some implementations, dense events or high density event frames must be transmitted (e.g., infrequently) as raw raster images (e.g., using 2bpp encoding 942), thereby bypassing the event camera's event accumulation (e.g., storage buffer 960). In some implementations, the high density event row encoding includes 2 bits of raw data per pixel with a y coordinate (e.g., row number) appended. In some implementations, a sparse event frame readout or a high density event frame readout (e.g., a dense row, frame, or subset of the pixel array) is predetermined before an event camera pixel array readout occurs. In some implementations, a sparse event frame readout or a high density event frame readout is determined based on the number of events (e.g., event density or occupancy percentage) in a synchronization data frame (e.g., see at least Figure 4 ). A dense frame signal (eg, 954) or a dense line signal (eg, 952) may be generated in the digital domain or the analog domain (see Figures 5 to 8 ).

[0056] Fig. 9 is a diagram illustrating an exemplary event camera architecture using synchronous readout according to some implementations. Fig. 9 As shown, the event camera architecture 900 uses low power synchronous readout. In some implementations, the event camera architecture 900 includes a storage buffer 960. In some implementations, during a first synchronous (e.g., sparse event) readout mode, events detected by the pixel array 910 are accumulated in the storage buffer 960. In some implementations, during the first synchronous (e.g., sparse event) readout mode, events detected by the pixel array 910 are encoded by the encoding layer 940 and transmitted to the storage buffer 960 for temporary storage. In some implementations, the first synchronous readout mode uses a polarity bit and an X coordinate (e.g., 11 bits) to encode event data attached to a row (e.g., a Y coordinate). In some implementations, the storage buffer 960 stores super data frames. In some implementations, super data frames are not transmitted as frequently as single data frames (e.g., all pixels of the pixel array 910), so that the output link 980 can be turned off to save power consumption when accumulating super frames in the storage buffer 960. In some implementations, a super data frame can store events from multiple synchronous data frames (e.g., sparse event data) output from pixel array 910. In some implementations, a super data frame can store events (e.g., sparse events) from more than 10 synchronous frames, more than 20 synchronous frames, more than 100 synchronous frames output from pixel array 910. In some implementations, the number of sparse event data frames stored in storage buffer 960 by a super data frame is based on the event density of the sparse event frames.

[0057] In some implementations, frames captured by the pixel array 910 have many events (e.g., caused by light intensity changes, moving scenes, flicker, or other reasons). In some implementations, during the second synchronous (e.g., dense event, raw raster image, 2bpp) readout mode, events detected by the pixel array 910 are transmitted directly to the grouping layer 970. In some implementations, during the second synchronous (e.g., dense event) readout mode, events detected by the pixel array 910 are encoded by the encoding layer 940 and transmitted to the grouping layer 970. In some implementations, the second synchronous readout mode is used for high-density event frames (e.g., greater than 20% occupancy) captured by the pixel array 910. In some implementations, the second synchronous readout mode uses 2 bits of raw data per pixel. In some implementations, the second synchronous readout mode uses positive events, negative events, or no events as 2 bits per pixel encoding.

[0058] In some implementations, before reading out a frame of data from the pixel array 910, it is determined whether to use the first synchronous readout mode or the second synchronous readout mode. In some implementations, the determination of whether to use the first synchronous readout mode or the second synchronous readout mode is based on an event occupancy rate of a frame (e.g., or a portion of a frame) captured by the pixel array 910. In some implementations, the determination of whether to use the first synchronous readout mode or the second synchronous readout mode is based on a dense row (DR) signal 952 or a dense frame (DF) signal 954 output by the event density detector 950 (e.g., see Figures 4 to 8 ). In some implementations, the second synchronous readout mode is used only when the dense frame (DF) signal 954 is enabled. In some implementations, based on the DR signal 952 or the DF signal 954, the encoding layer 940 encodes the event data in the first readout mode and transmits it to the storage buffer 960, and encodes the event data in the second readout mode and transmits it directly to the packet layer 970.

[0059] In some implementations, the packetization layer 970 packages the data output by the storage buffer 960 (e.g., superframe) or the data output by the coding layer 940 (coding) with additional information for the receiving device 920 (e.g., dense or sparse synchronization data). In some implementations, the packetization layer 970 packages the data output by the storage buffer 960 or the coding layer with a frame header. In some implementations, the device 920 is an electronic device such as Figures 1 to 3 Controller 110 or electronic device 120.

[0060] In some implementations, each time the pixel array 910 outputs sparse events, a fine timestamp 992 is assigned. In some implementations, each time the pixel array 910 outputs events in a synchronous readout mode, a fine timestamp 992 is assigned based on a row number. In some implementations, the fine timestamp 992 is 16 bits, 18 bits, or 24 bits.

[0061] In some implementations, for each data frame, a coarse timestamp 994 is output to the packet layer 970. In some implementations, the coarse timestamp 994 is assigned each time the pixel array 910 outputs a frame in the second synchronous readout mode. In some implementations, the coarse timestamp 994 is 5 bits or 6 bits.

[0062] Various embodiments disclosed herein include devices, systems, and methods for temporarily storing sparse event data before transmitting it to an output link, and directly transmitting dense event data to an output link during a readout frame (or a portion of a frame). In some embodiments, sparse event data of multiple frames of an event camera are stored before being transmitted to an output link. For example, event data of 40 frames from an event camera can be stored before being transmitted together when a single transmission occurs. In some embodiments, the system includes a matrix arrangement of pixels, wherein each of these pixels is configured to detect an event based on a threshold value of light received at a photodetector when a change in light intensity is detected to exceed. In some embodiments, an event buffer is configured to accumulate event data from pixels at a matrix arrangement for an event that occurs based on an accumulation criterion, and a transmission circuit is configured to read out event data from an event buffer and transmit event data. In some embodiments, the transmission circuit includes an output link that is disabled between transmissions. In some embodiments, the transmission circuit can read out or send event data based on the degree to which the event buffer is occupied by accumulated event data. In some implementations, the transmission circuit bypasses the event buffer based on events occurring at the matrix arrangement of pixels satisfying an event density criterion. In some implementations, the transmission circuit determines to use or bypass the event buffer before a frame is read out by the event camera. In some implementations, the event camera uses one or more synchronous readout modes. In some implementations, the transmission circuit is configured to read out event data from the matrix arrangement of pixels in a first readout mode, and is configured to read out event data from the event buffer in a second readout mode.

[0063] Fig.10 is a block diagram of a pixel sensor or dynamic vision sensor (DVS) for an exemplary event camera according to some specific implementations, and an exemplary circuit diagram of the pixel sensor. Fig.10As shown, pixel sensors 1015 may be arranged in a 2D matrix 1010 of rows and columns with respect to an electronic device (e.g., Figure 1 The electronic device 120) is set on the event camera at a known position. Fig.10 In this example, each of the pixel sensors 1015 is associated with an address identifier defined by a row value and a column value.

[0064] Fig.10 Also shown is an exemplary circuit diagram of a circuit 1020 suitable for implementing pixel sensor 1015. Fig.10 In this example, circuit 1020 includes photodiode 1021, resistor 1023, capacitor 1025, capacitor 1027, switch 1029, comparator 1031, and event compiler 1032. In operation, a voltage is formed across photodiode 1021 that is proportional to the intensity of light incident on the pixel sensor. Capacitor 1025 is in parallel with photodiode 1021, so the voltage across capacitor 1025 is the same as the voltage across photodiode 1021.

[0065] In circuit 1020, switch 1029 is inserted between capacitor 1025 and capacitor 1027. Therefore, when switch 1029 is in a closed position, the voltage across capacitor 1027 is the same as the voltage across capacitor 1025 and photodiode 1021. When switch 1029 is in an open position, the voltage across capacitor 1027 is fixed to the previous voltage across capacitor 1027 when switch 1029 was last in a closed position. Comparator 1031 receives the voltage across capacitor 1025 and capacitor 1027 on the input side and compares them. If the difference between the voltage across capacitor 1025 and the voltage across capacitor 1027 exceeds a threshold amount ("comparator threshold"), an electrical response (e.g., voltage) indicating the intensity of light incident on the pixel sensor appears on the output side of comparator 1031. Otherwise, there is no electrical response on the output side of comparator 1031.

[0066] When the electrical response appears on the output side of the comparator 1031, the switch 1029 is switched to a closed position, and the event compiler 1032 receives the electrical response. Upon receiving the electrical response, the event compiler 1032 generates a pixel event and fills the pixel event with information indicating the electrical response (e.g., the value or polarity of the electrical response). In one specific implementation, the event compiler 1032 also fills the pixel event with one or more of the following: timestamp information corresponding to the time point at which the pixel event was generated, and an address identifier corresponding to the specific pixel sensor that generated the pixel event.

[0067] An event camera typically includes a plurality of pixel sensors, such as pixel sensor 1015, each of which outputs a pixel event in response to detecting a light intensity change exceeding a comparison threshold. When aggregated, the pixel events output by the plurality of pixel sensors form a pixel event stream output by the event camera. In some implementations, light intensity data obtained from the pixel event stream output by the event camera is used to implement various applications.

[0068] Numerous specific details are set forth herein to provide a comprehensive understanding of the subject matter. However, those skilled in the art will appreciate that the subject matter may be practiced without these specific details. In other cases, methods, devices or systems known to those of ordinary skill in the art are not described in detail in order not to obscure the subject matter.

[0069] Unless otherwise specifically noted, it should be understood that throughout the specification, discussions utilizing terms such as "process," "compute," "calculate," "determine," and "identify" refer to the actions or processes of a computing device, such as one or more computers or similar electronic computing devices, that manipulate or transform data represented as physical electronic or magnetic quantities within a memory, register, or other information storage device, transmission device, or display device of a computing platform.

[0070] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more specific implementations of the subject matter of the present invention. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software for programming or configuring a computing device.

[0071] The specific implementation of the method disclosed herein can be performed in the operation of such a computing device. The order of the blocks presented in the above examples can be changed, for example, the blocks can be reordered, combined and / or divided into sub-blocks. Some blocks or processes can be executed in parallel.

[0072] The use of "suitable for" or "configured to" herein is meant to be open and inclusive language that does not exclude devices that are suitable for or configured to perform additional tasks or steps. In addition, the use of "based on" is meant to be open and inclusive, as a process, step, calculation, or other action "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbers included herein are for ease of explanation only and are not intended to be limiting.

[0073] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0074] The terms used herein are only for describing specific implementations and are not intended to be limiting. As used in the description of specific implementations, the singular forms of "one", "a kind of" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their grouping.

[0075] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

[0076] The foregoing description and summary of the present invention should be understood to be illustrative and exemplary in every aspect, rather than restrictive, and the scope of the present invention disclosed herein is determined not only by the detailed description of the exemplary specific implementations, but by the full breadth allowed by the patent law. It should be understood that the specific implementations shown and described herein are only illustrative of the principles of the present invention, and that various modifications can be implemented by those skilled in the art without departing from the scope and essence of the present invention.

Claims

1. A system for event reading, comprising: A matrix arrangement of a plurality of rows of pixels, each of said pixels comprising: a photodetector; and an event detector coupled to the photodetector, the event detector configured to detect an event based on detecting a change in light intensity of light received at the photodetector exceeding a change threshold; an event density detector configured to determine an event density of events detected at a subset of pixels arranged in the matrix based on input received from the subset of pixels; and A readout circuit, the readout circuit being configured to determine different synchronous readout modes based on the event density detector, and to read out event data based on the different synchronous readout modes, wherein a first readout mode is selected when the event density corresponds to greater than a threshold percentage of the subset of pixels at which events are detected and a second readout mode is selected when the event density corresponds to less than a threshold percentage of the subset of pixels at which events are detected, and wherein the second readout mode is configured to read out a first set of pixels in the subset of pixels that only includes pixels at which pixel events occur, and the first readout mode is configured to read out all pixels of the subset of pixels.

2. The system of claim 1, wherein each pixel is a dynamic vision sensor that detects events based on temporal contrast changes in the light received at each pixel.

3. The system of claim 1, wherein the density determination is performed before the readout circuit reads out the event data based on the readout pattern. The system of claim 1 , wherein the event density detector comprises an analog circuit or a digital circuit.

5. The system of claim 1, wherein the event density detector comprises: a plurality of density detectors, each density detector coupled to one or more pixels in the subset of pixels; and A comparator is coupled to the plurality of density detectors to receive a combined input from the plurality of density detectors, wherein the comparator is configured to compare the combined input from the plurality of detectors to the threshold percentage.

6. The system of claim 1, wherein the event density detector comprises: a plurality of first density detectors, each first density detector coupled to one or more pixels in the subset of pixels; and a plurality of row comparators, each row comparator coupled to a subset of the plurality of first density detectors coupled to a row of pixels of the subset of pixels to receive combined inputs from the subset of the first density detectors coupled to a row of pixels of the subset of pixels, wherein each of the row comparators is configured to compare the received combined inputs from the subset of the first density detectors coupled to the row of pixels of the subset of pixels with a row density threshold value to output a dense row signal for the row; a plurality of second density detectors, each second density detector coupled to receive one or more dense row signals of the plurality of row comparators; and A frame comparator is coupled to the plurality of second density detectors to receive combined inputs from the plurality of second density detectors, wherein the frame comparator is configured to compare the combined inputs from the plurality of second density detectors with a frame density threshold.

7. The system of claim 1, wherein the event density detector comprises: a plurality of current sources, each current source coupled to a pixel of the subset of pixels; and A current comparator is coupled to the plurality of current sources to compare a combined input from the plurality of current sources to the threshold percentage.

8. The system of claim 1, wherein the threshold percentage comprises a voltage value or a current value.

9. The system of claim 1, wherein the event density detector outputs a dense row signal or a dense frame signal.

10. The system of claim 1, wherein the threshold percentage is more than 10%, 20%, or 30% of the pixels in the subset of pixels have received an event since an immediately previous readout of the pixels in the subset of pixels.

11. The system of claim 1, wherein the subset of pixels comprises a row of pixels, multiple rows of pixels, a frame, a portion of multiple rows of pixels, or other subset of pixels.

12. The system of claim 1, wherein the event density of the events is a measure of pixels in the matrix arrangement of the multiple rows of pixels in which an event was detected since an immediately previous readout of the matrix arrangement of the multiple rows of pixels.

13. The system of claim 1, wherein the first readout mode reads out first event data for each pixel in the subset of pixels, and the second readout mode reads out different second event data for at least one pixel in the subset of pixels.

14. The system of claim 13, wherein the readout circuit reads first pixel data from one row of the pixel subset at a time in the first readout mode to reduce an amount of event data for each pixel in the pixel subset.

15. A system according to claim 13, wherein the readout circuit reads and obtains first pixel data from each pixel in the pixel subset in succession in the first readout mode, and wherein the first pixel data includes 2 bits per pixel, which indicates positive event data, negative event data or no event data.

16. The system of claim 13 , wherein the readout circuitry reads second pixel data from the subset of pixels one row at a time in the second readout mode to reduce an amount of event data for each pixel in the subset of pixels, and wherein the second event data comprises a polarity and an X coordinate of a pixel event.

17. A method for event reading, comprising: At an electronic device having a processor: A matrix arrangement of a plurality of rows of pixels is provided, wherein each of the pixels comprises: a photodetector; and an event detector coupled to the photodetector, the event detector configured to detect an event based on detecting a change in light intensity of light received at the photodetector exceeding a change threshold; determining an event density of events detected at a subset of pixels arranged in the matrix based on input received from the subset of pixels; determining different synchronized readout modes based on the event density of the detected events; and Event data of the subset of pixels arranged in the matrix are read out based on the different synchronous readout modes, wherein a first readout mode is selected when the event density corresponds to greater than a threshold percentage of the subset of pixels in detecting events and a second readout mode is selected when the event density corresponds to less than a threshold percentage of the subset of pixels in detecting events, and wherein the second readout mode is configured to read out a first set of pixels in the subset of pixels that only includes pixels in which pixel events occur, and the first readout mode is configured to read out all pixels of the subset of pixels.

18. A non-transitory computer-readable storage medium storing program instructions executable by a computer on a computer to perform operations comprising: At an electronic device having a processor: A matrix arrangement of a plurality of rows of pixels is provided, wherein each of the pixels comprises: a photodetector; and an event detector coupled to the photodetector, the event detector configured to detect an event based on detecting a change in light intensity of light received at the photodetector exceeding a change threshold; determining an event density of events detected at a subset of pixels arranged in the matrix based on input received from the subset of pixels; determining different synchronized readout modes based on the event density of the detected events; and Event data of the subset of pixels arranged in the matrix are read out based on the different synchronous readout modes, wherein a first readout mode is selected when the event density corresponds to greater than a threshold percentage of the subset of pixels in detecting events and a second readout mode is selected when the event density corresponds to less than a threshold percentage of the subset of pixels in detecting events, and wherein the second readout mode is configured to read out a first set of pixels in the subset of pixels that only includes pixels in which pixel events occur, and the first readout mode is configured to read out all pixels of the subset of pixels.

Images