Event-based vision sensor, image sensing device, and method for operating an event-based vision sensor

Abstract

The present disclosure provides an event-based vision sensor, an image sensing device, and a method for operating an event-based vision sensor including an array of event-based vision sensor pixels. The event-based vision sensor includes an array of event-based vision sensor pixels, an always-on row scanner, an always-on column scanner, a primary row scanner, and a primary column scanner. The array of event-based vision sensor pixels includes a plurality of always-on pixels and a plurality of primary pixels. The always-on row scanner and the always-on column scanner are for the plurality of always-on pixels. The primary row scanner and the primary column scanner are for the plurality of primary pixels. When the event-based vision sensor is configured to operate in an always-on mode, the plurality of primary pixels are powered off, and the plurality of always-on pixels, the always-on row scanner, and the always-on column scanner are in operation.

Description

Technical Field

[0001] This disclosure relates to an event-based sensor, and more specifically to a low-power, always-on event-based sensor for mobile applications. Background Technology

[0002] Image sensors face power consumption challenges. While semiconductor circuits consume power even in standby mode, always-on image sensors feature dynamic power control. This capability allows them to optimize power consumption based on system load, the required image capture, and data processing. This addresses the need for efficient power usage in image sensors, ensuring optimal performance while minimizing energy consumption. Summary of the Invention

[0003] One aspect of this disclosure provides an event-based vision sensor. The event-based vision sensor includes an event-based vision sensor pixel array, an always-on row scanner, an always-on column scanner, a main row scanner, and a main column scanner. The event-based vision sensor pixel array includes a plurality of always-on pixels and a plurality of main pixels. The always-on row scanner and the always-on column scanner are used for the plurality of always-on pixels. The main row scanner and the main column scanner are used for the plurality of main pixels. When the event-based vision sensor is configured to operate in always-on mode, the plurality of main pixels are powered off, and the plurality of always-on pixels, the always-on row scanner, and the always-on column scanner are operational.

[0004] Another aspect of this disclosure provides an image sensing device. The image sensing device includes: a hybrid pixel array, an always-on row scanner, an always-on column scanner, a CMOS image sensor (CIS) row scanner, and a CIS column scanner. The hybrid pixel array includes a plurality of always-on event-based visual sensor pixels and a plurality of CIS pixels. The always-on row scanner and the always-on column scanner are used for the plurality of always-on pixels. The CIS row scanner and the CIS column scanner are used for the plurality of CIS pixels. When the image sensing device is configured to operate in an always-on mode, the plurality of CIS pixels are powered off, and the plurality of always-on event-based visual sensor pixels, the always-on row scanner, and the always-on column scanner are operational.

[0005] Another aspect of this disclosure provides a method for operating an event-based vision sensor. The event-based vision sensor includes an event-based vision sensor pixel array. The event-based vision sensor pixel array includes a plurality of always-on pixels and a plurality of principal pixels. The method includes: when the event-based vision sensor is in an always-on mode: receiving a first event detected by the plurality of always-on pixels by a signal processor; determining by the signal processor whether the received first event satisfies the first predefined threshold; and in response to determining that the received first event satisfies the first predefined threshold, generating a valid trigger signal to activate at least a portion of the plurality of principal pixels, and switching the event-based vision sensor to an active mode. Attached Figure Description

[0006] Basis and Appendix Figure 1 The following detailed description is recommended for optimal understanding of all aspects of this disclosure. It should be noted that, in accordance with standard industry practice, the various features are not drawn to scale. In fact, the dimensions of the various features may be arbitrarily increased or decreased for clarity of discussion.

[0007] Figure 1 The diagram illustrates a low-power, always-on, event-based vision sensor operating according to some embodiments of the present disclosure.

[0008] Figure 2 The illustration shows a schematic diagram of a low-power, always-on, event-based vision sensor according to some embodiments of the present disclosure.

[0009] Figure 3 This is a flowchart of a method for operating an event-based visual sensor according to some embodiments of the present disclosure.

[0010] Figure 4 The illustration shows a time-map of an event-based visual sensor operating at low power and always online according to some embodiments of the present disclosure. Detailed Implementation

[0011] The following disclosure provides numerous different embodiments or examples for implementing various features of the provided subject matter. Specific examples of elements and arrangements are described below to simplify this disclosure. These are, of course, merely examples and are not intended to be limiting. For example, in the following description, the formation of a first feature over or on a second feature may include embodiments in which the first and second features are formed in direct contact, and may also include embodiments in which additional features may be formed between the first and second features such that the first and second features are not in direct contact. Furthermore, reference numerals and / or letters may be repeated in various instances of this disclosure. This repetition is for simplicity and clarity and is not substantially indicative of a relationship between the various embodiments and / or configurations discussed.

[0012] Furthermore, for ease of description, spatial relative terms such as “below,” “under,” “lower,” “above,” “upper,” “upper,” and “above” may be used herein to describe the relationship of one element or feature relative to another element or feature(s) as illustrated in the figures. These spatial relative terms are intended to encompass different orientations of the device in use or operation, other than those depicted in the figures.

[0013] As used herein, while terms such as “first,” “second,” and “third” describe various elements, components, regions, layers, and / or sections, these elements, components, regions, layers, and / or sections should not be limited by these terms. These terms may be used only to distinguish one element, component, region, layer, or section from another. Unless the context clearly indicates otherwise, the use of terms such as “first,” “second,” and “third” herein does not imply sequence or order.

[0014] While the numerical ranges and parameters stated in this disclosure are approximate, the values ​​stated in particular instances should be reported as precisely as possible. However, any numerical value inherently contains a normal deviation that is necessarily caused by the standard deviation present in the corresponding test measurement. Furthermore, as used herein, the terms “substantially,” “about,” and “approximately” generally mean values ​​or ranges that can be considered by one of skill in the art. Alternatively, the terms “substantially,” “about,” and “approximately” mean within an acceptable standard error of the average value considered by one of skill in the art. One of skill in the art will understand that said acceptable standard error may vary depending on the technology. Except in operational / working instances, or unless otherwise expressly stated, all numerical ranges, quantities, values, and percentages (e.g., numerical ranges, quantities, values, and percentages for the quantities of material, durations, temperatures, operating conditions, quantity ratios, etc., disclosed herein) should be understood to be modified by the terms “substantially,” “about,” or “approximately” in all instances. Therefore, unless indicated otherwise, the numerical parameters stated in this disclosure and the appended claims are approximate values ​​that may vary as necessary. At a minimum, each numerical parameter should be interpreted based on at least the number of significant digits reported and by applying general rounding techniques. Ranges may be expressed herein as extending from one endpoint to another or between two endpoints. Unless otherwise specified, all ranges disclosed herein include the endpoints.

[0015] Always-on (AO) is a cutting-edge sensing technology designed for mobile devices. It enables devices to sense their context, meaning that the device can recognize and understand any changes occurring in its environment or around it. This context awareness allows for the creation of autonomous user interfaces (UIs) that adapt based on information obtained from sensing.

[0016] One of the key advantages of AO technology is its power efficiency. It utilizes power-efficient coprocessors to offload tasks from the device's main processor, thereby reducing its workload and extending battery life. By distributing tasks to these dedicated coprocessors, the device can save energy and operate for longer periods without requiring frequent recharging.

[0017] Overall, AO technology revolutionizes how mobile devices interact with their surroundings. AO enables devices to collect and interpret data from their environment, allowing them to provide users with a more personalized and intuitive experience. Furthermore, power-efficient coprocessors ensure these devices can operate for extended periods without depleting the battery, making them more reliable and convenient for mobile users.

[0018] Always-on (AO) technology has a wide range of applications that enhance the functionality and user experience of mobile devices. One key application is the ability to launch applications using gestures or voice commands. With AO technology, users can simply wave their hand or speak a command to open their favorite applications, eliminating the need to manually navigate menus or search icons.

[0019] Another valuable application of autofocus (AO) technology is its ability to analyze the environment and adjust settings accordingly. For example, in dimly lit environments, AO technology can automatically reduce screen brightness to ensure optimal visibility while conserving battery life. This feature not only enhances user comfort but also helps improve energy efficiency.

[0020] AO technology also enables real-time monitoring of an individual's health, fitness, and well-being. By utilizing sensors and data analytics algorithms, mobile devices can continuously track vital signs, physical activity, and sleep patterns. This information can be used to provide personalized health advice, track progress toward fitness goals, and even detect potential health problems. With AO technology, users can gain a comprehensive understanding of their well-being and make informed decisions to improve their overall health.

[0021] Beyond these applications, AO technology has the potential to revolutionize various industries. For example, in the automotive sector, AO enables hands-free control of in-vehicle systems, allowing drivers to focus on the road while still accessing essential features. In the retail industry, AO technology can provide personalized shopping experiences by analyzing customer preferences and offering tailored recommendations.

[0022] Therefore, the applications of AO technology are wide-ranging and diverse. From enhancing user interaction to improving health monitoring and revolutionizing various industries, AO technology has the potential to transform how we use and interact with mobile devices, making them more intuitive, efficient, and personalized.

[0023] Semiconductor circuits are known to consume power even in standby mode, leading to unnecessary energy waste. However, AO image sensors offer a solution to this problem. Utilizing their dynamic power control features, AO image sensors can optimize power consumption based on the specific requirements of image capture and data processing. This means that AO image sensors adjust their power usage according to the system load, ensuring that only the necessary amount of power is consumed. By doing so, AO image sensors effectively minimize energy consumption and reduce unnecessary power usage. This not only helps save energy but also contributes to more sustainable and efficient operation of electronic devices.

[0024] AO (Active Activation) technology automatically activates the device by detecting when the user is looking at the screen, even when the device is in sleep mode. This eliminates the stress of constantly tapping the screen to keep it connected. With AO, users can interact with their devices effortlessly without worrying about the screen accidentally turning off. This innovative feature not only enhances the user experience but also conserves battery life by activating the screen only when necessary. Whether reading an article, watching a video, or simply checking a notification, AO ensures a seamless and convenient user experience.

[0025] The technology described herein is designed to optimize battery life by briefly activating the main processor only for user identification purposes. By doing so, the device saves energy and extends its battery life. Additionally, this technology provides the flexibility to configure the device to automatically switch to a standby screen when no user is detected. This feature further enhances battery efficiency by minimizing unnecessary power consumption during periods of user inactivity. This innovative technology aims to provide users with longer battery life while maintaining a seamless user experience.

[0026] Integrating algorithms into mobile / IoT devices presents significant challenges due to limitations in battery life and power. These devices are typically designed to be energy efficient, and running complex algorithms such as face detection or recognition and object detection or recognition can quickly deplete the battery. Therefore, developing algorithms optimized for low power consumption is crucial.

[0027] Another challenge is the limitation of the sensors used in these devices. Sensors can have limitations in terms of sensitivity, selectivity, resolution, accuracy, and precision. These limitations can affect the performance of algorithms because they rely on accurate and precise sensor data to obtain accurate results. Therefore, it is important to consider these limitations and develop algorithms that can effectively utilize the capabilities of available sensors.

[0028] Furthermore, the data collected by these sensors needs to be decoded and processed before it can be analyzed. This decoding and processing step adds an extra computational burden to the device, which can further deplete the battery. Therefore, it is necessary to develop efficient algorithms that can handle this data processing step in a timely manner.

[0029] Finally, real-time data analysis is crucial for providing timely feedback. However, the limited computing resources and processing power of mobile / IoT devices can make real-time analysis challenging. Therefore, it is necessary to develop algorithms that can rapidly analyze data and provide real-time feedback without compromising device performance or battery life.

[0030] Always-on CMOS image sensors typically face several limitations that hinder their performance. One of the main drawbacks is the response latency, which requires at least two frames at a rate of 3 to 5 frames per second. This delay in capturing and processing images can impact the real-time functionality of the device.

[0031] Another limitation is the existence of data redundancy in static areas. This means that the sensor can capture and store unnecessary data in areas that do not change over time. This redundancy not only consumes storage space but also increases the processing time required to analyze the captured images.

[0032] Furthermore, always-on devices using CMOS image sensors typically have low temporal resolution. This means the sensor may not be able to accurately capture fast-moving objects or events. The resulting images may lack necessary detail and appear blurry or distorted.

[0033] In addition, these sensors often have a low dynamic range, which refers to the sensor's ability to capture a wide range of light intensities. This limitation can lead to overexposed or underexposed areas in the captured image, resulting in the loss of important details.

[0034] Finally, CMOS image sensors are prone to image blurring, especially in the presence of motion or jitter. This is a significant drawback for always-on devices that require clear and sharp images.

[0035] In summary, the limitations of CMOS image sensors in always-on devices include response latency, data redundancy in the static region, low temporal resolution, low dynamic range, and susceptibility to image blur and distortion in rolling shutter sensors. These limitations need to be addressed to enhance the performance and functionality of such devices.

[0036] Event-based vision sensors (EVS) offer a unique solution for always-on applications due to their specific characteristics. Unlike traditional sensors, EVS generates data only when a change in illumination or motion is detected. This feature allows for efficient data generation and utilization in a variety of applications.

[0037] One of the key advantages of EVS is its fast response time coupled with high power efficiency. Compared to conventional sensors that operate at 5 frames per second (fps) with a power consumption of 10 milliwatts (mW), EVS operates at less than 1 mW, resulting in significant power savings.

[0038] Another advantage of EVS is the absence of redundant data generation. Unlike conventional sensors that capture and store consecutive frames, EVS generates data only when relevant changes occur in the scene. This eliminates the need for buffers that store previous data (as in CMOS image sensor frames), thereby reducing storage requirements and processing time.

[0039] EVS also offers high temporal resolution, allowing for accurate capture of fast-moving objects or events. This feature ensures that important details are not missed and enables real-time analysis and response.

[0040] Furthermore, EVS eliminates motion blur, a common problem in traditional sensors. By capturing data only when there are changes in illumination or motion, EVS produces clear and sharp, distortion-free images even in dynamic scenes.

[0041] EVS also provides high dynamic range, allowing for the capture of a wide range of light intensities. This ensures that both bright and dark areas in a scene are accurately represented, thus preserving important details.

[0042] Furthermore, EVS can be coupled with CIS in a hybrid sensor, thus combining the advantages of both technologies. This hybrid approach allows for greater flexibility and performance in a variety of applications.

[0043] Therefore, EVS offers several advantages for always-on applications. These advantages include fast response with high power efficiency, no redundant data generation, high temporal resolution, absence of motion blur, high dynamic range, and the ability to couple with CIS in hybrid sensors. These advantages make EVS a suitable choice for enhancing the performance and functionality of always-on devices.

[0044] Figure 1 Schematic diagram 10 illustrates an operation of a low-power always-on (AO) event-based vision sensor (EVS) according to some embodiments of the present disclosure.

[0045] As shown in box 101, the low-power AO EVS in AO mode is idle. The EVS sensor pixel array 1010 includes a plurality of AO pixels 1011. In box 102, AO pixels 1021 and 1022 of the AO pixels in the event-based vision sensor pixel array 1010 detect an event (e.g., a pulse shown in box 102), and then in box 104, the detected event undergoes simple event analysis to generate a partial trigger signal. As shown in decision box 105, it is determined whether the detected event meets predetermined conditions. If the detected event meets predetermined conditions, then the low-power AO EVS activity mode is triggered. Box 103 illustrates the low-power AO EVS activity mode. In the low-power AO EVS activity mode, a plurality of principal pixels 1031 of the EVS sensor pixel array 1010 are activated. If the detected event does not meet a predefined threshold, then the low-power AO EVS switches back to idle. In some embodiments of this disclosure, principal pixels 1031 within a specific region of interest 1032 of the EVS sensor pixel array 1010 are activated. In some embodiments of this disclosure, the region of interest 1032 is the area where object 1033 (e.g., a user's face) appears. In some embodiments of this disclosure, the position and / or size of the region of interest 1032 is determined based on the position of the AO pixel where the event is detected in AO mode. In some embodiments of this disclosure, the activated primary pixel 1031 in active mode has a higher resolution than the AO pixel in AO mode. In block 106, the detected object 1033 is identified and tracked, and then an application trigger signal is generated to activate an appropriate application or function to further process the detected object 1033.

[0046] Figure 2 The illustration shows a schematic diagram of a low-power always-on event-based vision sensor 20 according to some embodiments of the present disclosure.

[0047] In some embodiments of this disclosure, an event-based vision sensor 20 is provided. The event-based vision sensor 20 includes an event-based vision sensor pixel array 200, an always-on row scanner 2011, an always-on column scanner 2013, a main row scanner 2021, and a main column scanner 2023. The event-based vision sensor pixel array 200 includes a plurality of always-on pixels 2001 and a plurality of main pixels 2003. The always-on row scanner 2011 and the always-on column scanner 2013 are used for the plurality of always-on pixels 2001. The main row scanner 2021 and the main column scanner 2023 are used for the plurality of main pixels 2003. In some embodiments of this disclosure, the always-on row scanner 2011 and the main row scanner 2021 may be combined into a single row scanner. In some embodiments of this disclosure, the always-on column scanner 2013 and the main column scanner 2023 may be combined into a single column scanner.

[0048] When the event-based vision sensor 20 is configured to operate in always-on mode, a plurality of principal pixels 2003 are powered off, and a plurality of always-on pixels 2001, always-on row scanners 2011, and always-on column scanners 2013 are operational. In some embodiments of this disclosure, the event-based vision sensor 20 further includes a bias reference generator 205 for providing a bias voltage for the event-based vision sensor pixel array 200.

[0049] In some embodiments of this disclosure, when the event-based visual sensor 20 is configured to, for example Figure 1 When operating in the always-on mode shown, the main row scanner 2021 and the main column scanner 2023 are powered off. In some embodiments of this disclosure, when the event-based vision sensor 20 is configured to... Figure 1 When operating in the active mode shown, at least a portion of the plurality of principal pixels 2003 are turned on, and the principal row scanner 2021 and the principal column scanner 2023 are also turned on. In some embodiments of this disclosure, various types of operating modes similar to the active mode may exist as needed. For example, in full-pixel mode, all of the plurality of principal pixels 2003 can be turned on.

[0050] In some embodiments of this disclosure, the event-based vision sensor 20 further includes a region-of-interest (ROI) controller for controlling the power of a plurality of principal pixels. In some embodiments of this disclosure, at least a portion of the plurality of principal pixels 2003 are turned on based on the ROI controller. In some embodiments of this disclosure, the ROI controller includes a row ROI controller 2031 and a column ROI controller 2033.

[0051] In some embodiments of this disclosure, the event-based vision sensor 20 further includes a signal processor 204. When the event-based vision sensor 20 switches to... Figure 1 When the activity mode shown is activated, the signal processor 204 generates a valid trigger signal to the main row scanner 2021 and the main column scanner 2023. In some embodiments of this disclosure, when the event-based vision sensor 20 switches to, as shown in the example, Figure 1 When the activity mode is shown, the signal processor 204 generates valid row addresses Row_a / b and valid column addresses Col_a / b to the region of interest control to indicate at least a portion of the multiple major pixels 2003 that will be turned on.

[0052] In some embodiments of this disclosure, the EVS is overlaid with any arbitrary color filter array (CFA). In some embodiments of this disclosure, the image sensing device includes a hybrid pixel array comprising a plurality of always-on event-based visual sensor pixels and a plurality of CMOS image sensor (CIS) pixels. This image sensing device further includes an always-on row scanner, an always-on column scanner, a CIS row scanner, and a CIS column scanner. The always-on row scanner and always-on column scanner are used for the plurality of always-on pixels, and the CIS row scanner and CIS column scanner are used for the plurality of CIS pixels.

[0053] In some embodiments of this disclosure, when the image sensing device is configured to operate in always-on mode, a plurality of CIS pixels are powered off, and a plurality of always-on event-based visual sensor pixels, always-on row scanners, and always-on column scanners are operational. In some embodiments of this disclosure, when the image sensing device is configured to operate in always-on mode, the CIS row scanners and CIS column scanners are powered off. In some embodiments of this disclosure, when the image sensing device is configured to operate in active mode, at least a portion of the plurality of CIS pixels are powered on, and the CIS row scanners and CIS column scanners are also powered on.

[0054] In some embodiments of this disclosure, the image sensing device further includes a region of interest (ROI) controller that controls the power of a plurality of CIS pixels. In some embodiments of this disclosure, at least a portion of the plurality of CIS pixels is turned on based on the ROI controller. In some embodiments of this disclosure, the ROI controller includes row ROI controllers and column ROI controllers.

[0055] Instead of when the event-based vision sensor 20 is configured to, for example Figure 1 The activity mode shown in the image is activated during operation. Figure 2The main pixel 2003 in the event-based vision sensor pixel array 200 activates multiple CIS pixels when the image sensing device is configured to operate in active mode.

[0056] Figure 3 This is a flowchart of a method 30 for operating an event-based visual sensor according to some embodiments of the present disclosure.

[0057] In some embodiments of this disclosure, methods for operating such as Figure 2 The method 30 of the event-based visual sensor 20 shown herein. The event-based visual sensor 20 includes, as... Figure 2 The event-based vision sensor pixel array 200 shown herein includes a plurality of always-on pixels 2001 and a plurality of principal pixels 2003. In some embodiments of this disclosure, the event-based vision sensor 200 further includes a ROI controller for controlling the power of the plurality of principal pixels 2003.

[0058] Method 30 includes: when the event-based vision sensor 20 is in AO mode, as shown in box 301, receiving a first event detected by a plurality of always-on pixels by a signal processor 204 to check whether an event has occurred in decision box 302. If it is determined in decision box 302 that no event has occurred, the event-based vision sensor returns to idle in AO mode, as shown in box 301. If it is determined in decision box 302 that an event has occurred, the event is decoded by the signal processor in box 303 and the event undergoes simple event analysis in box 304 to determine whether the received event meets a predefined threshold. In some embodiments of this disclosure, the event-based vision sensor 20 further includes a decoder to decode the event before it is received by the signal processor 204. In some embodiments of this disclosure, the event data of the first event is raw data, and the event data of the decoded first event is text data or human-readable data. A partial trigger signal is generated by the signal processor 204. In decision box 305, in response to determining that the received first event satisfies a first predefined threshold, signal processor 204 generates a valid trigger signal to activate at least a portion of the plurality of principal pixels 2003 and switch the event-based visual sensor 20 to an active mode, as shown in box 306.

[0059] When the event-based vision sensor 20 is in active mode as shown in block 306, the resolution of the event-based vision sensor pixel array 200 may be increased or a smart ROI control function may be triggered in block 307. In some embodiments of this disclosure, at least a portion of a plurality of principal pixels are turned on based on the ROI control. Then, events detected by at least a portion of the plurality of principal pixels 2003 are received by the signal processor 204 to determine in decision block 308 whether the received event meets a predefined threshold. If it is determined that the received event does not meet the predefined threshold, then in block 301, an invalid trigger signal is generated to power off at least a portion of the plurality of principal pixels 2003 and switch the event-based vision sensor 20 to AO mode. However, if it is determined that the received event meets the predefined threshold, then in block 309, the signal processor 204 analyzes the event by object detection, object recognition, etc. In some embodiments of this disclosure, analyzing a second event includes analyzing shape, detecting one or more objects, or recognizing a human face. In some embodiments of this disclosure, in response to the analysis of a second event, the event is further processed for data fusion, tracking, or 3D reconstruction.

[0060] In some embodiments of this disclosure, after block 307, an invalid trigger signal can be triggered by receiving user input (e.g., pressing a power button) to power off at least a portion of the plurality of principal pixels 2003 and switch the event-based vision sensor 20 to AO mode (in block 301).

[0061] In some embodiments of this disclosure, when the event-based vision sensor 20 is switched to active mode, the signal processor 204 generates valid row addresses Row_a / b and valid column addresses Col_a / b to the ROI controller to indicate at least a portion of the plurality of principal pixels 2003 that will be turned on.

[0062] This disclosure relates to an EVS for coarse-grained and fine-grained detection and its operating modes. The EVS pixel array is divided into two types of pixels: AO pixels and main pixels. These pixels are connected to separate row and column buses, which are controlled by individual row and column scanners.

[0063] Separating the row and column scanners helps reduce the design complexity and power consumption of the scanner in AO mode. In AO mode, only the AO pixels and their corresponding row / column scanners are active for coarse-grained detection. The primary pixels and their row / column scanners are powered off.

[0064] In active mode, a portion of the primary pixels are selectively powered on based on external ROI controls. The primary row and column scanners are also activated accordingly. Power supply to the primary pixels is controlled by the row and column ROI controls.

[0065] In AO mode, all major pixels are powered off, and only major pixels with valid Row_ROI and Col_ROI are powered on for fine-grained detection in active mode. When switching to active mode, the signal processor sends valid trigger signals to the major row scanner and column scanner. Additionally, the signal processor provides valid row and column addresses (Row_a / b and Col_a / b) to the ROI controller, thereby indicating the major pixels to be powered on.

[0066] This configuration allows for efficient power management and selective pixel activation based on the desired detection mode. EVS can operate in AO mode for coarse-grained detection, where only AO pixels are active. In active mode, EVS can selectively activate primary pixels based on external ROI controls, enabling fine-grained detection.

[0067] This disclosure provides a novel method for optimizing power consumption and design complexity in event-based vision sensors. By dividing the pixel array into AO pixels and principal pixels and controlling the activation of the pixels through separate row and column scanners, EVS can efficiently perform both coarse-grained and fine-grained detection tasks.

[0068] In summary, the event-based vision sensor described in this paper provides improved power management and reduced design complexity through the separation of AO pixels from primary pixels and their corresponding row and column scanners. The ability to selectively activate pixels based on external ROI controls allows for both coarse-grained and fine-grained detection, making EVS a versatile and efficient sensor for a wide range of applications.

[0069] In some embodiments of this disclosure, the row scanners for the AO pixels and the main pixels can be uniform, and the column scanners for the AO pixels and the main pixels can be uniform in order to save chip area.

[0070] Figure 4 Illustration 40: Timing diagram of a low-power, always-on, event-based visual sensor operating according to some embodiments of this disclosure.

[0071] Timeline 40 further illustrates the details of the event-based visual sensor switching between AO mode and active mode. (e.g.) Figure 4As shown in some embodiments of this disclosure, the signal trigger is low when in AO mode and high when in active mode. The signal trigger powers on at least a portion of the plurality of major pixels 2003. In some embodiments of this disclosure, the signal AO pixel readout event is read by the AO pixel 2001 in AO mode. The dashed box 401 marks the switch from AO mode to active mode. Regarding the signal global pixel reset, as... Figure 4 As shown, the global pixel reset signal goes high during the switching process to stop the readout of AO pixel 2001 and reset all pixels in the event-based vision sensor pixel array 200. After the pixels are reset, the Row_ROI / Col_ROI signal shows how the ROI controller controls multiple principal pixels 2003 and activates ROI 403. The principal pixel readout signal shows the events read by principal pixel 2003 in active mode. The dashed box 402 marks the switch from active mode to AO mode. The global pixel reset signal goes high again during the switching process to stop the readout of principal pixel 2003 and reset all pixels in the event-based vision sensor pixel array 200.

[0072] Event-based vision sensors are a type of sensor that offers several unique features and capabilities. One of the key advantages of these sensors is their ability to operate in a low-power, always-on mode, consuming approximately 100 times less power compared to current CIS-based methods.

[0073] Event-based vision sensors are designed to have at least two operating modes: always-on (AO) mode and active mode. AO mode is a low-power mode that allows the sensor to operate continuously while consuming minimal power. On the other hand, active mode is a performance mode that provides enhanced capabilities and higher performance but consumes more power.

[0074] To enable AO mode, additional hardware logic is incorporated into the sensor. This hardware logic allows selection of AO pixels or regions of interest (ROIs). By selectively activating only specific pixels or ROIs, the sensor can significantly reduce power consumption while still capturing relevant events.

[0075] The data captured by the sensor is stored in a buffer. The buffer acts as temporary storage for the captured data before it is further processed. The data may undergo additional decoding to extract relevant information or features.

[0076] Once the data is available, a triggering algorithm block analyzes it. This triggering algorithm block can be implemented in hardware or software, depending on the specific implementation. The triggering algorithm block analyzes the data and identifies specific events or patterns of interest.

[0077] Based on the analysis performed by the triggering algorithm block, one or more trigger signals are generated. These trigger signals can be used to switch the sensor's operating mode, thereby switching the sensor from a low-power AO mode to a higher-performance activity mode. Additionally, the trigger signals can also wake up other processes or components connected to the sensor.

[0078] In summary, the event-based vision sensor described in this disclosure offers significant advantages in terms of power consumption and operational flexibility. By incorporating AO mode, additional hardware logic, and triggering algorithm blocks, the sensor can operate in a low-power always-on mode while still providing high performance capabilities when needed.

[0079] The foregoing outlines the features of several embodiments, enabling those skilled in the art to better understand various aspects of this disclosure. Those skilled in the art will understand that this disclosure can be readily used as a basis for designing or modifying other processes and structures for implementing the same purposes and / or achieving the same advantages as the embodiments described herein. Those skilled in the art should also recognize that such equivalent constructions do not depart from the spirit and scope of this disclosure, and that various changes, substitutions, and modifications can be made herein without departing from the spirit and scope of this disclosure.

Claims

1. An event-based visual sensor, comprising: An event-based visual sensor pixel array, comprising: Multiple always-on pixels; and Multiple main pixels; Always-on row scanners and always-on column scanners are used for the plurality of always-on pixels; A primary row scanner and a primary column scanner, used for the plurality of primary pixels; and The region of interest controller is used to control the power of the plurality of primary pixels. When the event-based vision sensor is configured to operate in always-on mode, the plurality of primary pixels are powered off, and the plurality of always-on pixels, the always-on row scanner, and the always-on column scanner are in operation.

2. The event-based visual sensor according to claim 1, wherein, When the event-based vision sensor is configured to operate in the always-on mode, the main row scanner and the main column scanner are powered off.

3. The event-based visual sensor according to claim 1, wherein, When the event-based vision sensor is configured to operate in active mode, at least a portion of the plurality of primary pixels are turned on, and the primary row scanner and the primary column scanner are also turned on.

4. The event-based visual sensor of claim 3, wherein at least a portion of the plurality of major pixels is turned on based on the region of interest control.

5. The event-based visual sensor according to claim 1, wherein the region of interest control includes a row region of interest control and a column region of interest control.

6. The event-based visual sensor of claim 3, further comprising a signal processor, wherein, When the event-based vision sensor switches to the activity mode, the signal processor generates a valid trigger signal to the main row scanner and the main column scanner.

7. The event-based visual sensor according to claim 6, wherein, When the event-based vision sensor switches to the activity mode, the signal processor generates valid row addresses and valid column addresses for the region of interest controller to indicate at least a portion of the plurality of major pixels to be turned on.

8. An image sensing device, comprising: A hybrid pixel array, comprising: Multiple always-on, event-based visual sensor pixels; and Multiple CMOS image sensor CIS pixels; Always-on row scanners and always-on column scanners for the plurality of always-on event-based visual sensor pixels; CIS row scanner and CIS column scanner, used for the plurality of CIS pixels; and The area of ​​interest control unit is used to control the power of the plurality of CIS pixels. When the image sensing device is configured to operate in always-on mode, the plurality of CIS pixels are powered off and the plurality of always-on event-based visual sensor pixels, the always-on row scanner, and the always-on column scanner are in operation.

9. The image sensing device according to claim 8, wherein, When the image sensing device is configured to operate in the always-on mode, the CIS row scanner and the CIS column scanner are powered off.

10. The image sensing device according to claim 8, wherein, When the image sensing device is configured to operate in active mode, at least a portion of the plurality of CIS pixels are turned on, and the CIS row scanner and the CIS column scanner are also turned on.

11. A method for operating an event-based vision sensor including an event-based vision sensor pixel array, wherein the event-based vision sensor pixel array includes a plurality of always-on pixels and a plurality of principal pixels, the method comprising: When the event-based vision sensor is in always-on mode: The signal processor receives the first event detected by the plurality of always-online pixels; The signal processor determines whether the received first event satisfies a first predefined threshold. and In response to determining that the received first event satisfies the first predefined threshold, the signal processor generates a valid trigger signal to turn on at least a portion of the plurality of primary pixels and switches the event-based visual sensor to an active mode, wherein the power of the plurality of primary pixels is controlled by a region of interest controller.

12. The method of claim 11, further comprising: When the event-based visual sensor is in the activity mode: The signal processor receives the second event detected by at least a portion of the plurality of major pixels; The signal processor determines whether the received second event satisfies a second predefined threshold. In response to determining that the received second event satisfies the second predefined threshold, the signal processor analyzes the second event; and In response to determining that the received second event does not meet the second predefined threshold, the signal processor generates an invalid trigger signal to power off at least a portion of the plurality of major pixels and switches the event-based visual sensor to the always-on mode.

13. The method of claim 11, wherein at least a portion of the plurality of major pixels is turned on based on the region of interest control.

14. The method of claim 13, further comprising: When the event-based vision sensor is switched to the activity mode, the signal processor generates valid row addresses and valid column addresses for the region of interest controller to indicate at least a portion of the plurality of major pixels to be turned on.

15. The method of claim 11, further comprising: The first event is decoded by the decoder before it is received by the signal processor.

16. The method of claim 15, wherein the event data of the first event is raw data, and the event data of the decoded first event is human-readable data.

17. The method of claim 12, wherein analyzing the second event includes analyzing shape, detecting one or more objects, or identifying a human face.

Images