Event-based visual sensor, image sensing device, and method for operating event-based visual sensor

Abstract

The present disclosure provides an event-based vision sensor, an image sensing device, and a method for operating an event-based vision sensor including an array of event-based vision sensor pixels. The event-based vision sensor includes an array of event-based vision sensor pixels, an always-on-line row scanner, an always-on-line column scanner, a primary row scanner, and a primary column scanner. The event-based visual sensor pixel array includes a plurality of always on-line pixels and a plurality of primary pixels. The always-on-line row scanner and the always-on-line column scanner are used for the plurality of always-on-line pixels. The primary row scanner and the primary column scanner are used for the plurality of primary pixels. When the event-based vision sensor is configured to operate in an always-on mode, the plurality of primary pixels are powered off, and the plurality of always-on pixels, the always-on-line row scanner, and the always-on-line column scanner are in operation.

Description

Technical Field

[0001] The present disclosure relates to an event-based sensor, and more particularly to a low-power always-on event-based sensor for mobile applications. Background Art

[0002] Image sensors face power consumption challenges. Semiconductor circuits consume power even in standby, but always-on image sensors have dynamic power control. This feature allows them to optimize power consumption based on the required image capture and data processing depending on the system load. This addresses the need for efficient power use in image sensors, ensuring optimal performance while minimizing energy consumption. Summary of the Invention

[0003] One aspect of the present disclosure provides an event-based vision sensor. The event-based vision sensor includes an event-based vision sensor pixel array, an always-on row scanner, an always-on column scanner, a main row scanner, and a main column scanner. The event-based vision sensor pixel array includes a plurality of always-on pixels and a plurality of main pixels. The always-on row scanner and the always-on column scanner are for the plurality of always-on pixels. The main row scanner and the main column scanner are for the plurality of main pixels. When the event-based vision sensor is configured to operate in the always-on mode, the plurality of main pixels are powered off, and the plurality of always-on pixels, the always-on row scanner, and the always-on column scanner are in operation.

[0004] Another aspect of the present disclosure provides an image sensing device. The image sensing device includes: a hybrid pixel array, an always-on row scanner, an always-on column scanner, a CMOS image sensor (CIS) row scanner, and a CIS column scanner. The hybrid pixel array includes a plurality of always-on event-based vision sensor pixels and a plurality of CIS pixels. The always-on row scanner and the always-on column scanner are for the plurality of always-on pixels. The CIS row scanner and the CIS column scanner are for the plurality of CIS pixels. When the image sensing device is configured to operate in the always-on mode, the plurality of CIS pixels are powered off, and the plurality of always-on event-based vision sensor pixels, the always-on row scanner, and the always-on column scanner are in operation.

[0005] Another aspect of the present disclosure provides a method for operating an event-based vision sensor. The event-based vision sensor includes an event-based vision sensor pixel array. The event-based vision sensor pixel array includes a plurality of always-on pixels and a plurality of primary pixels. The method includes: when the event-based vision sensor is in the always-on mode: receiving, by a signal processor, a first event detected by the plurality of always-on pixels; determining, by the signal processor, whether the received first event meets a first predefined threshold; and in response to determining that the received first event meets the first predefined threshold, generating, by the signal processor, a valid trigger signal to turn on at least a portion of the plurality of primary pixels and switching the event-based vision sensor to an active mode. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The various aspects of the present disclosure are best understood from the following detailed description when read in conjunction with the accompanying Figure 1 drawings. It should be noted that, in accordance with standard practice in the industry, the various features are not drawn to scale. In fact, the dimensions of the various features may be arbitrarily increased or reduced for clarity of discussion.

[0007] Figure 1 FIG. 1 is a schematic diagram illustrating the operation of a low-power always-on event-based vision sensor in accordance with some embodiments of the present disclosure.

[0008] Figure 2 FIG. 2 is a schematic diagram illustrating a low-power always-on event-based vision sensor in accordance with some embodiments of the present disclosure.

[0009] Figure 3 FIG. 3 is a flow chart of a method for operating an event-based vision sensor in accordance with some embodiments of the present disclosure.

[0010] Figure 4 FIG. 4 is a timing diagram illustrating the operation of a low-power always-on event-based vision sensor in accordance with some embodiments of the present disclosure. DETAILED DESCRIPTION

[0011] The following disclosure provides many different embodiments or examples for implementing different features of the provided subject matter. Specific examples of components and arrangements are described below to simplify the disclosure. Of course, these are merely examples and are not intended to be limiting. For example, in the following description, forming a first feature above or on a second feature may include embodiments in which the first and second features are formed in direct contact, and may also include embodiments in which additional features may be formed between the first and second features such that the first and second features are not in direct contact. Additionally, the disclosure may repeat reference numerals and / or letters in various examples. This repetition is for simplicity and clarity purposes and does not inherently indicate a relationship between the various embodiments and / or configurations discussed.

[0012] In addition, spatial relative terms, such as "beneath", "below", "lower", "above", "upper", "on top", etc., may be used herein for ease of description to describe the relationship of one element or feature to another (other) element or feature as illustrated in the various figures. The spatial relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the various figures.

[0013] As used herein, although terms such as "first", "second", and "third" describe various elements, components, regions, layers, and / or sections, these elements, components, regions, layers, and / or sections should not be limited by these terms. These terms may be used only to distinguish one element, component, region, layer, or section from another. Unless the context clearly indicates otherwise, terms such as "first", "second", and "third" do not imply a sequence or order when used herein.

[0014] While the numerical ranges and parameters setting forth the broad scope of the disclosure are approximations, the numerical values set forth in the specific examples are reported as precisely as possible. Any numerical value, however, inherently contains certain normal deviations necessarily resulting from the standard deviations found in their respective testing measurements. In addition, as used herein, the terms "substantially," "about," and "approximate" generally mean within the value or range considered by those skilled in the art. Alternatively, the terms "substantially," "about," and "approximate" mean within the acceptable standard error of the mean considered by those skilled in the art. Those skilled in the art will understand that the acceptable standard error may vary depending on the technology. Except in the operating / working examples, or unless otherwise expressly specified, all numerical ranges, amounts, values, and percentages (e.g., numerical ranges, amounts, values, and percentages for the amounts of materials, durations, temperatures, operating conditions, ratio of amounts, etc. disclosed herein) are to be understood as being modified in all instances by the terms "substantially," "about," or "approximate." Accordingly, unless indicated to the contrary, the numerical parameters set forth in the disclosure and the appended claims are approximations that may vary as desired. At the very least, each numerical parameter should be construed in light of the number of reported significant digits and by applying ordinary rounding techniques. Ranges may be expressed herein as from one endpoint to another endpoint or between two endpoints. All ranges disclosed herein are inclusive of the endpoints unless otherwise specified.

[0015] Always-on (AO) is a cutting-edge sensing technology designed for mobile devices. It enables the device to sense the context, which means the device can identify and understand the environment or any changes occurring around the device. This context awareness allows for the creation of an autonomous user interface (UI) that adapts based on the information obtained from sensing.

[0016] One of the key advantages of AO technology is its power efficiency. It utilizes a power-efficient coprocessor to share the work from the device's main processor, thereby reducing its workload and extending battery life. By offloading tasks to these specialized coprocessors, the device can save energy and operate for longer periods without the need for frequent recharging.

[0017] Overall, AO technology has revolutionized the way mobile devices interact with their surrounding environment. The AO technology enables the device to collect and interpret data from its environment, thereby allowing the device to provide a more personalized and intuitive experience for the user. Additionally, the power-efficient coprocessor ensures that these devices can operate for extended periods without draining the battery, making the device more reliable and convenient for users on the move.

[0018] Always-on (AO) technology has a wide range of applications that enhance the functionality and user experience of mobile devices. One of the key applications is the ability to launch applications using gestures or voice commands. With AO technology, users can simply wave their hand or speak a command to open their favorite applications, eliminating the need to manually browse menus or search for icons.

[0019] Another valuable application of AO technology is its ability to analyze the environment and adjust settings accordingly. For example, in a dimly lit environment, AO technology can automatically reduce the screen brightness to ensure optimal visibility while saving battery life. This feature not only enhances user comfort but also contributes to improved energy efficiency.

[0020] AO technology also enables real-time monitoring of an individual's health, fitness, and well-being. By leveraging sensors and data analysis algorithms, mobile devices can continuously track vital signs, physical activity, and sleep patterns. This information can be used to provide personalized health advice, track progress towards fitness goals, and even detect potential health issues. With AO technology, users can gain a comprehensive understanding of their well-being and make informed decisions to improve their overall health.

[0021] In addition to these applications, AO technology also has the potential to revolutionize various industries. For example, in the automotive sector, AO can enable hands-free control of in-vehicle systems, allowing drivers to focus on the road while still accessing essential features. In the retail industry, AO technology can provide a personalized shopping experience by analyzing customer preferences and offering tailored recommendations.

[0022] Therefore, the applications of AO technology are extensive and diverse. From enhancing user interaction to improving health monitoring and revolutionizing industries, AO technology has the potential to transform the way we use and interact with mobile devices, making them more intuitive, efficient, and personalized.

[0023] It is known that semiconductor circuits consume power even when they are in standby, which can lead to unnecessary energy waste. However, AO image sensors provide a solution to this problem. Using its dynamic power control feature, AO image sensors can optimize power consumption based on the specific requirements of image capture and data processing. This means that AO image sensors adjust their power usage according to the system load, ensuring that only the necessary amount of power is consumed. By doing so, AO image sensors effectively minimize energy consumption and reduce unnecessary power usage. This not only helps save energy but also contributes to the more sustainable and efficient operation of electronic devices.

[0024] The use of AO technology can automatically activate the device by detecting when the user is looking at the screen, even when the device is in sleep mode. This eliminates the stress of constantly tapping the screen to keep it on. With AO technology, users can effortlessly interact with their devices without worrying about the screen turning off accidentally. This innovative feature not only enhances the user experience but also saves battery life by activating the screen only when necessary. Whether reading an article, watching a video, or simply checking notifications, AO technology ensures a seamless and convenient user experience.

[0025] The technology described in this article is designed to optimize battery life by briefly activating the main processor only for user identification purposes. By doing so, the device can save energy and extend its battery life. Additionally, this technology provides flexibility in configuring the device to automatically switch to the standby screen when the user is no longer detected. This feature further enhances battery efficiency by minimizing unnecessary power consumption during periods of user inactivity. This innovative technology aims to provide users with a longer battery life while maintaining a seamless user experience.

[0026] Integrating algorithms into mobile / IoT devices poses significant challenges due to battery life and power limitations. These devices are typically designed to be energy-efficient, and running complex algorithms such as face detection or recognition and object detection or recognition can quickly drain the battery. Therefore, it is crucial to develop algorithms that are optimized for low power consumption.

[0027] Another challenge is the limitations of the sensors used in these devices. Sensors can have limitations in terms of sensitivity, selectivity, resolution, accuracy, and precision. These limitations can affect the performance of the algorithms because the algorithms rely on accurate and precise sensor data to obtain accurate results. Therefore, it is important to consider these limitations and develop algorithms that can work effectively using the available sensor capabilities.

[0028] In addition, the data collected by these sensors needs to be decoded and processed before it can be analyzed. This decoding and processing step adds an additional computational burden to the device, which can further drain the battery. Therefore, it is necessary to develop efficient algorithms that can handle this data processing step in a timely manner.

[0029] Finally, real-time analysis of the data is crucial for providing timely feedback. However, the limited computational resources and processing capabilities of mobile / IoT devices can make real-time analysis challenging. Therefore, it is necessary to develop algorithms that can quickly analyze the data and provide feedback in real time without compromising the performance or battery life of the device.

[0030] CMOS image sensor always-on devices typically face several limitations that impede their performance. One of the main drawbacks is the response latency, which requires at least two frames at a rate of 3 to 5 frames per second. This delay in capturing and processing images can affect the real-time functionality of the device.

[0031] Another limitation is the presence of data redundancy in static regions. This means that the sensor can capture and store unnecessary data in areas that do not change over time. This redundancy not only consumes storage space but also increases the processing time required to analyze the captured images.

[0032] In addition, always-on devices using CMOS image sensors typically have low temporal resolution. This means that the sensor may not be able to accurately capture fast-moving objects or events. The resulting images may lack necessary details and appear blurry or distorted.

[0033] Furthermore, these sensors tend to have a low dynamic range, which refers to the ability of the sensor to capture a wide range of light intensities. This limitation can result in overexposed or underexposed regions in the captured images, leading to the loss of important details.

[0034] Finally, CMOS image sensors are prone to image blur, especially in the presence of motion or jitter. This is a significant drawback for always-on devices that require clear and sharp images.

[0035] In summary, the limitations of CMOS image sensors in always-on devices include response latency, data redundancy in static regions, low temporal resolution, low dynamic range, and susceptibility to image blur and distortion in rolling shutter sensors. These limitations need to be addressed to enhance the performance and functionality of such devices.

[0036] Event-based vision sensors (EVS) offer a unique solution for always-on applications due to their specific characteristics. Unlike traditional sensors, EVS generates data only when a change in illumination or motion is detected. This feature allows for efficient data generation and utilization in various applications.

[0037] One of the key advantages of EVS is its fast response time coupled with high power efficiency. Compared to traditional sensors that operate at 5 frames per second (fps) with a power consumption of 10 milliwatts (mW), EVS operates at less than 1 mW, resulting in significant power savings.

[0038] Another advantage of EVS is the absence of redundant data generation. Unlike conventional sensors that capture and store consecutive frames, EVS generates data only when there are relevant changes in the scene. This eliminates the need for buffers to store previous data (such as in CMOS image sensor frames), reducing storage requirements and processing time.

[0039] EVS also provides high temporal resolution, allowing for the accurate capture of fast-moving objects or events. This feature ensures that important details are not missed and enables real-time analysis and response.

[0040] In addition, EVS eliminates motion blur, a common problem in traditional sensors. By capturing data only when there is a change in illumination or motion, EVS produces clear and sharp distortion-free images even in dynamic scenes.

[0041] EVS also offers a high dynamic range, allowing for the capture of a wide range of light intensities. This ensures the accurate representation of both bright and dark regions in a scene, thus preserving important details.

[0042] Furthermore, EVS can be coupled with a CIS in a hybrid sensor, combining the advantages of both technologies. This hybrid approach allows for greater flexibility and performance in various applications.

[0043] Therefore, EVS offers several advantages for always-on applications. These advantages include fast response with high power efficiency, no redundant data generation, high temporal resolution, absence of motion blur, high dynamic range, and the ability to be coupled with a CIS in a hybrid sensor. These advantages make EVS a suitable choice for enhancing the performance and functionality of always-on devices.

[0044] Figure 1 FIG. 10 is a schematic diagram illustrating the operation of a low-power always-on (AO) event-based vision sensor (EVS) in accordance with some embodiments of the present disclosure.

[0045] As shown in block 101, the low-power AO EVS in AO mode is idle. The EVS sensor pixel array 1010 includes a plurality of AO pixels 1011. In block 102, AO pixels 1021 and 1022 among the AO pixels of the event-based vision sensor pixel array 1010 detect an event (e.g., the pulse shown in block 102), and then in block 104, the detected event undergoes simple event analysis to generate a partial trigger signal. As shown in decision block 105, it is determined whether the detected event meets a predetermined condition. If the detected event meets the predetermined condition, the active mode of the low-power AO EVS is triggered. Block 103 shows the active mode of the low-power AO EVS. In the active mode of the low-power AO EVS, a plurality of main pixels 1031 of the EVS sensor pixel array 1010 are activated. If the detected event does not meet the predefined threshold, the low-power AO EVS switches back to idle. In some embodiments of the present disclosure, the main pixels 1031 within a specific region of interest 1032 of the EVS sensor pixel array 1010 are activated. In some embodiments of the present disclosure, the region of interest 1032 is the region where an object 1033 (e.g., the user's face) appears. In some embodiments of the present disclosure, the position and / or size of the region of interest 1032 are determined according to the positions of the AO pixels that detect events in AO mode. In some embodiments of the present disclosure, the activated main pixels 1031 in the active mode have a higher resolution than the AO pixels in AO mode. In block 106, the detected object 1033 is identified and tracked, and then an application trigger signal is generated to activate an appropriate application or function to further process the detected object 1033.

[0046] Figure 2 Schematic diagram illustrating a low-power always-on event-based vision sensor 20 according to some embodiments of the present disclosure.

[0047] In some embodiments of the present disclosure, an event-based vision sensor 20 is provided. The event-based vision sensor 20 includes an event-based vision sensor pixel array 200, an always-on row scanner 2011, an always-on column scanner 2013, a primary row scanner 2021, and a primary column scanner 2023. The event-based vision sensor pixel array 200 includes a plurality of always-on pixels 2001 and a plurality of primary pixels 2003. The always-on row scanner 2011 and the always-on column scanner 2013 are used for the plurality of always-on pixels 2001. The primary row scanner 2021 and the primary column scanner 2023 are used for the plurality of primary pixels 2003. In some embodiments of the present disclosure, the always-on row scanner 2011 and the primary row scanner 2021 can be combined into a single row scanner. In some embodiments of the present disclosure, the always-on column scanner 2013 and the primary column scanner 2023 can be combined into a single column scanner.

[0048] When the event-based vision sensor 20 is configured to operate in the always-on mode, the plurality of primary pixels 2003 are powered off, and the plurality of always-on pixels 2001, the always-on row scanner 2011, and the always-on column scanner 2013 are in operation. In some embodiments of the present disclosure, the event-based vision sensor 20 further includes a bias reference generator 205 for providing a bias voltage for the event-based vision sensor pixel array 200.

[0049] In some embodiments of the present disclosure, when the event-based vision sensor 20 is configured to operate in the always-on mode as Figure 1 shown, the primary row scanner 2021 and the primary column scanner 2023 are powered off. In some embodiments of the present disclosure, when the event-based vision sensor 20 is configured to operate in the active mode as Figure 1 shown, at least a portion of the plurality of primary pixels 2003 are turned on and the primary row scanner 2021 and the primary column scanner 2023 are also turned on. In some embodiments of the present disclosure, various types of operating modes similar to the active mode may exist as needed. For example, in the full pixel mode, all of the plurality of primary pixels 2003 may be turned on.

[0050] In some embodiments of the present disclosure, the event-based vision sensor 20 further includes a region-of-interest (ROI) controller for controlling the power of the plurality of primary pixels. In some embodiments of the present disclosure, based on the ROI controller, at least a portion of the plurality of primary pixels 2003 are turned on. In some embodiments of the present disclosure, the ROI controller includes a row ROI controller 2031 and a column ROI controller 2033.

[0051] In some embodiments of the present disclosure, the event-based vision sensor 20 further includes a signal processor 204. When the event-based vision sensor 20 switches to the active mode as shown in Figure 1 , the signal processor 204 generates valid trigger signals to trigger the main row scanner 2021 and the main column scanner 2023. In some embodiments of the present disclosure, when the event-based vision sensor 20 switches to the active mode as shown in Figure 1 , the signal processor 204 generates a valid row address Row_a / b and a valid column address Col_a / b to the region of interest control to indicate at least a portion of the plurality of main pixels 2003 that will be turned on.

[0052] In some embodiments of the present disclosure, the EVS is overlaid with any arbitrary color filter array (CFA). In some embodiments of the present disclosure, the image sensing device includes a hybrid pixel array, the hybrid pixel array including a plurality of always-on event-based vision sensor pixels and a plurality of CMOS image sensor (CIS) pixels. This image sensing device further includes an always-on row scanner, an always-on column scanner, a CIS row scanner, and a CIS column scanner. The always-on row scanner and the always-on column scanner are for the plurality of always-on pixels, and the CIS row scanner and the CIS column scanner are for the plurality of CIS pixels.

[0053] In some embodiments of the present disclosure, when the image sensing device is configured to operate in the always-on mode, the plurality of CIS pixels are powered off, and the plurality of always-on event-based vision sensor pixels, the always-on row scanner, and the always-on column scanner are in operation. In some embodiments of the present disclosure, when the image sensing device is configured to operate in the always-on mode, the CIS row scanner and the CIS column scanner are powered off. In some embodiments of the present disclosure, when the image sensing device is configured to operate in the active mode, at least a portion of the plurality of CIS pixels are turned on and the CIS row scanner and the CIS column scanner are also turned on.

[0054] In some embodiments of the present disclosure, the image sensing device further includes a region of interest (ROI) control for controlling the power of the plurality of CIS pixels. In some embodiments of the present disclosure, based on the region of interest control, at least a portion of the plurality of CIS pixels are turned on. In some embodiments of the present disclosure, the region of interest control includes a row ROI control and a column ROI control.

[0055] Instead of turning on when the event-based vision sensor 20 is configured to operate in the active mode as shown in Figure 1 , Figure 2The main pixel 2003 in the event-based vision sensor pixel array 200 turns on a plurality of CIS pixels when the image sensing device is configured to operate in the active mode.

[0056] Figure 3 It is a flowchart of a method 30 for operating an event-based vision sensor according to some embodiments of the present disclosure.

[0057] In some embodiments of the present disclosure, there is provided a method 30 for operating an event-based vision sensor as Figure 2 shown in. The event-based vision sensor 20 includes an event-based vision sensor pixel array 200 as Figure 2 shown in. The event-based vision sensor pixel array 200 includes a plurality of always-on pixels 2001 and a plurality of main pixels 2003. In some embodiments of the present disclosure, the event-based vision sensor 20 further includes an ROI control for controlling the power of the plurality of main pixels 2003.

[0058] The method 30 includes: when the event-based vision sensor 20 is in the AO mode, as shown in block 301, receiving, by the signal processor 204, a first event detected by the plurality of always-on pixels to check, in decision block 302, whether an event has occurred. If it is determined in decision block 302 that no event has occurred, then the event-based vision sensor returns to idle in the AO mode, as shown in block 301. If it is determined in decision block 302 that an event has occurred, then the event is decoded by the signal processor in block 303 and the event undergoes a simple event analysis in block 304 to determine whether the received event meets a predefined threshold. In some embodiments of the present disclosure, the event-based vision sensor 20 further includes a decoder to decode the event before it is received by the signal processor 204. In some embodiments of the present disclosure, the event data of the first event is raw data, and the event data of the decoded first event is text data or human-readable data. A partial trigger signal is generated by the signal processor 204. In decision block 305, in response to determining that the received first event meets a first predefined threshold, the signal processor 204 generates a valid trigger signal to turn on at least a portion of the plurality of main pixels 2003 and switch the event-based vision sensor 20 to the active mode, as shown in block 306.

[0059] When the event-based vision sensor 20 is in the active mode as shown in block 306, then in block 307, the resolution of the event-based vision sensor pixel array 200 can be increased or the intelligent ROI control function can be triggered. In some embodiments of the present disclosure, based on the ROI control, at least a portion of the plurality of primary pixels is turned on. Then, the events detected by at least a portion of the plurality of primary pixels 2003 are received by the signal processor 204 to determine in decision block 308 whether the received events meet a predefined threshold. If it is determined that the received events do not meet the predefined threshold, then in block 301, an invalid trigger signal is generated to power off at least a portion of the plurality of primary pixels 2003 and switch the event-based vision sensor 20 to the AO mode. However, if it is determined that the received events meet the predefined threshold, then in block 309, the signal processor 204 analyzes the events through object detection, object identification, etc. In some embodiments of the present disclosure, analyzing the second event includes analyzing the shape, detecting one or more objects, or identifying a human face. In some embodiments of the present disclosure, in response to analyzing the second event, the events are further processed for data fusion, tracking, or three-dimensional reconstruction.

[0060] In some embodiments of the present disclosure, after block 307, an invalid trigger signal can be triggered by receiving a user input (e.g., pressing a power button) to power off at least a portion of the plurality of primary pixels 2003 and switch the event-based vision sensor 20 to the AO mode (in block 301).

[0061] In some embodiments of the present disclosure, when switching the event-based vision sensor 20 to the active mode, the signal processor 204 generates a valid row address Row_a / b and a valid column address Col_a / b to the ROI control to indicate at least a portion of the plurality of primary pixels 2003 that will be turned on.

[0062] The present disclosure relates to an EVS for coarse-grained and fine-grained detection and its operating modes. The EVS pixel array is divided into two types of pixels: AO pixels and primary pixels. These pixels are connected to separate row buses and column buses, which are controlled by individual row scanners and column scanners.

[0063] The separation of the row scanner and the column scanner helps to reduce the design complexity and power consumption of the scanner in the AO mode. In the AO mode, only the AO pixels and their corresponding row scanners / column scanners are active for coarse-grained detection. The primary pixels and their row scanners / column scanners are powered off.

[0064] For the active mode, a portion of the primary pixels is selectively turned on based on an external ROI control. The primary row scanner and column scanner are also activated accordingly. The power supply to the primary pixels is controlled by the row ROI control and the column ROI control.

[0065] In the AO mode, power is turned off for all primary pixels, and only the primary pixels with valid Row_ROI and Col_ROI are turned on for fine-grained detection in the active mode. When switching to the active mode, the signal processor issues a valid trigger signal to the primary row scanner and column scanner. Additionally, the signal processor provides valid row addresses and column addresses (Row_a / b and Col_a / b) to the ROI control to indicate the primary pixels to be turned on.

[0066] This configuration allows for efficient power management and selective activation of pixels based on the detection mode to be performed. The EVS can operate in the AO mode for coarse-grained detection, where only the AO pixels are active. In the active mode, the EVS can selectively activate the primary pixels based on an external ROI control to enable fine-grained detection.

[0067] The present disclosure provides novel methods for optimizing power consumption and design complexity in an event-based vision sensor. By partitioning the pixel array into AO pixels and primary pixels and controlling the activation of the pixels by separate row scanners and column scanners, the EVS can efficiently perform both coarse-grained detection tasks and fine-grained detection tasks.

[0068] In summary, the event-based vision sensor described herein provides improved power management and reduced design complexity through the separation of AO pixels from primary pixels and their corresponding row scanners from column scanners. The ability to selectively activate pixels based on an external ROI control allows for both coarse-grained detection and fine-grained detection, making the EVS a versatile and efficient sensor for various applications.

[0069] In some embodiments of the present disclosure, the row scanners for the AO pixels and the primary pixels can be unified, and the column scanners for the AO pixels and the primary pixels can be unified to save chip area.

[0070] Figure 4 FIG. 40 is a timing diagram illustrating the operation of a low-power always-on event-based vision sensor according to some embodiments of the present disclosure.

[0071] Timing diagram 40 further shows details of switching the event-based vision sensor between the AO mode and the active mode. As Figure 4As shown, in some embodiments of the present disclosure, when in the AO mode, the signal trigger is low and when in the active mode, the signal trigger is high. The signal trigger powers on at least a portion of the plurality of main pixels 2003. In some embodiments of the present disclosure, the signal AO pixel readout shows events read by the AO pixel 2001 in the AO mode. The dashed box 401 marks the transition from the AO mode to the active mode. Regarding the signal global pixel reset, as Figure 4 As shown, the signal global pixel reset goes high during the transition to stop the readout of the AO pixel 2001 and reset all the pixels in the event-based vision sensor pixel array 200. After the pixels are reset, the signal Row_ROI / Col_ROI shows how the ROI control controls the plurality of main pixels 2003 and activates the ROI 403. The signal main pixel readout shows events read by the main pixels 2003 in the active mode. The dashed box 402 marks the transition from the active mode to the AO mode. The signal global pixel reset goes high again during the transition to stop the readout of the main pixels 2003 and reset all the pixels in the event-based vision sensor pixel array 200.

[0072] An event-based vision sensor is a type of sensor that offers several unique features and capabilities. One of the key advantages of these sensors is their ability to operate in a low-power always-on mode, which consumes approximately 100 times less power compared to current state-of-the-art CIS-based methods.

[0073] The event-based vision sensor is designed to have at least two operating modes: an always-on (AO) mode and an active mode. The AO mode is a low-power mode that allows the sensor to operate continuously while consuming minimal power. On the other hand, the active mode is a performance mode that offers enhanced capabilities and higher performance but consumes more power.

[0074] To enable the AO mode, additional hardware logic is incorporated into the sensor. This hardware logic allows the selection of AO pixels or regions of interest (ROIs). By selectively activating only specific pixels or ROIs, the sensor can significantly reduce power consumption while still capturing relevant events.

[0075] The data captured by the sensor is stored in a buffer. The buffer acts as a temporary storage space for the captured data before the captured data is further processed. The data may undergo additional decoding to extract relevant information or features.

[0076] Once the data is available, it is analyzed by a trigger algorithm block. This trigger algorithm block can be implemented in hardware or software, depending on the specific implementation. The trigger algorithm block analyzes the data and identifies specific events or patterns of interest.

[0077] One or more trigger signals are generated based on the analysis performed by the trigger algorithm block. These trigger signals can be used to switch the operating mode of the sensor, thereby transitioning the sensor from the low-power AO mode to a higher-performance active mode. Additionally, the trigger signals can wake up other processes or components connected to the sensor.

[0078] In summary, the event-based vision sensor described in this disclosure offers significant advantages in terms of power consumption and operational flexibility. By incorporating the AO mode, additional hardware logic, and the trigger algorithm block, the sensor can operate in a low-power always-on mode while still providing high-performance capabilities when needed.

[0079] The foregoing outlines the features of several embodiments, enabling those skilled in the art to better understand various aspects of this disclosure. Those skilled in the art should understand that they can readily use this disclosure as a basis for designing or modifying other processes and structures for implementing the same purposes and / or achieving the same advantages as the embodiments described herein. Those skilled in the art should also recognize that such equivalent constructs do not depart from the spirit and scope of this disclosure, and that various changes, substitutions, and alterations can be made herein without departing from the spirit and scope of this disclosure.

Claims

1. An event-based visual sensor comprising: An event-based vision sensor pixel array comprising: Multiple always-on pixels; and Multiple primary pixels; an always-on row scanner and an always-on column scanner for the plurality of always-on pixels; and a primary row scanner and a primary column scanner for the plurality of primary pixels, Wherein when the event-based vision sensor is configured to operate in an always-on mode, the plurality of primary pixels are powered off, and the plurality of always-on pixels, the always-on row scanner, and the always-on column scanner are in operation.

2. The event-based visual sensor of claim 1, wherein: When the event-based vision sensor is configured to operate in the always-on mode, the primary row scanner and the primary column scanner are powered off.

3. The event-based visual sensor of claim 1, wherein: When the event-based vision sensor is configured to operate in an active mode, at least a portion of the plurality of primary pixels are turned on and the primary row scanner and the primary column scanner are also turned on.

4. The event-based vision sensor of claim 3, further comprising a region of interest control for controlling power of the plurality of primary pixels. 5 . The event-based vision sensor of claim 4 , wherein the at least a portion of the plurality of primary pixels are turned on based on the region of interest control.

6. The event-based vision sensor of claim 4, wherein the region of interest controls include a row region of interest control and a column region of interest control.

7. The event-based vision sensor of claim 3, further comprising a signal processor, wherein: When the event-based vision sensor switches to the active mode, the signal processor generates a valid trigger signal to the primary row scanner and the primary column scanner.

8. The event-based vision sensor of claim 5, wherein: When the event-based vision sensor switches to the active mode, the signal processor generates an effective row address and an effective column address to the region of interest control to indicate the at least a portion of the plurality of primary pixels to be turned on.

9. An image sensing device, comprising: A hybrid pixel array comprising: Multiple always-on event-based vision sensor pixels; and Multiple CMOS image sensor CIS pixels; an always-on row scanner and an always-on column scanner for the plurality of always-on pixels; and A CIS row scanner and a CIS column scanner for the plurality of CIS pixels, Wherein, when the image sensing device is configured to operate in an always-on mode, the plurality of CIS pixels are powered off and the plurality of always-on event-based vision sensor pixels, the always-on row scanner, and the always-on column scanner are in operation.

10. The image sensing device according to claim 9, wherein: When the image sensing device is configured to operate in the always-on mode, the CIS row scanner and the CIS column scanner are powered off.

11. The image sensing device according to claim 9, wherein: When the image sensing device is configured to operate in an active mode, at least a portion of the plurality of CIS pixels are turned on and the CIS row scanner and the CIS column scanner are also turned on.

12. The image sensing apparatus of claim 11, further comprising a region of interest control for controlling power of the plurality of CIS pixels.

13. A method for operating an event-based vision sensor comprising an event-based vision sensor pixel array, wherein the event-based vision sensor pixel array comprises a plurality of always-on pixels and a plurality of primary pixels, the method comprising: When the event-based vision sensor is in always-on mode: receiving, by a signal processor, a first event detected by the plurality of always-on pixels; determining, by the signal processor, whether the received first event satisfies a first predefined threshold; and In response to determining that the received first event satisfies the first predefined threshold, an active trigger signal is generated by the signal processor to turn on at least a portion of the plurality of primary pixels and switch the event-based vision sensor into an active mode.

14. The method according to claim 13, further comprising: When the event-based vision sensor is in the active mode: receiving, by the signal processor, a second event detected by the at least a portion of the plurality of primary pixels; determining, by the signal processor, whether the received second event satisfies a second predefined threshold; In response to determining that the received second event satisfies the second predefined threshold, analyzing, by the signal processor, the second event; and generating, by the signal processor, a deactivation trigger signal to power off the at least a portion of the plurality of primary pixels in response to determining that the received second event does not satisfy the second predefined threshold, And switching the event-based vision sensor into the always-on mode.

15. The method of claim 13, further comprising a region of interest control for controlling power of the plurality of primary pixels.

16. The method of claim 15, wherein the at least a portion of the plurality of primary pixels are turned on based on the region of interest control.

17. The method according to claim 16, further comprising: When the event-based vision sensor is switched to the active mode, an effective row address and an effective column address are generated by the signal processor to the region of interest control to indicate the at least a portion of the plurality of primary pixels to be turned on.

18. The method of claim 13, further comprising: The first event is decoded by a decoder before the first event is received by the signal processor.

19. The method of claim 18, wherein the event data of the first event is raw data and the event data of the decoded first event is human-readable data.

20. The method of claim 14, wherein analyzing the second event comprises analyzing a shape, detecting one or more objects, or recognizing a human face.

Images