Sensor system architecture with feedback loop and multiple power states

By introducing feedback loops and event-driven sensor architectures with multiple power states in the event camera system, dynamically adjusting the operating state of sensor pixels, solving the inefficiency problem caused by incomplete data sets in image processing by event-driven sensors, achieving more Efficient power and bandwidth usage.

CN113647095BActive Publication Date: 2024-11-01APPLE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202080024705.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-03-27
Filing Date
2020-03-20
Publication Date
2024-11-01
Estimated Expiration
2040-03-20

Smart Images

  • Figure CN113647095B_ABST
    Figure CN113647095B_ABST
Patent Text Reader

Abstract

In one specific implementation, a system includes an image pipeline and an event sensor having a pixel array. The pixel array is configured to operate a first subset of pixels in an enabled state and a second subset of pixels in a non-enabled state. The event sensor is configured to output pixel events. In response to a particular pixel within the first subset of pixels detecting a change in light intensity exceeding a comparator threshold, each corresponding pixel event is generated. The image pipeline is configured to consume image data derived from the pixel event and transmit feedback information to the event sensor based on the image data. The feedback information causes the pixels within the first subset of pixels to transition from an enabled state to another state.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is related to international application _______________, filed on March 20, 2020, entitled “HARDWARE IMPLEMENTATION OF SENSOR ARCHITECTURE WITH MULTIPLE POWER STATES”, which is incorporated herein by reference in its entirety when reproduced. Technical Field

[0003] The present disclosure relates generally to the field of image processing, and in particular, to techniques for implementing a sensor system architecture having a feedback loop and a sensor configured to support multiple power states. Background Art

[0004] An event camera may include an image sensor known as a dynamic vision sensor ("DVS"), a silicon retina, an event-based sensor, or a frameless sensor. Thus, an event camera generates (and transmits) data regarding changes in light intensity at each pixel sensor, whereas a frame-based camera outputs data regarding the absolute light intensity at each pixel. In other words, while the illumination level of a scene disposed within the field of view remains stable, a frame-based camera will continue to generate (and transmit) data regarding the absolute light intensity at each pixel, whereas an event camera will refrain from generating or transmitting data until a change in illumination level is detected.

[0005] Some image processing operations utilize incomplete image data sets derived from pixel events output by event-driven sensors. Such image processing operations can improve computational efficiency by cropping image data, and processing the cropped image data to save power, etc. However, pixels of the event-driven sensor corresponding to image data outside the cropped image data continue to operate and, therefore, continue to consume power. Therefore, it is desirable to address this inefficiency problem that occurs when image processing operations utilize incomplete image data sets derived from pixel events output by event-driven sensors. Summary of the invention

[0006] Various embodiments disclosed herein relate to techniques for implementing an event camera system architecture having a feedback loop and an event-driven sensor configured to support multiple power states. In one embodiment, a system includes an image pipeline and an event sensor having a pixel array. The pixel array is configured to operate a first subset of pixels in an enabled state and a second subset of pixels in a non-enabled state. The event sensor is configured to output pixel events. In response to a particular pixel within the first subset of pixels detecting a change in light intensity exceeding a comparator threshold, each corresponding pixel event is generated. The image pipeline is configured to consume image data derived from the pixel event and transmit feedback information to the event sensor based on the image data. The feedback information causes the pixels within the first subset of pixels to transition from an enabled state to another state.

[0007] In another specific implementation, a system includes an event sensor, a processor, and a computer-readable storage medium. The event sensor includes a pixel array configured to have an enabled pixel area operating in full power mode and a non-enabled pixel area operating in a power-off mode or a low-power mode. The computer-readable storage medium includes instructions that cause the system to perform operations when executed by the processor. The operation includes outputting pixel events from the event sensor to an image pipeline. In response to a specific pixel in the enabled pixel area detecting a light intensity change exceeding a comparator threshold, each corresponding pixel event is generated. The operation also includes receiving feedback information from the image pipeline based on image data derived from the pixel event. In response to receiving the feedback information, the pixels in the enabled pixel area are guided to operate in the power-off mode or the low-power mode.

[0008] In another specific implementation, a system includes a processor, an image pipeline, and a computer-readable storage medium, the computer-readable storage medium including instructions that, when executed by the processor, cause the system to perform operations. The operations include receiving pixel events by the image pipeline from an event sensor having a pixel array, the pixel array including a first subset of pixels in an enabled state and a second subset of pixels in a non-enabled state. In response to a particular pixel within the first subset of pixels detecting a change in light intensity exceeding a comparator threshold, each corresponding pixel event is generated. The operations also include deriving image data from the pixel events using the image pipeline. The image pipeline generates feedback information based on the image data. The feedback information causes the event sensor to direct the pixels within the first subset of pixels to transition from an enabled state to another state. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] So that the present disclosure may be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0010] Figure 1A functional block diagram of an event sensor according to some implementations is shown.

[0011] Figure 2 is a block diagram of an exemplary system for implementing an event-driven sensor with a hardware architecture configured to support enabled, standby, and non-enabled operating states.

[0012] Figure 3 An example of a complete image data set derived by an image pipeline from pixel events output by an event sensor is shown.

[0013] Figure 4 An example of cropped image data derived by an image pipeline from pixel events output by an event sensor is shown.

[0014] Figure 5 An example of different pixels within a pixel array of an event sensor having different operating states based on feedback information received from an image pipeline is shown.

[0015] Figure 6 An example of a pixel array having different pixels having different operating states is shown, with the operating states of some pixels being modified as feedback information received from an image pipeline is updated between a first time and a second time.

[0016] Figure 7 Shows Figure 6 A subset of pixels within a pixel array is configured to transition from one operating state to another operating state as feedback information received from an image pipeline is updated between a first time and a second time.

[0017] Figure 8 An exemplary two-dimensional ("2D") plot of pixel events output by an event sensor for further processing according to some implementations is shown.

[0018] Fig. 9 is a block diagram of an exemplary head mounted device (HMD) according to some implementations.

[0019] Fig.10 is a flow chart illustrating an example of a method of implementing an event camera system architecture having a feedback loop and an event-driven sensor configured to support multiple power states.

[0020] Fig.11 is a flow chart illustrating another example of a method of implementing an event camera system architecture having a feedback loop and an event-driven sensor configured to support multiple power states.

[0021] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. DETAILED DESCRIPTION

[0022] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the exemplary implementations described herein.

[0023] A functional block diagram of an exemplary event sensor 100 is shown by Figure 1 The event sensor 100 includes a plurality of pixels 105 positioned to receive light from a scene disposed within a field of view of the event sensor 100. Figure 1 In the embodiment of the present invention, the plurality of pixels 105 are arranged in a matrix 107 of rows and columns, and thus, each pixel of the plurality of pixels 105 is associated with a row value and a column value. Each pixel of the plurality of pixels 105 includes a photodetector circuit 110 and an event circuit 180.

[0024] The photodetector circuit 110 is configured to generate a signal indicative of the intensity of light ("incident illumination") incident on the corresponding pixel 105. To this end, the photodetector circuit 110 includes a photodiode 112 configured to generate a photocurrent proportional to the intensity of the incident illumination. The photocurrent generated by the photodiode 112 flows into a logarithmic amplifier 120 formed by transistors 121, 123, 125 and 127. The logarithmic amplifier 120 is configured to convert the photocurrent into a voltage at node A, the value of which is the logarithm of the photocurrent value. The voltage at node A is then amplified by a buffer amplifier 130 formed by transistors 131 and 133 before being applied to the input side of the differential circuit 140 of the event circuit 180.

[0025] The pixel 105 also includes an event circuit 180, which includes a differential circuit 140, a comparator 160, and a controller 170. The differential circuit 140 is composed of an alternating current ("AC") coupling capacitor 145 and a switched capacitor amplifier 150. The differential circuit 140 is configured to remove a direct current ("DC") voltage component from the voltage at node A to produce pixel data at a sampling node B. By removing the DC voltage component from the voltage at node A, the pixel data at the sampling node B provides a differential value of the incident illumination intensity detected by the photodiode 112. The gain provided by the amplifier 151 corresponds to a ratio defined by the respective capacitance values ​​of the AC coupling capacitor 145 and the capacitor 153. When a reset signal is received from the controller 170, the reset switch 155 is activated (i.e., transitioned from an open state to a closed state). By activating the reset switch 155, the operating point of the amplifier 151 is reset to a reference voltage associated with the threshold of the comparator 160.

[0026] Comparator 160 is configured to provide pixel-level processing of pixel data received from sample node B. To this end, when the pixel data received from sample node B indicates that photodiode 112 detects a change in incident illumination intensity that breaks through a threshold, comparator 160 outputs an electrical response (e.g., a voltage). Alternatively, when the pixel data received from sample node B indicates that photodiode 112 does not detect a change in incident illumination intensity that breaks through a threshold, comparator 160 refrains from outputting the electrical response. In some cases, the electrical response output by comparator 160 is referred to as event data.

[0027] In one implementation, comparator 160 is implemented using a plurality of comparators including a first comparator configured to output an electrical response indicative of a positive event (e.g., an event having a positive polarity) and a second comparator configured to output an electrical response indicative of a negative event (e.g., an event having a negative polarity). In one implementation, the first comparator outputs an electrical response when the pixel data received from sample node B indicates that photodiode 112 detects a change in incident illumination intensity that breaks through a positive threshold. In one implementation, the second comparator outputs an electrical response when the pixel data received from sample node B indicates that photodiode 112 detects a change in incident illumination intensity that breaks through a negative threshold.

[0028] The controller 170 is configured to cooperate with other components of the event sensor 100 (e.g., controllers within other pixels) to transmit an event signal (e.g., a sample of event data) to the event compiler 190 for each electrical response output by the comparator 160. In one specific implementation, the reset switch 155 receives a reset signal from the controller 170 each time the comparator 160 obtains pixel data that exceeds the threshold at the sampling node B.

[0029] Event compiler 190 receives an event signal (e.g., a sample of event data) from each pixel in the plurality of pixels 105, each of which indicates that a change in incident illumination intensity has breached a threshold. In response to receiving a sample of event data from a particular pixel in the plurality of pixels 105, event compiler 190 generates a pixel event. When the event signal is associated with pixel data indicating a change in incident illumination intensity that breaches a positive threshold (or voltage), the pixel event generated by event compiler 190 may be referred to as a “positive” pixel event. In one specific implementation, a positive pixel event is a pixel event having a value indicating that the incident illumination intensity exceeds a value defined by an upper threshold or voltage (“V th ”). When the event signal is associated with pixel data indicating a change in incident illumination intensity that crosses a negative threshold (or voltage), the pixel event generated by the event compiler may be referred to as a “negative” pixel event. In one implementation, a negative pixel event is a pixel event having a positive polarity that indicates a net increase in incident illumination intensity beyond a lower threshold or voltage (“-V th ”) defines a net decrease in amplitude for pixel events of negative polarity.

[0030] In addition, the event compiler 190 populates the pixel event with information indicating the electrical response included in the event signal (e.g., the value or polarity of the electrical response). In one specific implementation, the event compiler 190 also populates the pixel event with one or more of: timestamp information corresponding to the time point in time when the pixel event was generated and an address identifier corresponding to the specific pixel that sent the event signal that triggered the pixel event. The pixel event stream including each pixel event generated by the event compiler 190 can then be transmitted to an image pipeline (e.g., image or video processing circuitry) associated with the event sensor 100 (not shown) for further processing.

[0031] By way of example, the pixel event streams generated by the event compiler 190 may be accumulated or otherwise combined to produce image data. In some implementations, the pixel event streams are combined to provide an intensity reconstruction image. In this implementation, an intensity reconstruction image generator (not shown) may accumulate pixel events over time to reconstruct / estimate absolute intensity values. As additional pixel events accumulate, the intensity reconstruction image generator changes the corresponding values ​​in the reconstructed image. In this way, it generates and maintains an updated image of values ​​for all pixels of the image, even though only some pixels may have recently received events.

[0032] In various implementations, event-driven sensors are implemented using a hardware architecture configured to support enabled, standby, and operational states. Generally, this involves the event sensor 210 outputting pixel events to the image pipeline 220 and, in response, receiving feedback information from the image pipeline 220, such as Figure 2The image pipeline 220 is configured to consume image data derived from pixel events output by the event sensor 210. To this end, the image pipeline 220 includes one or more components, such as those described above with reference to Figure 1 The intensity reconstruction image generator discussed above is used to derive image data from pixel events. One or more components of the image pipeline 220 may be implemented using various combinations of hardware components (e.g., application specific integrated circuits, digital signal processors, etc.) and software components (e.g., noise reduction processes, image scaling processes, color space conversion processes, etc.).

[0033] In various implementations, the image pipeline 220 implements some functionality that utilizes an incomplete image data set derived from pixel events output by the event sensor 210. By way of example, the image pipeline 220 may also include a feature tracker configured to detect features depicted in image data derived from pixel events (e.g., using techniques such as SIFT, KAZE, etc.) and track the features over time (e.g., using techniques such as Kanade-Lucas-Tomasi tracker, Shi-Tomasi tracker, etc.). In this example, the feature tracker of the image pipeline 220 may implement eye tracking functionality by detecting and tracking gaze characteristics (e.g., pupil center, pupil outline, glint position, gaze direction, etc.) using image data depicting the user's eyes derived from pixel events output by the event sensor 210.

[0034] Figure 3 An example of a full image dataset 300 is shown depicting an eye of a user that the image pipeline 220 may derive from pixel events output by the event sensor 210. To implement the eye tracking functionality, a feature tracker of the image pipeline 220 has used a subset of the image data 300 that resides in the region of interest 320 to estimate the location of the pupil center (“estimated pupil center”) 310 within the eye. Processing the full image dataset 300 to implement the eye tracking functionality may be computationally intensive for the feature tracker of the image pipeline 220 and consume excessive power and computing resources. To improve computational efficiency and reduce power consumption, the feature tracker of the image pipeline 220 may process a subset of the image data that resides in the region of interest 320. The image data that resides outside the region of interest 320 may be cropped to form cropped image data 400, as shown. Figure 4 shown.

[0035] One technique for cropping image data residing outside of the region of interest 320 may be implemented using the image pipeline 220. According to the technique, the image pipeline 220 may receive pixel events corresponding to the field of view of the event sensor 210. To form the cropped image data 400, the image pipeline 220 may ignore pixel events corresponding to image data residing outside of the region of interest 320, or crop image data residing outside of the region of interest 320 after the full image data set 300 is derived. However, in either case, the event sensor 210 includes a subset of pixels that generate pixel events corresponding to image data residing outside of the region of interest 320, which continue to consume power. In addition, the pixel events corresponding to image data residing outside of the region of interest 320 continue to consume bandwidth of the communication path between the event sensor 210 and the image pipeline 220. Therefore, implementing a technique for cropping image data residing outside of the region of interest 320 involving the event sensor 210 may further reduce power and bandwidth consumption.

[0036] To this end, the image pipeline 220 transmits feedback information to the event sensor 210, such as Figure 2 As shown. In various embodiments, such feedback information represents a feedback loop between an event sensor (e.g., event sensor 210) and an image pipeline (e.g., image pipeline 220). As discussed in more detail below, the image pipeline consumes image data derived from pixel events output by the event sensor. Based on the image data, the image pipeline generates feedback information corresponding to a subset of the image data (e.g., a region of interest), which feedback information may be more useful for a particular image processing operation than for other portions of the image data. That is, the feedback information corresponds to a subset of the image data on which processing is performed for a particular image processing operation. In response to the feedback information, the operating state of each pixel within the pixel array of the event sensor can be modified accordingly. Specifically, different pixels within the pixel array of the event sensor can have different operating states based on the feedback information received from the image pipeline.

[0037] Figure 5An example of a pixel array 500 of an event sensor is shown, wherein the pixels are configured to support different operating states. The pixel array 500 includes a plurality of pixels positioned to receive light from a scene disposed within a field of view of the event sensor. Therefore, when the operating state of each of the plurality of pixels is an enabled state, image data derived from pixel events output by the event sensor typically depict the field of view of the event sensor. As used herein, an "enabled state" refers to an operating state of a pixel in which the photodetector circuit and event circuit of the pixel are each activated (or fully functional). In a specific implementation, pixels having event circuits and photodetector circuits that are each activated (or fully functional) are defined as operating in full power mode.

[0038] When the event sensor receives feedback information from the image pipeline that an incomplete image data set is being processed by a specific image processing operation, some pixels of the event sensor can be transformed from an enabled state to another operating state. For example, some pixels of the event sensor can be transformed into a non-enabled state. As used herein, a "non-enabled state" refers to an operating state of a pixel in which the pixel function is not complete. In one specific implementation, the photodetector circuit and the event circuit of the pixel in the non-enabled state are each deactivated (or non-functional). In a specific implementation, a pixel having an event circuit and a photodetector circuit that are each deactivated (or non-functional) is defined as operating in a power-off mode.

[0039] In some cases, the pixels of the event sensor may not be able to immediately transition from the non-enabled state to the enabled state. In order to alleviate such delay problems, some pixels of the event sensor can transition from the enabled state to the standby state. As used herein, "standby state" refers to an operating state of a pixel in which the pixel function is not complete but is stronger than the pixel function in the non-enabled state. In one specific implementation, the event circuit of the pixel is deactivated (or non-functional) when the pixel is transitioned to the standby state, and the photodetector circuit of the pixel is activated (or fully functional). In a specific implementation, a pixel having a deactivated (or non-functional) event circuit and an activated (or fully functional) photodetector circuit is defined as operating in a low power mode.

[0040] By way of example, an image pipeline can be based on Figure 3 In response to the feedback information, a first subset of pixels in region 520 of pixel array 500 is in an enabled state, a second subset of pixels in region 510 is in a standby state, and a third subset of pixels outside regions 510 and 520 is in a non-enabled state. In this example, the first subset of pixels in region 520 may be in a state corresponding to the image data 300. Figure 3 and Figure 4The pixel events of the region of interest 320 are associated with each other. In one implementation, the region 520 defines an enabled region of the pixel array 500. In one implementation, a subset of pixels within an enabled region (eg, the region 520) is operated in an off power mode or a low power mode.

[0041] In one specific implementation, the feedback information includes parameters defining the location of one or more regions within pixel array 500. For example, the parameters defining the location of region 510 may include an offset value specified relative to a boundary of pixel array 500, such as x-offset 512, y-offset 514, or a combination thereof. For another example, the parameters defining the location of region 520 may include an offset value specified relative to a boundary of pixel array 500, such as some combination of x-offset 512, x-offset 522, y-offset 514, and y-offset 524.

[0042] In one implementation, one or more regions of pixel array 500 have predefined dimensions. For example, region 510 may have predefined dimensions specified as width 516 and height 518. As another example, region 520 may have predefined dimensions specified as width 526 and height 528. In one implementation, the feedback information includes parameters defining the dimensions of one or more regions within pixel array 500. For example, the parameters of the feedback information may define one or more of width 516, width 526, height 518, and height 528.

[0043] Figure 6 An example of a pixel array 600 having event sensors with different pixels having different operating states is shown, with the operating states of some pixels being modified as feedback information received from an image pipeline is updated between a first time and a second time. At the first time, the image pipeline may generate feedback information based on image data derived from pixel events output by the event sensors. In response to receiving the feedback information generated by the image pipeline at the first time, a first subset of pixels within a region 620A of the pixel array 600 is in an enabled state, a second subset of pixels within a region 610A is in a standby state, and a third subset of pixels outside of regions 610A and 620A is in a non-enabled state.

[0044] After the first time, the image pipeline may receive additional pixel events from the event sensor that change the image data being processed by the image pipeline. For example, a feature of interest (e.g., Figure 3The position of the pupil center 310 of the pixel array 600 within the image data may change as the image data is updated by the additional pixel event. At a second time, the image pipeline may generate feedback information that describes the change in the image data caused by the additional pixel event. In response to receiving the feedback information generated by the image pipeline at the second time, a first subset of pixels within region 620B of the pixel array 600 is in an enabled state, a second subset of pixels within region 610B is in a standby state, and a third subset of pixels outside of regions 610B and 620B is in a non-enabled state.

[0045] Between the first time and the second time, some pixels within pixel array 600 transition from one operating state to another operating state in response to feedback information received from the image pipeline. Figure 7 As shown, the pixels in the sub-region 710 that were in the standby state at the first time will be changed to the non-enabled state at the second time. The pixels in the sub-region 720 of the pixel array 600 that were in the enabled state at the first time will be changed to the standby state at the second time. Similarly, the pixels in the sub-region 730 that were in the standby state at the first time will be changed to the enabled state at the second time, and the pixels in the sub-region 740 that were in the non-enabled state at the first time will be changed to the standby state at the second time.

[0046] In various implementations, an event sensor (e.g., Figure 2 The event sensor 210 of FIG. 1 may be configured to output pixel events to an image pipeline (eg, image pipeline 220). Figure 1 As discussed, in various implementations, an event compiler (e.g., event compiler 190) of an event sensor may populate each pixel event with some combination of the following: (i) an address identifier corresponding to a particular pixel that sent an event signal that triggered a corresponding pixel event (e.g., the x / y coordinates of the particular pixel - [x, y]); (ii) information indicating an electrical response included in the event signal (e.g., the value or polarity of the electrical response - "intensity"); and (iii) timestamp information corresponding to a point in time ("T") at which the corresponding pixel event was generated. If the event compiler of the event sensor generates a certain number ("N") of pixel events and populates each pixel event with all three data points, then these N pixel events (pixel events 1...N) may be represented as the following pixel event list: ([x, y], intensity 1, T1), ([x, y], intensity 2, T2), ..., ([x, y], intensity N , T N). In one specific implementation, the event sensor is configured to output such pixel events as pixel event bins to an image pipeline (e.g., image pipeline 220). Generally speaking, a pixel event bin is considered to be a collection of pixel events. Those skilled in the art will recognize that binning includes grouping individual data values ​​(e.g., pixel events) into defined intervals (or bins).

[0047] In one implementation, such intervals may be defined based on event counts. For example, an event sensor may output each pixel event bin after generating a predefined number of pixel events (e.g., 10 pixel events). In this implementation, continuing the previous example and assuming N=40, the 40 pixel events generated by the event compiler may be grouped into 4 pixel event bins for output to the image pipeline. The four bins of this example would include: a first pixel event bin including pixel events 1...10; a second pixel event bin including pixel events 11...20; a third pixel event bin including pixel events 21...30; and a fourth pixel event bin including pixel events 31 to 40. In one implementation, a hardware / software-based event counter of an event sensor may monitor the number of pixel events generated by one or more event compilers and cause the event sensor to output a pixel event bin when the number reaches a predefined number of pixel events.

[0048] In one implementation, such intervals may be defined using a periodic basis, e.g., every 0.5 milliseconds ("ms"). In this implementation, continuing with the previous example and assuming regularly spaced timestamps between 40 pixel events within a 4 ms period, the 40 pixel events may be grouped into 8 pixel event bins. The eight bins of this example would include: a first pixel event bin including pixel events (pixel events 1...5) generated between 0ms and 0.5ms of a 4ms time period; a second pixel event bin including pixel events (pixel events 6...10) generated between 0.5ms and 1.0ms; a third pixel event bin including pixel events (pixel events 11...15) generated between 1.0ms and 1.5ms; a fourth pixel event bin including pixel events (pixel events 16...20) generated between 1.5ms and 2.0ms; a fifth pixel event bin including pixel events (pixel events 21...25) generated between 2.0ms and 2.5ms; a sixth pixel event bin including pixel events (pixel events 26...30) generated between 2.5ms and 3.0ms; a seventh pixel event bin including pixel events (pixel events 31...35) generated between 3.0ms and 3.5ms; and an eighth pixel event bin including pixel events (pixel events 36...40) generated between 3.5ms and 4.0ms.

[0049] In one implementation, the periodic basis is synchronized with a global readout operation or a global reset operation of the event sensor. In one implementation, the global readout operation includes the respective controller of each pixel within a particular subset of pixels (e.g., a particular row or a particular column) triggering the respective comparator to process the pixel data at a common (or substantially common) time. In one implementation, the global reset operation includes resetting the pixel data value (or voltage) to a reference value (or voltage) V each time a sample of the pixel data is processed by the respective comparator. ref .

[0050] In one implementation, each pixel event bin is output as a pixel event list (eg, similar to the pixel event list presented above). In one implementation, each pixel event bin is output as a two-dimensional ("2D") tile of pixel events. Figure 8 An example of such a 2D tile of pixel events that an event sensor may generate for output to an image pipeline is depicted. In one implementation, each pixel event is mapped to a specific location of a corresponding 2D tile of the pixel event using address identifier information. In one implementation, each 2D tile encodes the value or polarity of an electrical response provided by each pixel event included in a corresponding pixel event box. Upon receiving a particular 2D tile (e.g., tile 810), the image pipeline may identify a particular pixel that has detected both a positive incident illumination change within a given interval (e.g., pixel event 812) and a negative incident illumination change within a given interval (e.g., pixel event 814). In one implementation, the image pipeline may use the value or polarity of the electrical response encoded in the 2D tile to update an intensity reconstructed image.

[0051] Fig. 9 A block diagram of a head mounted device 900 according to some implementations is shown. The head mounted device 900 includes a housing 901 (or shell) that houses various components of the head mounted device 900. The housing 901 includes (or is coupled to) an eye pad 905 disposed at a proximal end of the housing 901 relative to a user 10 of the head mounted device 900. In various implementations, the eye pad 905 is a plastic or rubber piece that comfortably and snugly holds the head mounted device 900 in place on the face of the user 10 (e.g., around the eyes of the user 10).

[0052] In some implementations, the image data is presented to the user 10 of the head mounted device 900 via a display 910 disposed within the housing 901. Fig. 9 A head mounted device 900 is shown including a display 910 and an eye pad 905 , but in various implementations, the head mounted device 900 does not include the display 910 or includes an optical see-through display without the eye pad 905 .

[0053] The head mounted device 900 also includes a gaze tracking system disposed within the housing 901, the gaze tracking system including an event sensor 924, a controller 980, and optionally one or more light sources 922. Generally, the controller 980 is configured to communicate with the event sensor 924 and an image pipeline (e.g., Figure 2 The controller 980 interacts with a feature tracker of the image pipeline 220 of the image pipeline to detect and track the gaze characteristics of the user 10. In one specific implementation, the system includes one or more light sources 922 that emit light that is reflected from the eyes of the user 10 in a light pattern (e.g., a flashing circle) and detected by the event sensor 924. To this end, the controller 980 is configured to activate the one or more light sources 922 in response to information received from the image pipeline (e.g., feedback information). Based on the light pattern, the feature tracker of the image pipeline can determine the gaze tracking characteristics of the user 10 (e.g., gaze direction, pupil center, pupil size, etc.).

[0054] In one implementation, the controller 980 is configured to activate the light source 922 by pulsing the light source 922 at a defined frequency (e.g., 300 Hz). In one implementation, pulsing the light source 922 at the defined frequency causes at least one subset of pixels (e.g., pixels in an enabled state) within the event sensor 924 to generate event data at a rate proportional to the defined frequency.

[0055] In one implementation, no light source is used, and the eye is passively illuminated by light present in the environment. Gaze tracking characteristics (e.g., gaze direction, pupil center, pupil size, etc.) can be determined by analyzing the image and extracting features such as pupil location, appearance, and shape (e.g., using template matching, or combining corner or feature detectors with classifiers, or using a trained neural network), and correlating them with the location and appearance of additional features of the eye (such as the iris outline (limbus) or eyelid shape and eyelid corner location).

[0056] Fig.10 is a flow chart illustrating an example of a method 1000 for implementing an event camera system architecture having a feedback loop and an event-driven sensor configured to support multiple power states. In one specific implementation, the method 1000 comprises Figure 2 The method 1000 includes outputting pixel events from an event sensor having a pixel array to an image pipeline. The pixel array is configured to have an enabled pixel region operating in a full power mode and a non-enabled pixel region operating in a power-off mode or a low power mode. In response to a particular pixel in the enabled pixel region detecting a light intensity change exceeding a comparator threshold, each corresponding pixel event is generated.

[0057] In one implementation, the pixel events are output from the event sensor as pixel event bins. In one implementation, the event sensor is configured to output the pixel event bins on a periodic basis. In one implementation, the periodic basis is synchronized with a global readout operation or a global reset operation of the event sensor. In one implementation, the pixel event bins are output from the event sensor after a predefined number of pixel events are generated. In one implementation, the pixel event bins are output as 2D tiles (e.g., Figure 8 In one specific implementation, the pixel events are output from the event sensor as a list of pixel events.

[0058] At block 1004, the method 1000 includes receiving feedback information from the image pipeline based on the image data derived from the pixel events. In one specific implementation, the feedback information is an enable region offset that defines a region of the pixel array that corresponds to a region of interest tracked by the image pipeline within the image data. At block 1006, the method 1000 includes directing pixels within the enable pixel region to operate in an off power mode or a low power mode in response to receiving the feedback information.

[0059] In one embodiment, the feedback information may include a bit mask that encodes the target pixel state for each individual pixel, for example, the bit mask may represent a circular area in an enabled state while the rest of the sensor is in a ready or non-enabled state; furthermore, it can be easily seen how the mask may represent any arbitrarily shaped area or group of areas in the sensor being in one of the mentioned states, where the smallest size of the area is any individual pixel.

[0060] Fig.11 is a flow chart illustrating another example of a method 1100 for implementing an event camera system architecture having a feedback loop and an event-driven sensor configured to support multiple power states. In one specific implementation, the method 1000 comprises Figure 2 The method 1100 is implemented by the image pipeline 220. At box 1102, the method 1100 includes receiving pixel events from an event sensor having a pixel array by the image pipeline, and the pixel array includes a first pixel subset in an enabled state and a second pixel subset in a non-enabled state. In response to a specific pixel in the first pixel subset detecting a light intensity change exceeding a comparator threshold, each corresponding pixel event is generated. In a specific implementation, the pixel event is received from the event sensor as a pixel event box. In a specific implementation, the pixel event is received from the event sensor as a pixel event list.

[0061] At box 1104, method 1100 includes deriving image data from pixel events using an image pipeline. At box 1106, method 1100 includes generating feedback information based on the image data using the image pipeline. The feedback information causes the event sensor to direct pixels within the first subset of pixels to transition from an enabled state to another operating state. In one specific implementation, the feedback information causes the event sensor to direct pixels within the first subset of pixels to transition from an enabled state to a non-enabled state. In one specific implementation, the feedback information causes the event sensor to direct pixels within the first subset of pixels to transition from an enabled state to a standby state. In one specific implementation, generating the feedback information includes tracking a region of interest within the image data using the image pipeline. In one specific implementation, the feedback information is an enabled region offset that defines a region of the pixel array that corresponds to a region of interest tracked by the image pipeline within the image data.

[0062] In one implementation, method 1100 further includes pulsing a light source at a defined frequency, the light source being configured to emit light toward a scene disposed within a field of view of the event sensor. In one implementation, pulsing the light source at the defined frequency causes pixels within the first subset of pixels to generate event data at a rate proportional to the defined frequency.

[0063] The use of "suitable for" or "configured to" herein is meant to be open and inclusive language that does not exclude devices that are suitable for or configured to perform additional tasks or steps. In addition, the use of "based on" is meant to be open and inclusive, as a process, step, calculation, or other action "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbers included herein are for ease of explanation only and are not intended to be limiting.

[0064] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0065] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "a" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts or their groupings.

[0066] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

[0067] The foregoing description and summary of the present invention should be understood to be illustrative and exemplary in every aspect, rather than restrictive, and the scope of the present invention disclosed herein is determined not only by the detailed description of the exemplary specific implementations, but by the full breadth allowed by the patent law. It should be understood that the specific implementations shown and described herein are only illustrative of the principles of the present invention, and that various modifications can be implemented by those skilled in the art without departing from the scope and essence of the present invention.

Claims

1. A system comprising: an event sensor having a pixel array configured to operate a first subset of pixels in an enabled state and a second subset of pixels in an inactive state, the event sensor configured to output pixel events, each respective pixel event being generated in response to a particular pixel within the first subset of pixels detecting a change in light intensity that exceeds a comparator threshold; and An image pipeline is configured to consume image data derived from the pixel events and to transmit feedback information to the event sensor based on the image data, the feedback information causing pixels within the first subset of pixels to transition from the enabled state to a non-enabled state. 2 . The system of claim 1 , wherein the feedback information corresponds to a region of interest tracked by the image pipeline within the image data. 3 . The system of claim 1 , wherein each pixel in the second subset of pixels operates in a power-off mode. 4 . The system of claim 1 , wherein the pixel array further comprises a third subset of pixels in a standby state, each pixel in the third subset of pixels operating in a low power mode. 5 . The system of claim 4 , wherein the feedback information causes the pixels within the first subset of pixels to transition to the standby state.

6. A system comprising: an event sensor having a pixel array configured to have an enabled pixel region operating in a full power mode and an inactive pixel region operating in an off power mode or a low power mode; processor; and A computer-readable storage medium comprising instructions that, when executed by the processor, cause the system to perform operations comprising: outputting pixel events from the event sensor to an image pipeline, each respective pixel event being generated in response to a particular pixel within the enabled pixel region detecting a change in light intensity exceeding a comparator threshold; receiving feedback information from the image pipeline based on image data derived from the pixel events; and In response to receiving the feedback information, pixels within the enabled pixel region are directed to operate in the off power mode or the low power mode.

7. The system of claim 6, wherein each pixel in the non-enabled pixel region operates in the off power mode, and wherein the pixel array further includes a standby pixel region that operates in the low power mode.

8. The system of claim 7, wherein the standby pixel region is interposed between the enabled pixel region and the non-enabled pixel region within the pixel array.

9. The system of claim 6, wherein a subset of pixels within the enabled pixel region operates in the off power mode or the low power mode.

10. The system of claim 6, wherein outputting the pixel event comprises: Pixel event bins are output from the event sensor to the image pipeline.

11. The system of claim 10, wherein the pixel event bins are output from the event sensor on a periodic basis. 12 . The system of claim 11 , wherein the periodic basis is synchronized with a global readout operation or a global reset operation of the event sensor.

13. The system of claim 10, wherein the event sensor outputs each pixel event bin after a predefined number of pixel events are generated.

14. The system of claim 10, wherein each pixel event bin is output as a two-dimensional tile of pixel events.

15. The system of claim 10, wherein each pixel event bin is output from the event sensor as a pixel event list.

16. A system comprising: processor; Image pipeline; and A computer-readable storage medium comprising instructions that, when executed by the processor, cause the system to perform operations comprising: receiving, by the image pipeline, pixel events from an event sensor having a pixel array, the pixel array including a first subset of pixels in an enabled state and a second subset of pixels in an inactive state, each respective pixel event being generated in response to a particular pixel within the first subset of pixels detecting a change in light intensity exceeding a comparator threshold; deriving image data from the pixel events using the image pipeline; and Feedback information is generated using the image pipeline based on the image data, the feedback information causing the event sensor to direct pixels within the first subset of pixels to transition from the enabled state to a non-enabled state.

17. The system of claim 16, wherein generating the feedback information comprises: The image pipeline is utilized to track a region of interest within the image data.

18. The system of claim 16, further comprising: A light source is configured to emit light toward a scene disposed within a field of view of the event sensor.

19. The system of claim 18, wherein the instructions, when executed, further cause the system to perform additional operations, the additional operations comprising: The light source is pulsed at a defined frequency such that pixels within the first subset of pixels generate event data at a rate proportional to the defined frequency.

20. The system of claim 16, wherein the feedback information is an enabled region offset that defines a region of the pixel array that corresponds to a region of interest tracked by the image pipeline within the image data.