Systems and methods for ultra-low power pre-roll buffering

Abstract

Systems and methods for ultra low-power pre-roll buffering are provided. The system leverages ephemeral memory as a data buffer that is used to maintain pre-roll data that are processed after a wake-up interrupt is sent from the classification layer of an always-on neural network. The Ephemeral Memory Pre-roll Buffer (EMPB) operates without reference to a system clock which removes the power requirements associated with a system clock. The systems and methods described herein can be employed for always-on applications where a wake-up function / process wakes up a device from a sleep state to pass on information onto subsequent layers of a processing pipeline for further processing. Beyond always-on applications, the present systems and methods can be used to encode or temporarily store historical "initial" data in an ephemeral memory for later decoding or processing e.g., by a neural network.

Description

SYSTEMS AND METHODS FOR ULTRA-LOW POWER PRE-ROLL BUFFERINGTechnical Field

[0001] The embodiments disclosed herein relate to systems and devices requiring ultra low power processing, such as systems running in an always-on state, and, in particular to systems and methods for ultra-low power buffering, such as in always-on applications.Introduction

[0002] Ultra-low power systems such as always-on systems have one or more sensors that are always switched on to sense an input (e.g., an image or audio), and wake the rest of the device from a sleep state. Typically, always-on components are separate from the rest of the device’s components that are switched off in the sleep state to save power.

[0003] FIG. 1 is a diagram of a traditional always-on system 100. Digital always-on solutions 100 require encoding of the sensor information into an intermediate form as an initial stage to detect a wake-up event (e.g., a face, a sound or a vibrational pattern) and generate an output signal or interrupt to wake up the rest of the device, including the main processor 108. Existing digital encoders have a significant power draw and use a high volume of data. This is especially true for always-on video / image processing. For example, traditional always-on image sensors 102 output a full frame image to an encoder (e.g., an always-on neural network 104) configured to detect a wake-up event and generate an output signal to wake up other device components such as the processor 108.

[0004] Traditional always-on systems 100 are typically required to have a data buffer 110 to allow for the analysis of a small amount data that is captured just before a wake-up event is detected. This buffer 110 is typically known as a pre-roll buffer 110. The pre-roll buffer 110 captures full resolution data that is stored in typical memory (e.g. SRAM) and uses a level of power that compromises the system’s 100 ability to run on battery or harvested energy for long periods in an always-on state. The powerrequirements are compounded by the requirements of a system clock in traditional always-on systems 100.

[0005] Accordingly, there is a need for new ultra-low power systems and methods such as for always-on applications.Summary

[0006] Systems and methods are provided to create imaging or audio devices running with ultra-low power such as in an always-on state. The system leverages ephemeral memory as a data buffer that is used to maintain pre-roll data that are processed after a wake-up interrupt is sent from the classification layer of an always-on neural network. The Ephemeral Memory Pre-roll Buffer (EMPB) operates without reference to a system clock which removes the power requirements associated with a system clock. The EMPB is populated with encoded data output from an encoding neural network comprising charge-trapped transistors (CTTs).

[0007] According to some embodiments, there is an ephemeral memory circuit. The ephemeral memory circuit comprises an input counter configured to count up during an activation phase and a temporary memory for storing a value of the input counter reached during the activation phase. The input counter is configured to start counting when an input temporal unary data pulse is high and stop counting up when the input temporal unary data pulse is low. The ephemeral memory circuit further comprises an output counter configured to count up to the value stored in the temporary storage during an activation regeneration phase. A gated ring oscillator is configured to establish a frequency for the input counter and the output counter. An ephemeral memory state machine configured to switch the circuit between the activation capture phase and the activation regeneration phase.

[0008] According to some embodiments, there is a system comprising a sensor configured to operate in a low power state, a neural network configured to operate when the sensor is on to encode the sensor data and classify encoded sensor data to detect a wake-up event and an ephemeral memory pre-roll buffer (EMPB) configured to temporarily store the encoded sensor data until the wake-up event is detected.

[0009] A method for processing data stored in an ephemeral memory pre-roll buffer (EMPB) comprises: encoding data by a neural network configured to encode the data; temporarily storing encoded data in the EMPB; and decoding the encoded data by a second neural network configured to receive the encoded data from the EMPB. The method may further comprise detecting a wake-up event among the encoded data by a classification layer of the neural network and sending an interrupt signal to wake-up one or more components from a sleep state upon detecting the wake-up event or sending an interrupt signal to a switch that opens communication from the EMPB to a decoder upon detecting the wake-up event. The method may further comprise decoding a series of vectors or matrices by a second neural network configured to regenerate the data.

[0010] Other aspects and features will become apparent to those ordinarily skilled in the art, upon review of the following description of some exemplary embodiments.Brief Description of the Drawings

[0011] The drawings included herewith are for illustrating various examples of articles, methods, and apparatuses of the present specification. In the drawings:

[0012] FIG. 1 is a diagram of a traditional digital always-on solution;

[0013] FIGS. 2A-2C are diagrams of always-on systems, according to several embodiments;

[0014] FIG. 3 is a diagram of an ephemeral memory pre-roll buffer structure, in accordance with an embodiment; and

[0015] FIG. 4 is the ephemeral memory pre-roll buffer of FIG. 3 in further detail, in accordance with an embodiment.

[0016] FIG. 5 is another ephemeral memory pre-roll buffer structure, in accordance with an embodiment.Detailed Description

[0017] Various apparatuses or processes will be described below to provide an example of each claimed embodiment. No embodiment described below limits any claimed embodiment and any claimed embodiment may cover processes or apparatusesthat differ from those described below. The claimed embodiments are not limited to apparatuses or processes having all of the features of any one apparatus or process described below or to features common to multiple or all of the apparatuses described below.

[0018] One or more systems described herein may be implemented in computer programs executing on programmable computers, each comprising at least one processor, a data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. For example, and without limitation, the programmable computer may be a programmable logic unit, a mainframe computer, server, and personal computer, cloud based program or system, laptop, personal data assistance, cellular telephone, smartphone, or tablet device.

[0019] Each program is preferably implemented in a high-level procedural or object oriented programming and / or scripting language to communicate with a computer system. However, the programs can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Each such computer program is preferably stored on a storage media or a device readable by a general or special purpose programmable computer for configuring and operating the computer when the storage media or device is read by the computer to perform the procedures described herein.

[0020] A description of an embodiment with several components in communication with each other does not imply that all such components are required. On the contrary a variety of optional components are described to illustrate the wide variety of possible embodiments of the present invention.

[0021] Further, although process steps, method steps, algorithms or the like may be described (in the disclosure and I or in the claims) in a sequential order, such processes, methods and algorithms may be configured to work in alternate orders. In other words, any sequence or order of steps that may be described does not necessarily indicate a requirement that the steps be performed in that order. The steps of processes described herein may be performed in any order that is practical. Further, some steps may be performed simultaneously.

[0022] When a single device or article is described herein, it will be readily apparent that more than one device I article (whether or not they cooperate) may be used in place of a single device I article. Similarly, where more than one device or article is described herein (whether or not they cooperate), it will be readily apparent that a single device I article may be used in place of the more than one device or article.

[0023] When a tilde (~) is used herein, it signifies a range of ± 10% of the indicated value. For example, ~1 .0 means between 0.9 to 1 .1 .

[0024] The terms “processor,” “post processor” or “compute unit” as used herein, means a computer processor e.g., a central processing unit (CPU), a graphics processing unit (GPU), an integrated circuit, a system-on-a-chip, a microcontroller unit (MCU) or cloud services, including associated hardware and software, being configured to execute instructions, perform calculations and / or process signals / data as the case may be.

[0025] The term “ephemeral memory,” as used herein, means the ephemeral memory storage structures described in U.S. Patent Publication No. 20220374698, the entirety of which is incorporated by reference herein. Ephemeral memory is used to temporarily store / retain information and may operate asynchronously without reference to a system clock. Herein, the ephemeral memory used to store the pre-roll buffer for an always-on application (e.g., an always-on video system or an always-on audio device) is referred to as the “ephemeral memory pre-roll buffer” (“EMPB”).

[0026] The term “neural network,” as used herein, means a neural network comprising charge-trapped transistors (CTTs) for supplying synaptic weights and configured to process input signals transformed into time pulses as described in U.S. Patent Publication No. 20220374698 and U.S. Patent Publication No. 20230238014, the entirety of which is incorporated by reference herein.

[0027] The systems and methods described herein can be employed for always- on applications where a wake-up function / process wakes up a device from a sleep state to pass on information onto subsequent layers of a “pipeline” for further processing. For example, in a “gesture pipeline” a first always-on layer is configured to detect whether a hand is moving at all and then pass the data to subsequent layers of the pipeline to actually interpret the gesture. More generally, the present systems and methods can beapplied to any always-on neural network front end or wake-up function. Beyond always- on applications, the present systems and methods can be used to encode or temporarily store historical “initial” data in an ephemeral memory for later decoding or processing e.g., by a neural network.

[0028] Referring to FIG. 2A, illustrated therein is a diagram of an always-on system 200, according to an embodiment. The always-on system 200 may be incorporated into a larger device or system such as a smartphone, a laptop computer, or more generally, a mobile device powered by a battery.

[0029] The always-on system 200 includes an always-on image sensor 202 such as a low-power CMOS image sensor 202. The image sensor 202 is configured to be powered on in an always-on state to capture image data at a framerate e.g., 5 fps. Image sensor data is fed through a data pipeline that can follow different paths depending on whether the device implementing the system 200 is in a sleep state or an awake state. Generally, in a sleep state, lower framerate, lower resolution image sensor data from the image sensor 202 is encoded and buffered before it is passed to the processor 208. A high-resolution path 214 is used when the device is in an awake state to pass full frame, full resolution image sensor data from the image sensor 202 to the processor 208.

[0030] When the device is in a sleep state, image sensor data from the image sensor 202 is fed, via a low power parallel or serial interface, into an always-on neural network 204 configured to both compress / encode the image data and classify (or process) content therein, looking for a wake-up event. The compressed frame data is output as a series of vectors or matrices from the last layer (e.g., a convolutional layer of the neural network 204) immediately before a classification layer 206, to an ephemeral memory pre-roll buffer (EMPB) 210. As shown, the EMPB 210 is configured to store / retain a 2 second buffer consisting of 10 frames. According to other embodiments, the EMPB 210 may be configured to buffer longer or shorter time duration and more or fewer frames.

[0031] The series of vectors / matrices is also fed into the last classification layer 206 of the neural network 204, which is configured to process and classify the encoded frame data. When the classification layer 206 classifies the encoded frame data as a targeted wake-up event, a signal (i.e., an interrupt) is sent to the rest of the device,including the post processor 208. Generally, the interrupt signal enables communication between the EMPB 210 and downstream components. For example, the interrupt can be sent to a switch that opens communication from the EMPB 210 to downstream components such as the post processor 208 that processes the data stored in the EMPB 210. At this point, the device implementing the system 200 will fully turn on (i.e. , wake up from a sleep state) and will start operating in fully awake conditions. After wake-up, the data stored in the EMPB 210 can be sent to be decoded, for example, by the post processor 208, which will allow the data in the EMPB 210 to be further processed as defined by the specifications of the system.

[0032] Referring to FIGS. 2B and 2C, shown therein are diagrams of always-on systems 220, 240, according to other embodiments. The always-on systems 220, 240 are substantially similar to the always-on system 200 and include an image sensor 202, an always-on neural network 204 with an always-on classification layer 206, an EMBR 210 and a post processor 208 and further include a decoder 212, 214.

[0033] “Decoding,” as used herein is defined as processing that regenerates the original data towards human use. Decoding the buffered data usually consumes significantly more power relative to the power used in an always-on sleep state. Additionally, the data stored in the EMPB 210 can be further processed for use in the systems 220, 240 in a fashion not consistent with human observation. Like decoding, the further processing can occur in either the silicon chip 230 that contains the EMPB 210 (FIG. 2B) or in another system compute component such as an MCU (FIG. 2C).

[0034] As shown in FIG. 2B, decoding could be performed by a decoder neural network 212 on the same silicon chip 230 as the always-on encoding / classifying neural network 204 and EMPB 210. According to another embodiment, as shown in FIG. 2C, the decoder 214 could be on a separate silicon chip 250 which could include additional system components (e.g. Post Processor 208). Referring now to FIG. 3, shown therein is an ephemeral memory pre-roll buffer (EMPB) structure 1800 for temporarily retaining information, in accordance with an embodiment. In an aspect, the information retained may be frame data encoded by the always-on image encoder neural network 204. In an embodiment, the EMPB 1800 may be an ephemeral memory storage. In the EMPB 1800,a significant advantage over existing apparatus, devices, methods, and systems is that the apparatus may function asynchronously. The EMPB 1800 provides a solution to store data representing a pulse width (time) temporarily using nanowatt / picowatt orders of power consumption in a physically small space, which enables functionality at lower power and small silicon area.

[0035] The EMPB 1800 includes an inner ring 1804 including a plurality of subthreshold pass transistor logic (PTL) delay line blocks 1802 for providing asynchronous controllable delay. An outer ring 1806 includes D flip-flops (not shown) forming an asynchronous counter 1808 for supporting the inner ring 1804.

[0036] The EMPB 1800 may advantageously be used to store data representing a pulse width (or time) for each individual value for each encoded frame. The EMPB 1800 stores the pulse width or time in the asynchronous counter 1808, which can be subsequently decoded after a wake up event is detected.

[0037] The inner ring 1804 acts as an oscillator when enabled and is enabled for the time when a neuron capacitor in the last layer (e.g., a convolutional layer of the neural network 204) immediately before a classification layer 206 is discharging. The oscillator triggers the counter 1808 to count up when enabled. Once the neuron capacitor is discharged, the oscillator is disabled and the counter 1808 holds a value. At a subsequent time, the oscillator is enabled again and the counter 1808 counts back down to zero, during which time a pulse is generated by an asynchronous counter 1808) as the output of the EMPB 1800. The absolute frequency of the asynchronous counter generating the pulse is not critical. In an embodiment, only the temporary stability of the oscillation frequency is important for accuracy.

[0038] Referring now to FIG. 4, shown therein is an ephemeral memory pre-roll buffer (EMPB) 1900 for temporarily retaining information, in accordance with an embodiment. The EMPB 1900 is an implementation of the EMPB 1800 shown in FIG. 3 shown in greater detail. In an embodiment, the information retained is frame data encoded by the always-on image encoder neural network 204.

[0039] In an embodiment, the EMPB 1900 is ephemeral memory storage.

[0040] The EMPB 1900 includes an inner ring 1904 for providing asynchronous controllable delay. The EMPB 1900 further includes an outer ring 1906 for supporting theinner ring 1904. The outer ring 1906 includes asynchronous counters 1908. The asynchronous counters 1908 include a first asynchronous counter 1909. The inner ring 1904 and the outer ring 1906 function in tandem to achieve efficient short-term accurate storage of delay state.

[0041] The inner ring 1904 includes an analog subthreshold delay block 1902 for providing asynchronous controllable delay.

[0042] The asynchronous counters 1908 may be a plurality of D flip-flops.

[0043] Using the analog subthreshold delay block 1902 with positive feedback and the first asynchronous counter 1909, there may be created a self-timed oscillator (not shown) that oscillates based on a frequency of the delay elements. In the EMPB 1900, the first asynchronous counter 1909 is clocked by the self-timed oscillator.

[0044] A discharge period generates an activation pulse (not shown) from a neuron in the last layer (e.g., a convolutional layer of the neural network 204) immediately before a classification layer 206. The asynchronous counters 1908 may advantageously retain the stored number for a period of time. In an embodiment, the asynchronous counters 1908 preferably retain the stored number for several seconds.

[0045] The EMPB 1900 may advantageously use dynamic logic to save space and power.

[0046] During the activation pulse, the asynchronous counters 1908 count up during a count-up period. When the EMPB 1900 is to send the buffered / encoded data to a decoder 212 after a wake up event is detected, the asynchronous counters 1908 count down, enabling the analog subthreshold delay block 1902, in order to replay the process and generate a subsequent activation pulse (not shown) equal to the previous activation pulse during the count-up period.

[0047] As the asynchronous counters 1908 count back down, the stored number may be lost. In an embodiment, the asynchronous counters 1908 may be paired with another version of the counters (not shown) running in the opposite direction, i.e., counting up when the asynchronous counters 1908 count down and vice-versa.

[0048] In an embodiment, the asynchronous counters 1908 are 1 -bit asynchronous sub-threshold counters.

[0049] Further provided in FIG. 4 are examples of unit time and variation therein. In the EMPB 1900, up-counting is enabled by a storage capacitor discharge cycle. Up- counting is disabled by a threshold of a neuron. In the EMPB 1900, down-counting is enabled by a switch (i.e. , an interrupt) that is triggered when the classification layer 206 detects a wake-up event.

[0050] In an embodiment, defects in an absolute delay provided by the analog subthreshold delay block 1902 do not prevent successful operation of the EMPB 1900 so long as the EMPB 1900 is stable for short periods (measured in milliseconds) and so long as the asynchronous counters 1908 have sufficient extra states to compensate therefor.

[0051] In the EMPB 1900, a significant advantage over existing apparatus, devices, methods, and systems is that the EMPB 1900 may function asynchronously. A solution is provided to store pulse widths (time) temporarily in the nanowatt / picowatt power consumption space. An advantage of the present disclosure is functionality at lower power.

[0052] FIG. 5 shows an alternative embodiment of an ephemeral memory structure configured as an EMPB 300. The functionality of this embodiment may be similar to the EMPBs 1800, 1900 shown in FIGS. 3 and 4, and may be implemented in an always-on system (e.g., an always-on image processing system) however in the EMPB 300 a gated ring oscillator 306 is utilized to establish a frequency for the system 300. Similar to the EMPBs 1800, 1900 in FIGS. 4 and 5, the absolute frequency of the oscillator 306 is not critical as long as it remains somewhat constant over the period of time in which data is captured by the EMPB 300 and then regenerated by the EMPB 300.

[0053] During an activation capture phase, the gated ring oscillator 306 is enabled and the input counter 302 is enabled to count up as the input temporal unary data pulse is high (activation capture). Once the input data goes low, the input counter 302 stops and the value of the counter 302 is stored in the temporary storage 304. The temporary storage 304 can be constructed from any digital storage element, such as an SRAM. The activation capture process is controlled by an ephemeral memory (EM) state machine 308. Once all of the data associated with an encoded frame is captured and stored in the temporary storage 304 the EM state machine 308 can regenerate the temporal unary data (activation regeneration) by enabling the gated ring oscillator 306 and utilizing theoutput counter 306 to count up based upon data fetched from the temporary storage 304. This process is also controlled by the EM state machine 308.

[0054] It should be noted that the systems and methods described herein can be adapted to work with any sensor modality, including (but not limited to) imaging, audio, or accelerometers. The commonality between sensor data streams is that signals are input to neural networks where the data is encoded / com pressed for storage in an EMPB as well as processed through a classification layer looking for a wake-up event.

[0055] While the above description provides examples of one or more apparatus, methods, or systems, it will be appreciated that other apparatus, methods, or systems may be within the scope of the claims as interpreted by one of skill in the art.

Claims

Claims:1 . An ephemeral memory circuit, comprising: an input counter configured to start counting up during an activation capture phase when an input temporal unary data pulse is high and stop counting up when the input temporal unary data pulse is low; a temporary storage for storing a value of the input counter reached during the activation capture phase; an output counter configured to count up to the value stored in the temporary storage during an activation regeneration phase; a gated ring oscillator configured to establish a frequency for the input counter and the output counter; and an ephemeral memory state machine configured to switch the circuit between the activation capture phase and the activation regeneration phase.

2. A system, comprising: a sensor configured to operate in a low power state; a neural network configured to operate when the sensor is on, the neural network comprising: an encoding layer configured to receive sensor data from the sensor and encode the sensor data; and a classification layer configured to classify encoded sensor data to detect a wake-up event; andan ephemeral memory pre-roll buffer (EMPB) configured to temporarily store the encoded sensor data until the wake-up event is detected by the classification layer.

3. The system of claim 2, wherein the system is configured as an always-on system.

4. The system of claim 2, wherein the neural network is an analog neural network comprising charge-trap transistors.

5. The system of claim 2, wherein the encoding layer is configured to encode the sensor data as a series of vectors or matrices.

6. The system of any of claims 2, 3 or 4, wherein the classification layer is further configured to send an interrupt signal to wake-up one or more components from a sleep state upon detecting the wake-up event.

7. The systems of any one of claims 2, 3 or 4, wherein the classification layer is further configured to send an interrupt signal to enable communication between the EMPB and a decoder upon detecting the wake-up event.

8. The system of claim 7, wherein the decoder is a second neural network configured to decode the encoded sensor data stored in the EMPB.

9. The system of claim 7 further comprising a processor configured to process an output from the decoder.

10. The system of claim 8, wherein the decoder is on a chip separate from the EMBP.11 . The system of claim 8, wherein the decoder and the EMBP are on the same chip.

12. The system of claim 2, wherein the sensor is one of: an image sensor, an audio sensor, and an accelerometer.

13. The system of claim 2, wherein the wake-up event is one of: a gesture, an image, a sound, a vibration, and a motion.

14. The system of claim 2, 3 and 4, wherein the EMPB is configured to operate asynchronously without reference to a system clock.

15. A method for processing data stored in an ephemeral memory pre-roll buffer (EMPB), the method comprising: encoding data by a neural network configured to encode the data; temporarily storing encoded data in the EMPB; decoding the encoded data by a second neural network configured to receive the encoded data from the EMPB.

16. The method of claim 15 further comprising: detecting a wake-up event among the encoded data by a classification layer of the neural network; and sending an interrupt signal to wake-up one or more components from a sleep state upon detecting the wake-up event.

17. The method of claim 15 further comprising: detecting a wake-up event among the encoded data by a classification layer of the neural network; and sending an interrupt signal to a enable communication between the EMPB and a decoder upon detecting the wake-up event.

18. The method of claim 17 further comprising:decoding the series of vectors or matrices by a second neural network configured to regenerate the data.

19. The method of claim 18 further comprising: processing the data by a processor after regenerating the data.

Images