Virtualizing external memory as local to a machine learning accelerator

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for virtualizing external memory as local to a machine learning accelerator. One ambient computing system comprises: an ambient machine learning engine; a low-power CPU; and an SRAM that is shared among at least the ambient machine learning engine and the low-power CPU; wherein the ambient machine learning engine comprises virtual address logic to translate from virtual addresses generated by the ambient machine learning engine to physical addresses within the SRAM.

Description

[Previous Technology]

[0001] This manual pertains to machine learning accelerators.

[0002] A machine learning ("ML") accelerator is a device or a component on a device, for example, an integrated circuit having a specialized architecture designed to efficiently train a machine learning model, execute a machine learning model, or both train and execute a machine learning model.

[0003] An ML accelerator can be configured to perform inference passes through one or more machine learning models. Each inference pass uses inputs and learned parameter values ​​of a machine learning model to produce one or more outputs predicted by the learned model. The ML accelerator may contain one or more computation blocks. Generally, a computation block is a self-contained computational component configured to independently perform a set of computations. The blocks of an ML accelerator can be configured as a network and programmed such that each block of the ML accelerator is configured to perform a portion of the operation of an inference pass through a machine learning model. For example, if the machine learning model is a neural network, then each block in the main ML engine 150 can be configured to compute the computation of a layer of the neural network.

[0004] ML accelerators require a large amount of memory to flexibly process different types of machine learning models. If an ML accelerator is a component of an surrounding computing device (e.g., a mobile phone or other computing device) that can be in a low-power state but still monitors and responds to input from the environment, this requirement presents at least two problems:

[0005] First, allocating a large upper memory area to accommodate a large machine learning model is too costly for most peripheral computing devices using an ML accelerator. Furthermore, this approach also results in wasted memory dedicated to an ML accelerator, which can typically execute models that only require a fraction of the allocated memory. Physical space is also wasted on a tightly packed chip within a peripheral computing device designed for portability. The memory allocated to the ML accelerator cannot be used for other processing components of the device.

[0006] Secondly, memory not used for processing machine learning models still draws power from the device, thus wasting energy. This problem is exacerbated in many peripheral computing devices designed with low power consumption in mind. [Summary of the Invention]

[0007] This specification describes techniques for virtualizing external memory for use by an ML accelerator. The ML accelerator may include, for example, logic implemented as an integrated circuit for translating virtual memory addresses accessed by the ML accelerator while processing or training a machine learning model. These virtual memory addresses are translated into memory locations outside the ML accelerator (such as on RAM or on a system-level cache communicatively connected to a processing subsystem in which the ML accelerator is implemented). A machine learning model with corresponding parameters can be streamed from outside the ML accelerator and accessed by the ML accelerator to simulate readings and writings to locations in the ML accelerator's local memory.

[0008] Specific embodiments of the subject matter described in this specification may be implemented to achieve one or more of the following advantages: A small, low-power ML accelerator may be implemented to process ambient signals received by an ambient computing device. The ML accelerator may access a larger shared cache memory rather than limiting access to the larger cache memory to a single computing block of the ML accelerator.

[0009] Virtual memory addresses can be assigned to economically stream machine learning models and corresponding parameters because the ML accelerator can expand or shrink a virtual memory address range as indicated by the memory requirements of a specific machine learning model. The ML accelerator can access a machine learning model and parameters stored in memory outside the ML accelerator without having to re-stream the data; for example, it does not have to re-stream commonly reused model parameters in a convolutional neural network. Furthermore, the ML accelerator can access machine learning models and parameters stored externally without any special configuration. In other words, from the ML accelerator's perspective, it appears to be accessing the accelerator's local memory.

[0010] Similarly, in a device with that memory configuration, the ML accelerator can also access data stored in a system-level cache memory. The ability to allocate virtual memory addresses avoids the need for a large amount of memory dedicated to the ML accelerator, thereby reducing power consumption and the physical space that the ML accelerator must occupy in the surrounding computing device.

[0011] Machine learning models compiled for execution on ML accelerators can utilize memory virtualization features revealed without extensive configuration or customization. A compiler for a machine learning model can compile the model by means of the minimum additional instructions that indicate that memory virtualization should be used.

[0012] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, appearances and advantages of the subject matter will become apparent from the description, drawings and the claims.

Implementation Method

[0014] This specification describes a technique for implementing memory virtualization as local memory for a machine learning ("ML") accelerator. Instead of dedicating a large amount of memory to local memory for the ML accelerator, the ML accelerator can access external memory through virtual memory addresses automatically assigned by virtual address logic local to the ML accelerator. The virtual address logic for the ML accelerator may also include logic to enable or disable this memory virtualization feature depending on a configuration option added to a machine learning model during the compilation of a program that performs inference iterations via the model.

[0015] This and other configuration options may be included during the compilation of a machine learning model into compiled instructions that can be executed by an ML accelerator (such as one of the surrounding ML engines 134 or a main ML engine 150 discussed below).

[0016] Figure 1 is a diagram of an instance computing device 100 implementing an ML accelerator with virtual address logic. Device 100 may be contained in an ambient computing system 110 implemented in any suitable computing device (e.g., a smartphone, a smartwatch, a fitness tracker, a digital assistant, a tablet computer, a laptop computer, to name a few). The ambient computing system 110 of computing device 100 can be used such that computing device 100 can remain in a low-power state while continuously monitoring and responding to inputs from the environment by sequentially waking up the appropriate processing components of the system. Although the ambient computing system 110 is discussed herein with reference to Figure 1, a general discussion of implementing a low-power ambient computing system can be found in International Application No. PCT / US2018 / 062329, which is incorporated herein by reference in its entirety.

[0017] One or more components of the computing device 100 may be implemented on a system-on-a-chip ("SOC") within the computing device. An SOC may be an integrated circuit that includes each component of the system on a single silicon substrate or on multiple interconnected dies, for example, using silicon interposers, stacked dies, or interconnect bridges. Other components of the computing device (including a main CPU cluster 140, a main ML engine 150, or a processing subsystem 130) may be implemented on the same die or on a separate die.

[0018] The computing device 100 may include components separate from and independent of the SOC (including sensors 112, one or more displays, a battery, and other components), and may (for example) be mounted on a common housing. The computing device 100 includes a control subsystem 120 for controlling the supply of power and sensor signals to the components in the system. The device 100 includes a processing subsystem 130 for processing sensor signals and generating outputs.

[0019] Device 100 may include a plurality of peripheral sensors 112. Peripheral sensors 112 may include one or more audio sensors 102, one or more radar sensors 104, one or more touch sensors 106, a Global Positioning System (“GPS”) sensor 108, and / or an accelerometer 109. The system may include additional, fewer, or alternative peripheral sensors. Peripheral sensors 112 may be devices configured to generate sensor signals in response to environmental input.

[0020] The ambient computing system 110 may include one or more ambient interfaces 114. The ambient interface 114 may be a component of the computing device 100 that is powered on even when the device is in its lowest power state. The ambient interface 114 may include any suitable ambient interface for converting input received from the ambient sensor 112 into sensor signals to be used by the ambient computing system 110.

[0021] Each of the peripheral interfaces 114 is configured to generate an individual interrupt immediately upon detecting an environmental input. Generally, each interrupt can identify a source of sensor data, such as an identifier of a peripheral interface or sensor responsible for the interrupt. These interrupts are received and processed by one or more interrupt controllers 124. For example, upon receiving an interrupt, the interrupt controller 124 can immediately wake up a power control unit ("PCU") 121, which includes a power management unit ("PMU") 122 and a clock control unit 123. The PMU 122 can control which components of the device 100 receive power and how much power each component receives. The clock control unit 123 can control the frequency of operation of the components of the device 100.

[0022] In this specification, whenever a sensor signal is described as an input to other processing components, such input may be an analog electrical signal generated by the sensor itself, a digital representation of the sensor signal, or a processed digital representation of the sensor signal representing one or more properties of the original signal.

[0023] Upon receiving an interrupt, the PCU 121 can immediately determine, based on the source of the interrupt, which other components of the surrounding computing system 110 should be activated to further process the sensor signal that caused the interrupt. In order to provide processing support for these components, the PCU 121 can wake up the static random access memory ("SRAM") 139 and the system communication architecture.

[0024] The system communication architecture is a communication subsystem that communicatively couples internal components of the surrounding computing system 110, their communication with external components, or a combination thereof. The architecture may include any suitable combination of communication hardware (e.g., buses or dedicated interconnect circuitry systems).

[0025] Although not shown, computing device 100 may also include one or more other components typically present on such computing devices, such as a display, a modem, a graphics processing unit, a display processor, or a special purpose image processor, to name just a few. These components may be powered off during the low-power states described below and activated when the system determines that a sensor signal matches an application that requires its activation.

[0026] The device 100 also includes a main CPU cluster 140. The main CPU cluster 140 is a component of the computing device 100 that may include one or more general-purpose processors separate from the components in the processing subsystem 130. The processor of the main CPU cluster 140 generally has more computing power than any of the components in the processing subsystem 130, and therefore, the processor of the main CPU cluster 140 may also consume more power than any of the components in the processing subsystem 130.

[0027] The control subsystem 120 may also include a timer 129, which is an electronic timer capable of detecting and resolving system faults. During normal operation, the control subsystem 120 may periodically restart the timer 129 to prevent it from timeout. If the control subsystem 120 fails to restart a timer (e.g., due to a hardware failure or a programming error), the timer will elapse and generate a timeout signal. This timeout signal can be used to initiate one or more corrective actions. A corrective action may include placing the surrounding computing system 110 in a safe state and restoring normal system operation.

[0028] The processing subsystem 130 includes a peripheral machine learning engine 134. The peripheral ML engine 134 is a special-purpose processing device configured to perform inference iterations through one or more machine learning models.

[0029] The surrounding ML engine 134 may include one or more multiply-accumulate ("MAC") units for computing neural network startup or other neural network layer outputs and one or more summation registers, as well as a controller for controlling data exchange between the summation registers and the MAC units. The surrounding ML engine 134 may also include instruction memory, direct memory access paths, registers, and other processing components. In some embodiments, the surrounding ML engine 134 is a single machine learning computation block configured to accelerate the computation of machine learning inference iterations.

[0030] The peripheral ML engine includes virtual address logic 135. Virtual address logic 135 may be a dedicated circuit in the peripheral ML engine 134 that translates virtual addresses generated by the peripheral ML engine into physical memory addresses in SRAM 139, which is a non-local memory used by the peripheral ML engine 134. In this specification, the non-local memory system for a component of the computing device 100 (e.g., the peripheral ML engine 134) refers to memory used by that component and one or more other components. In other words, the non-local memory is not exclusively used by that component.

[0031] For example, SRAM 139 may be a general-purpose static random access memory device that can be shared by multiple processing components of processing subsystem 130 (e.g., low-power DSP 136, high-power DSP 138, low-power CPU 132, and peripheral ML engine 134). Therefore, SRAM is used as non-local memory for peripheral ML engine 134. In contrast, peripheral ML engine 134 may also include local memory that is exclusively used by peripheral ML engine 134 and can be integrated into the same silicon die as the rest of peripheral ML engine. For example, peripheral ML engine 134 may have local memory including one or more integrated registers. These integrated registers are used as local memory for peripheral ML engine 134 because data in the registers can be read from or written only by peripheral ML engine 134.

[0032] SRAM 139 can store sensor signals, processor instructions and data, system outputs and other data, such as neural network parameters of a neural network model implemented by or to be implemented by the surrounding ML engine 134.

[0033] Generally speaking, the difference between SRAM and DRAM is that SRAM does not require periodic renewal. As explained in more detail below, the processing components in the processing subsystem 130 can access the SRAM 139 directly or through the direct memory access ("DMA") controller 128. In some embodiments, the SRAM 139 comprises multiple memory groups, each having substantially similar data capacity, for example, each having 1 MB, 10 MB, or 100 MB. Additionally, each memory group may contain multiple memory blocks that can be individually powered off when entering a low-power state. By carefully ordering the power-off sequence of the blocks among the multiple memory groups, the SRAM memory address space can be kept contiguous.

[0034] The virtual address logic 135 can transform the virtual memory address generated by the surrounding ML engine 134. In some embodiments, the virtual address logic 135 (e.g.) uses the most significant bit of the virtual address generated by the surrounding ML engine 134 to maintain a mapping between virtual pages and physical pages.

[0035] The virtual address logic 135 can receive a request to read or write a virtual address generated by compiled instructions executed by the free surrounding ML engine 134. The virtual address logic 135 can then map the virtual address to a physical address in SRAM 139. In some embodiments, the virtual address logic 135 maps a virtual page number to a physical page number and copies the least significant bits to generate a physical address.

[0036] Transforming a virtual memory address into a physical memory address means that, alternatively, when an ML accelerator executes compiled instructions for a machine learning model, each read or write instruction on data at a memory address in the compiled instructions causes data at a corresponding physical memory address location to be read or written. In some embodiments, the virtual address logic 135 is configured to issue a read or write instruction to a corresponding physical memory address location in SRAM 139 in response to the surrounding ML engine 134 executing a read or write instruction on data at a virtual memory address location mapped to a corresponding physical memory address.

[0037] Virtual address logic 135 can map virtual memory addresses to physical memory addresses located in memory that is not local to the surrounding ML engine 134. In some embodiments, processing subsystem 130 can rewrite existing data at the location referenced by the mapped physical memory address. Virtual address logic 135 can be configured to perform this initialization step automatically or in response to a configuration option indicated by a compiler for performing an inference iteration on the machine learning model (e.g., as one or more instructions to be executed on the surrounding ML engine 134 in that compiler).

[0038] The surrounding ML engine 134 can execute a compiled program having one or more instructions that perform an inference iteration using a machine learning model by accessing allocated non-local memory. From the perspective of the surrounding ML engine 134, when data is actually accessed from a shared memory source (such as from SRAM 139 or a system-level cache), the data accessed at the virtual memory address location is considered local to the surrounding ML engine 134.

[0039] The apparatus 100 may also include a primary ML engine 150, if appropriate. The primary ML engine 150 is a special-purpose processing apparatus configured to perform inference passes (i.e., execute machine learning models on the primary ML engine 150) using one or more machine learning models. Similar to the surrounding ML engine 134, each inference pass uses inputs and learned parameter values ​​from a machine learning model to produce one or more outputs predicted by the learned model. The primary ML engine 150 may include one or more computational blocks that can be configured into a network and programmed such that each block of the primary ML engine 150 is configured to perform a portion of the operation of an inference pass using a machine learning model. A machine learning engine having multiple computational blocks is described in U.S. Patent No. 9,710,265, which is incorporated herein by reference in its entirety.

[0040] When the device 100 includes both a primary ML engine 150 and a peripheral ML engine 134, the peripheral ML engine 134 generally has fewer compute blocks and therefore has less processing power and consumes less power than the primary ML engine 150. For example, the peripheral ML engine 134 may be implemented as one or two compute blocks, while the primary ML engine 150 may have eight or more interconnect blocks.

[0041] Each computation block may have a small amount of memory locally. The amount of memory locally in a computation block is usually insufficient for processing a machine learning model alone, which is why (as discussed above) individual blocks can be configured into a network to share resources and allocate the task of processing a machine learning model for a given input as a series of subtasks assigned to each computation block.

[0042] Since the peripheral ML engine 134 is generally implemented with relatively fewer compute blocks than the main ML engine 150, one or two available compute blocks may be insufficient to process a machine learning model, even after optimizing a network configuration or partitioning the processing within a block. This may be because the compute block may lack the computational power to process the machine learning model, or because the compute block may not have sufficient memory. In some implementations, even the most basic networking or partitioning is unavailable because the peripheral ML engine 134 is implemented as a single compute block. Therefore, the virtual address logic 135 can perform memory virtualization as required by the peripheral ML engine 134 to execute a machine learning model.

[0043] Although not shown in Figure 1, the primary ML engine 150 may also include virtual addressing logic to virtualize non-local memory used by the primary ML engine 150. While the primary ML engine 150 generally has more computing resources than the surrounding ML engines 134, the primary ML engine 150 may also need to access memory that is not local to the primary ML engine 150 to execute a specific machine learning model. In such cases, the same techniques described for the virtual addressing logic 135 in the surrounding ML engine 134 can be used to implement the virtual addressing logic for the primary ML engine 150. Instead of translating virtual memory addresses into physical memory addresses in SRAM as described with respect to the surrounding ML engine 134, the virtual addressing logic for a primary ML engine 150 can be configured to translate virtual memory addresses into physical memory addresses in a system-level cache ("SLC").

[0044] An SLC can be a device or a component of a device that can cache data from memory or data to be stored in the memory of multiple different hardware devices in a system, for example, computing device 100. In other words, different cache lines of the SLC can store data belonging to different hardware devices. In some embodiments and as discussed below with respect to FIG2, virtual address logic can be implemented on the main ML engine 150 to transform virtual addresses corresponding to physical memory locations on the SLC.

[0045] Next, an example operation of the peripheral computing system 110 will be described. In this specification, the terms "wake up" and "start up" will be used to mean supplying an increased amount of power to a specific processing component or other circuitry used in electronic devices. The peripheral computing system 110 may or may not have supplied power to the wake-up or started processing component or other circuitry. In other words, the wake-up or started component may or may not have been completely powered off previously. Wake up or start up a processing component causes the processing component to execute a boot procedure and load instructions and data used by the processing component into random access memory. Alternatively, or additionally, wake up or start up a processing component may involve restarting from a previously suspended state.

[0046] When PCU 121 wakes up SRAM 139, PCU 121 may wake up not all blocks or all memory groups of SRAM 139. PCU 121 may instead wake up only a few blocks sufficient for a next component of processing subsystem 130 to determine whether to further power up components of device 100.

[0047] PCU 121 can also supply different power levels to different blocks of SRAM 139. For example, in power state monitoring, PMU 122 can supply a lower holding voltage to the entire SRAM 139 to reduce its power consumption. If the processing component does not need to access SRAM 139, PMU 122 can also supply a holding voltage to SRAM 139. In power state processing, PMU 122 can provide normal voltage to all or part of SRAM 139 and reduce voltage or not provide voltage to other parts of SRAM 139.

[0048] During the process of handling an interrupt, the peripheral computing system 110 may also wake up one or more DMA controllers 128. DMA controllers 128 can manage DMA paths that allow for higher data bandwidth for incoming sensor signals. For example, a DMA controller can be used to continuously stream audio data from a microphone to SRAM 139 for access by processing components in the processing subsystem 130. Conversely, a DMA controller can also be used to continuously stream audio data stored in SRAM 139 for output as sound through one or more speakers. DMA controllers 128 can also be used to stream any suitable sensor data to SRAM 139, but using programmed I / O is computationally cheaper than activating a DMA controller for small amounts of data. Therefore, the peripheral computing system 110 can activate and use the DMA controller 128 for relatively high bandwidth sensor data (e.g., audio data and radar data), and can use programmed I / O for other types of sensor data.

[0049] After fabricating the architecture and SRAM 139, PCU 121 can then use interrupts to determine which other components of the wake-up processing subsystem 130. For example, PMU 122 may control whether to provide power to the low-power CPU 132, low-power DSP 136, or other components of the processing subsystem 130 depending on which of one or more sensors generates an interrupt. In some embodiments, the components of peripheral interface 114 and control subsystem 120 are the only components of the device 100 that is powered on in a power monitoring state, which is a power state in which the peripheral computing system 110 is waiting to receive an interrupt due to environmental input to the computing device.

[0050] The processing components of the processing subsystem 130 may include a low-power CPU 132, a peripheral ML engine 134, a low-power DSP 136, and a high-power DSP 138. In some embodiments, the processing subsystem has multiple examples of one or more of these components, for example, multiple low-power DSPs or multiple high-power DSPs. For example, the processing subsystem 130 may have a high-power DSP dedicated to processing audio signals and a separate high-power DSP dedicated to processing radar signals. Alternatively, or additionally, the processing subsystem 130 may have a high-power DSP dedicated to processing image data.

[0051] During power status monitoring, the processing components in the processing subsystem 130 can be maintained in a hold mode. The PCU 121 can maintain a component in a hold mode by reducing or eliminating the power supplied to that component. For example, in hold mode, the PCU 121 can supply a processing component with just enough power to maintain the state of the register, rather than enough power to process the data in the register.

[0052] The low-power CPU 132 may be a general-purpose programmable processor including registers, a control circuit system, and an arithmetic logic unit ("ALU"). Generally, the low-power CPU 132 consumes less power than the main CPU cluster 140 of the computing device and may contain fewer processing cores. In some embodiments, the low-power CPU 132 is primarily a scalar processor that operates on a single instruction and a single data input. Based on the type of sensor signal, the low-power CPU 132 receives and, based on the nature of those sensor signals, can determine whether other components of the system should be activated, such as some or all, or a combination of these, of the communication architecture, DMA controller 128, and SRAM 139. After activating these components, the low-power CPU 132 may return to a non-operating state as appropriate.

[0053] The low-power CPU 132 may provide the sensor signal, or a processed version thereof, to the surrounding ML engine 134 for further interpretation. For example, if the low-power CPU 132 receives a sensor signal corresponding to an accelerometer input, the low-power CPU 132 may determine that the surrounding ML engine 134 should further process the sensor signal. The surrounding ML engine 134 may then further process the sensor signal.

[0054] One task of the ambient ML engine 134 is to use sensor signals to perform an inference iteration via a machine learning model to generate an output that can trigger the wake-up of other processing components to further process the sensor signals. In other words, the ambient ML engine 134 may receive sensor signals or a processed version thereof generated by the low-power CPU 132 or another processing component, and the ambient ML engine 134 may generate an output indicating which other processing components should further process the sensor signals.

[0055] The surrounding ML engine 134 can also execute machine learning models for various tasks, including on-chip automatic speech recognition, text-to-speech generation, or gesture recognition for users of a computing device. The surrounding ML engine 134 can provide the output generated by executing a machine learning model to a low-power CPU 132 or another processing component for additional actions.

[0056] As discussed above, the virtual address logic 135 can be configured to perform memory virtualization when instructed (e.g., according to one of the compiled instructions for a machine learning model). This option can be set during the compilation of the machine learning model by a suitable configuration compiler, for example, as a default option or in response to user prompts from a program that implements the compiler. The compiled instructions for executing the machine learning model on the surrounding ML engine 134 remain unchanged, whether memory virtualization is selected or not, but the compiler can additionally instruct the surrounding ML engine 134, for example, via one or more instructions, to use memory virtualization to execute the machine learning model.

[0057] If memory virtualization is enabled, the surrounding ML engine 134 can stream model parameters and other model configuration resources from memory sources not local to the surrounding ML engine 134 using mapped virtual memory addresses generated by virtual address logic 135. For example, model parameters for machine learning models can be stored in SRAM 139 and referenced by one or more physical memory addresses.

[0058] The compiled instructions for a machine learning model may include instructions for loading, reading, and writing data into memory. The virtual address logic may be configured such that a virtual memory address referenced in the compiled instructions of a compiled machine learning model is translated into the address of the corresponding physical memory without changing the reference in the memory of the compiled instructions.

[0059] As an additional step, the processing subsystem 130 may initially load model parameters and other configuration resources for the machine learning model from another memory device into the SRAM 139. For example, the DMA controller 128 may stream the model parameters from DRAM to the SRAM 139. The DRAM may be local to or external to the surrounding computing system 110. Then, the virtual address logic 135 may map virtual memory addresses to physical memory locations in the SRAM 139 where the model parameters are loaded. As part of streaming the parameters to the SRAM 139, the DMA controller 128 may be configured to overwrite existing data stored in the SRAM 139, or alternatively, to stream the model parameters to available space in the SRAM 139. One or more instructions in the compiled instructions of the machine learning model may specify whether the DMA controller 128 should overwrite existing data in the SRAM 139.

[0060] As discussed above, since SRAM 139 may contain multiple memory groups that may or may not be activated depending on the state of computing device 100, certain memory blocks or groups in SRAM 139 may not be available, either because there is already existing data or because such particular memory blocks or groups have not yet been activated.

[0061] Although the DRAM may be part of the peripheral computing system 110, it may also be external to the peripheral computing system 110. In some embodiments, the DRAM is external to the peripheral computing system 110 but still on the same SOC. In some embodiments, the DRAM is external to the SOC on which the peripheral computing system is implemented. In any embodiment, the DMA controller 128 may be configured to stream model parameters from the DRAM.

[0062] If the model parameters cannot be streamed to SRAM 139, for example, because SRAM 139 is currently being used by other processing components of the surrounding computing system 110, the virtual address logic 135 can directly translate the virtual memory address into the physical memory address where the model parameters are stored. In some embodiments, the virtual address logic can be configured to translate the physical memory address on DRAM where the model parameters are stored, instead of first using the DMA controller 128 to stream the model parameters to SRAM 139.

[0063] Configuration resources for the machine learning model may also be included in one or more instructions specified at compile time, which instruct the surrounding ML engine 134 how much local memory should be accessed. For example, if the compiler compiles a machine learning model into a set of instructions under the assumption that the ML engine executing the model will allocate a specific size of memory to it, then the virtual address logic may be configured to provide that size of memory as a virtual memory address mapped to the physical memory address in SRAM 139. Alternatively, at compile time, a program executing the compiler may set a memory size in response to a user prompt or preset condition.

[0064] The surrounding ML engine 134 may reference the virtual memory address as described above in conjunction with accessing data stored locally in the surrounding ML engine 134. In some embodiments in which the surrounding ML engine 134 includes one or more registers, the surrounding ML engine 134 may access data not local to the surrounding ML engine 134 via a virtual memory address, and then load the accessed data into a register of the surrounding ML engine 134. Thus, data most likely to be repeatedly accessed by the surrounding ML engine 134 (such as parameters or a subset of common parameters for one convolution in a convolutional neural network model) may be stored in the fastest memory available to the surrounding ML engine 134 (which is typically a local register).

[0065] The output generated by the ambient ML engine 134 may explicitly specify a combination of processing component IDs or an identifier of a listed power state, or the output may be a representation of a power state interpreted by a low-power processing component (e.g., a low-power CPU or a low-power DSP) to identify other higher-power processing components that should process the sensor signal. As part of this process, the low-power processing component may explicitly or implicitly determine whether any additional processing is required. For example, the low-power processing component may determine, based on the output of the ambient ML engine 134, that no additional processing is required and that the ambient computing system 110 can switch back to monitoring the power state.

[0066] In the lowest level monitoring power state, PCU 121 can keep the peripheral ML engine 134 in a low-power state or completely shut down. In the processing power state, PCU 121 may or may not supply power to the peripheral ML engine 134 depending on what sensor signals are available at the peripheral interface 114 and how the low-power CPU 132 or low-power DSP 136 interprets the signals. In some embodiments, the low-power DSP 136 or low-power CPU 132 may interpret signals to instruct PCU 121 to supply power for an additional intermediate power state, wherein the peripheral ML engine 134 is also powered on for inference cycles, but other high-power processing components are not yet powered on.

[0067] The low-power DSP 136 and the high-power DSP 138 are special-purpose processors configured for efficient decoding and processing of highly vectorized signals. The processing subsystem 130 may include various DSPs designed for different purposes. For example, the processing subsystem 130 may include a DSP configured to process radar signals or a DSP configured to process audio signals.

[0068] As explained above, the low-power DSP 136 can perform initial interpretation of sensor signals from the control subsystem 120. Similarly, the low-power DSP 136 can also perform other signal processing tasks. Generally speaking, high-power DSPs consume more power than low-power DSPs because they have more active registers, access and process more data in parallel, and rely more heavily on memory operations, or a combination thereof.

[0069] FIG2 is a diagram of an exemplary system implemented on a SOC 200 that is communicatively connected to an SLC of the computing device 100 of FIG1. ​​For example, the SOC 200 may be mounted on or integrated into the computing device 100, or may be a separate device or a component of a separate device.

[0070] The computing device component 250 can communicate with an SLC 230 through a SOC architecture 240. The computing device component 250 can be any component on the computing device 100 configured to communicate with the SLC 230, and can include a main ML engine 150, a main CPU cluster 140 and a peripheral computing system 110.

[0071] The SOC architecture 240 is a communication subsystem of the SOC 200 and may include communication paths that allow the computing device components 250 to communicate with each other and issue requests to read and write data on the SLC 230. The SLC 230 has a dedicated cache memory that can be implemented using a dedicated register or high-speed RAM. The SOC architecture 240 may include any suitable combination of communication hardware (e.g., buses or dedicated interconnect circuitry systems).

[0072] The SOC 200 also includes a communication path 252 that allows communication between the SLC 230 and a memory controller 220, and an inter-chip communication path 254 that allows communication between the memory controller 220 and the DRAM 210, which is not part of the SOC 200. The memory controller 220 can handle requests to read memory from the SLC 230 and the DRAM 210 and to write memory to the SLC 230 and the DRAM 210. Although only the DRAM 210 is shown in FIG. 2, the memory controller 220 can communicate with other memory devices not shown (such as any volatile or non-volatile memory devices, for example, a hard disk drive or a solid-state drive).

[0073] The SLC 230 can retrieve read requests, write requests, or both from the computing device component 250 cache. The SLC 230 can retrieve read requests from the user device cache by responding to the request with data stored in the cache memory instead of fetching data from the DRAM 210. Similarly, the SLC can retrieve write requests from the user device cache by writing new data to the cache memory instead of writing new data to the DRAM. The SLC 230 can then perform a write-back at a later time to store the updated data in the DRAM 210.

[0074] As discussed above, the main ML engine 150 may include virtual address logic to transform virtual memory addresses into physical memory addresses mapped to memory not local to the main ML engine 150. Given that the virtual address logic 135 maps physical memory addresses to the shared SRAM 139 of the processing subsystem 130, in some embodiments, the virtual address logic for the main ML engine 150 may map virtual memory addresses to physical memory addresses located in the SLC 230.

[0075] When memory virtualization is enabled for one of the machine learning models executed on the main ML engine 150, the virtual address logic can be configured to communicate with the SLC 230 through the SOC architecture 240 to translate the virtual memory address into the physical memory address in the DRAM 210. As an initialization step, the memory controller 220 can stream model parameters that are first streamed from the DRAM 210 or other memory devices not local to the SOC 200 into the SLC 230.

[0076] Figure 3 is a flowchart of an example program for executing a machine learning model on an ML accelerator using memory virtualization. For convenience, the program in Figure 3 will be described as being executed by one or more computer systems located in one or more locations. For example, a peripheral computing system (e.g., peripheral computing system 110 of Figure 1) appropriately programmed according to this specification can execute the program in Figure 3. Additional details regarding the system for implementing the program in Figure 3 can be found in the above description of Figures 1 and 2.

[0077] The system streams model parameters from a memory device not local to the system and streams such model parameters to a shared memory device (310). As discussed above with reference to Figures 1 and 2, model parameters for executing a machine learning model on an ML accelerator can be stored in memory not local to the system, such as DRAM. For example, the system can stream model parameters and any configuration options for executing the machine learning model through a DMA controller. For example, the memory device can be shared SRAM. In some implementations, as discussed earlier, the system can overwrite existing data in the shared memory device. As discussed above, the ML accelerator can be a peripheral ML engine or the main ML engine of the system. If the ML accelerator is the main ML engine of the system, the shared memory device can be a system-level cache.

[0078] The system generates virtual memory addresses through virtual address logic on the ML accelerator. These virtual memory addresses are mapped to the corresponding physical memory addresses in the shared memory device (320) where model parameters are stored. As discussed above with reference to Figure 1, the virtual address logic can generate virtual memory addresses based on memory addresses referenced in read or write instructions in the compiled instructions used for the machine learning model. The virtual memory logic can also be configured to generate a mapping between virtual memory addresses and physical memory addresses and store the mapping in local memory of the ML accelerator, such as a temporary register.

[0079] The system executes a compiled program to perform an inference iteration on a machine learning model by executing compiled instructions transformed by virtual address logic (300). As discussed above with reference to Figure 1, the system executes a compiled program that performs an inference iteration using a machine learning model (specifically, using model parameters stored in a shared memory device). As also discussed above, the transformation between virtual memory addresses and physical memory addresses means that appropriate read or write instructions are issued to a physical memory address location, which causes the read or write instructions in the compiled instructions to be matched to a location of a virtual memory address mapped to that physical memory address.

[0080] The objects and embodiments of functional operation described in this specification can be implemented in digital electronic circuit systems, tangibly embodied computer software or firmware, computer hardware, or in one or more combinations thereof, including the structures disclosed in this specification and their structural equivalents. Embodiments of the objects described in this specification can also be implemented as one or more computer programs, i.e., one or more computer program instruction modules encoded on a tangible non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof. Alternatively, or additionally, program instructions can be encoded on an artificially generated propagated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0081] The term "data processing device" means data processing hardware and encompasses all kinds of devices, apparatuses, and machines for processing data. By way of example, it includes a programmable processor, a computer, or multiple processors or computers. The device may also be or further include a special purpose logic circuit system, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit). In addition to hardware, the device may also include, where appropriate, code that creates an execution environment for computer programs, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination thereof.

[0082] A computer program, also referred to or described as a program, software, a software application, an application, a module, a software module, a code, or a piece of code, may be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages) and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not need to, correspond to a file in a file system. A program may be stored in a portion of a file that stores other programs or data (e.g., stored in one or more code snippets in a markup language file), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., a file that stores portions of one or more modules, subroutines, or pieces of code). A computer program can be deployed to run on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communication network.

[0083] One or more computer systems configured to perform specific operations or actions means that the system has software, firmware, hardware, or a combination thereof installed thereon that causes the system to perform the operations or actions during operation. One or more computer programs configured to perform specific operations or actions means that one or more programs contain instructions that cause the data processing device to perform such operations or actions when executed by the data processing device.

[0084] As used in this specification, an "engine" or "software engine" refers to a hardware or software implementation of an input / output system that provides an output other than an input. An engine may be implemented in a dedicated digital circuit system or as computer-readable instructions to be executed by a computing device. Each engine may be implemented on any suitable type of computing device (e.g., server, mobile phone, tablet computer, laptop computer, music player, e-book reader, laptop or desktop computer, PDA, smartphone or other fixed or portable device) that includes one or more processors and computer-readable media. In addition, two or more engines may be implemented on the same computing device or on different computing devices.

[0085] The programs and logic flows described in this specification can be executed by one or more programmable processors that execute one or more computer programs to perform functions by manipulating input data and generating output. Such programs and logic flows can also be executed by a special purpose logic circuit system (e.g., an FPGA or an ASIC) or by a combination of a special purpose logic circuit system and one or more programmable computers.

[0086] A computer suitable for executing a computer program may be based on a general-purpose or special-purpose microprocessor or both, or any other type of central processing unit. Generally, a central processing unit receives instructions and data from a read-only memory or a random access memory or both. The basic components of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory may be supplemented by or incorporated into special-purpose logic circuitry. Generally, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data or be operatively coupled to receive data from, transfer data to, or both receive and transfer data from such mass storage devices. However, a computer need not have such devices. In addition, a computer can be embedded in another device, such as a mobile phone, a digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name just a few.

[0087] Computer-readable media suitable for storing computer program instructions and data includes all forms of non-volatile memory, media and memory devices, including by way of example: semiconductor memory devices, such as EPROM, EEPROM and flash memory devices; magnetic disks, such as internal hard disks or removable magnetic disks; magneto-optical disks; and CD-ROM and DVD-ROM discs.

[0088] To provide interaction with a user, embodiments of the subject matter described in this specification can be implemented on a host device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user and a keyboard and pointing device (e.g., a mouse, trackball, or presence-sensitive display or other surface) by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with a user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound, voice, or tactile input). Additionally, a computer can interact with a user by sending and receiving files from a device used by the user; for example, by sending a webpage to a web browser in response to a request received from a web browser on the user's device. Furthermore, a computer can interact with a user by sending text messages or other forms of messages to a person's device (e.g., a smartphone that runs a messaging application and then receives responsive messages from the user).

[0089] In addition to the embodiments described above, the following embodiments are also innovative:

[0090] Embodiment 1 is an ambient computing system, which includes: an ambient machine learning engine; a low-power CPU; and an SRAM shared at least in the ambient machine learning engine and the low-power CPU; wherein the ambient machine learning engine includes virtual address logic to convert virtual addresses generated by the ambient machine learning engine into physical addresses in the SRAM.

[0091] Embodiment 2 is a peripheral computing system as in Embodiment 1, wherein the peripheral computing system is configured to stream parameters for a machine learning model from a separate DRAM to the SRAM.

[0092] Embodiment 3 is a peripheral computing system as described in any of Embodiments 1 to 2, wherein the peripheral computing system is integrated into a system single chip, and wherein the DRAM is separate from the system single chip.

[0093] Embodiment 4 is a peripheral computing system as in any of Embodiments 1 to 3, wherein the parameters of the machine learning model are streamed from the DRAM to rewrite the instructions or data previously in the SRAM.

[0094] Example 5 is a peripheral computing system as described in any of Examples 1 to 4, wherein the peripheral machine learning engine is configured to perform an inference iteration via a machine learning model by reading parameters from the SRAM.

[0095] Embodiment 6 is a peripheral computing system as described in any of Embodiments 1 to 5, wherein reading the parameters of the machine learning model from the SRAM includes executing a read instruction by the peripheral machine learning engine having a virtual address that does not correspond to an address in the SRAM.

[0096] Embodiment 7 is a peripheral computing system as described in any of Embodiments 1 to 6, wherein the peripheral machine learning engine is configured to provide the virtual addresses of the read instruction to the peripheral machine learning engine in order to generate physical addresses representing the locations in the SRAM.

[0097] Embodiment 8 is a peripheral computing system as described in any of Embodiments 1 to 7, wherein the peripheral machine learning engine is configured to execute instructions generated from an input program by a compiler, the compiler taking a size of local memory available for the peripheral machine learning engine as an input argument, and wherein the instructions are generated by providing a size of SRAM, which is available local memory, to the compiler.

[0098] Embodiment 9 is an ambient computing system as described in any of Embodiments 1 to 8, wherein the ambient computing system is configured to perform operations including: receiving an interrupt indicating reception of one or more sensor signals to be processed; activating a first processing component of one or more other processing components, including streaming instructions to be executed by the first processing component to the SRAM; processing the one or more sensor signals using the instructions in the SRAM by the first processing component to determine that the ambient machine learning engine should further process the one or more sensor signals; activating the ambient machine learning engine, including streaming parameters to be used by the ambient machine learning engine to the SRAM; and performing an inference iteration of a machine learning model using the parameters stored in the SRAM by the ambient machine learning engine.

[0099] Embodiment 10 is a peripheral computing system as described in any of Embodiments 1 to 9, wherein streaming the parameters to be used by the peripheral machine learning engine to the SRAM will overwrite the instructions stored in the SRAM that are executed by the first processing component.

[0100] Example 11 is a peripheral computing system as described in any of Examples 1 to 10, wherein the SRAM is further shared with a direct memory access controller, one or more other machine learning engines, or one or more other processors.

[0101] Example 12 is a peripheral computing system as described in any of Examples 1 to 11, wherein the one or more other machine learning engines include a primary machine learning engine.

[0102] Embodiment 13 is a method for virtualizing memory on a peripheral computing system, the system comprising: a peripheral machine learning engine including virtual address logic; a low-power CPU; and an SRAM shared at least in the peripheral machine learning engine and the low-power CPU, wherein the method comprises: receiving a virtual address generated by the peripheral machine learning engine by the virtual address logic of the peripheral machine learning engine; and converting the virtual address generated by the peripheral machine learning engine into a physical address in the SRAM by the virtual address logic on the peripheral machine learning engine.

[0103] Example 14 is a method for virtualizing memory as in Example 13, wherein the surrounding computing system is as in any of Examples 1 to 12.

[0104] Embodiment 15 is a computer-readable storage medium encoding one or more instructions to be executed by a peripheral machine learning engine of a peripheral computing system, the peripheral computing system further including a low-power CPU and at least one SRAM shared by the peripheral machine learning engine and the low-power CPU, wherein one or more of the instructions include a virtual address corresponding to a physical address in the SRAM shared by the peripheral machine learning engine and the low-power CPU.

[0105] Embodiment 16 is one or more computer-readable storage media as in Embodiment 15, wherein the surrounding computing system is any one of Embodiments 1 to 12.

[0106] Although this specification contains numerous specific implementation details, such details should not be construed as limiting the scope of any invention or the scope of the claims, but rather as descriptions of features specific to particular embodiments of a particular invention. Specific features set forth in this specification within the context of a single embodiment may also be implemented in combination within a single embodiment. Conversely, various features set forth in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in a particular combination and even initially claimed, one or more features from a claimed combination may be removed from that combination in certain cases, and the claimed combination may be for a sub-combination or a variation thereof.

[0107] Similarly, although operations are depicted in a specific order in the diagrams, this should not be construed as requiring the execution of such operations in the shown specific order or sequential order, or the execution of all illustrated operations to achieve the desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system modules and components in the embodiments described above should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.

[0108] Specific embodiments of the subject matter have been described. Other embodiments exist within the scope of the appended claims. For example, the actions referenced in the claims can be performed in a different order and still achieve the desired result. As an example, the procedures illustrated in the drawings do not necessarily require the specific order or sequence shown to achieve the desired result. In certain situations, multitasking and parallel processing may be advantageous. [Simplified Explanation of the Diagram]

[0013] Figure 1 is a diagram of an instance computing device implementing an ML accelerator with virtual address logic. Figure 2 is a diagram of an instance system implementing an SOC communicatively connected to the SLC of the computing device in Figure 1. Figure 3 is a flowchart of an instance program for executing a machine learning model on an ML accelerator using memory virtualization. In the various diagrams, similar element symbols and names indicate similar elements.

Claims

1. An apparatus comprising: A primary memory shared by multiple client devices, wherein the multiple client devices include an ambient computing device; wherein the ambient computing device includes multiple ambient processing devices and a shared local memory shared by the multiple ambient processing devices, wherein the multiple ambient processing devices include an ambient machine learning (ML) engine; wherein the ambient computing device includes virtual address logic configured to translate virtual addresses used by the ambient ML engine into physical addresses of the shared local memory.

2. The device as claimed in claim 1, wherein the peripheral computing device is configured to process sensor signals before other user devices sharing the primary memory are started from a low-power state.

3. The apparatus of claim 1, wherein the plurality of client devices include a primary ML engine that generates entity addresses in the primary memory.

4. The apparatus of claim 1, wherein, upon receiving an interrupt, the apparatus is configured to stream model parameters from the primary memory to the shared local memory.

5. The apparatus of claim 4, wherein the model parameters are streamed to the shared local memory to overwrite space in the shared local memory used by one or more other peripheral processing devices.

6. The apparatus of claim 4, wherein streaming the model parameters to the common local memory includes streaming the model parameters to the available space in the common local memory.

7. The apparatus of claim 4, wherein the interruption indicates the reception of one or more sensor signals to be processed.

8. The apparatus of claim 4, wherein the surrounding ML engine is configured to use the virtual address logic to access the model parameters transmitted via streaming to the shared local memory to perform an inference pass of a machine learning model.

9. The apparatus of claim 1, wherein the shared local memory comprises a plurality of memory banks configured to be individually powered down when a low-power state is entered.

10. The apparatus of claim 1, wherein the plurality of peripheral processing devices includes at least one of a direct memory access controller, one or more other ML engines, or one or more processors.

11. The apparatus of claim 1, wherein the surrounding ML engine comprises a single ML compute tile.

12. A system comprising: A plurality of user-end devices include a peripheral computing unit and a main memory shared by the plurality of user-end devices; wherein the peripheral computing unit includes a plurality of peripheral processing units and a shared local memory shared by the plurality of peripheral processing units, wherein the plurality of peripheral processing units include a peripheral machine learning (ML) engine; wherein the peripheral computing unit includes virtual address logic configured to translate virtual addresses used by the peripheral ML engine into physical addresses of the shared local memory.

13. The system of claim 12, wherein the peripheral computing device is configured to process sensor signals before other user devices sharing the primary memory are started from a low-power state.

14. The system of request item 12, wherein the plurality of client devices include a main ML engine that generates entity addresses in the main memory.

15. The system of claim 12, wherein, upon receiving an interrupt, the device is configured to stream model parameters from the primary memory to the shared local memory.

16. The system of claim 15, wherein the model parameters are streamed to the shared local memory to overwrite the space in the shared local memory used by one or more other peripheral processing devices.

17. The system of claim 15, wherein streaming the model parameters to the shared local memory includes streaming the model parameters to the available space in the shared local memory.

18. The system of request item 15, wherein the interruption represents the reception of one or more sensor signals to be processed.

19. The system of claim 15, wherein the surrounding ML engine is configured to use the virtual address logic to access the model parameters that are streamed to the shared local memory to perform an inference iteration of a machine learning model.

20. The system of claim 12, wherein the shared local memory comprises a plurality of memory groups configured to individually disconnect power when entering a low-power state.

Images