MEMORY MANAGEMENT FOR MATHEMATICAL OPERATIONS IN COMPUTING SYSTEMS HAVING HETEROGENEOUS MEMORY ARCHITECTURES - Patent application

Abstract

Some aspects of the present disclosure provide techniques and apparatus for performing mathematical operations on a processor. The method generally includes initializing at least a portion of weight data of a machine learning model in a first memory component associated with the processor. Input data is stored in a second memory component coupled to the processor. An operation using the machine learning model is performed based on at least a portion of the weight data and the input data via a functional unit associated with the processor. Results of the operation using the machine learning model are stored in the second memory component.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS)

[0001] This application claims priority to U.S. patent application Ser. No. 17 / 934,178, filed Sep. 21, 2022, which is incorporated herein by reference.

[0002] introduction Aspects of the present disclosure relate to memory management during the execution of mathematical operations on computing devices having heterogeneous memory architectures.

[0003]

[0003] The von Neumann architecture implemented by many computing systems generally separates the processor and memory, such that the processor reads and writes data from and to the memory via a bus to perform operations. The memory of a von Neumann architecture computing system may be divided into different portions having different performance characteristics and different distances from the processor. For example, the memory may be organized into a hierarchy in which smaller, faster memory components have higher performance and may be located closer to the processor than larger, slower memory components. For example, high-performance memory may be used as a cache to temporarily store data and may be located closer to or co-located with the processor that performs various operations on that data. An off-processor memory component may be used to temporarily store other data not currently being used by the processor but that has been used recently or may be expected to be used in the near future. This off-processor memory component may be larger than the cache but may have a longer latency (e.g., longer read / write times) relative to the cache. Still other memory components that are further away from the processor and have lower performance characteristics may be included in a computing system, such as solid state or hard disk drives on which data is persistently stored, and network storage devices from which the processor may retrieve data for processing over a network connection.

[0004]

[0004] Due to the cost of high-performance memory that can be used to store data close to the processor that performs operations on that data, computing systems may be designed to have a limited amount of high-performance memory and a greater amount of lower-performance memory. Often, the amount of high-performance memory included in a computing system may be significantly less than the amount of memory required to perform various mathematical operations on the computing device, such as mathematical operations used to perform various machine learning tasks. Thus, to perform various mathematical operations on the computing device, the computing device may need to swap data between high-performance memory and low-performance memory, which may use power each time data is swapped. This power usage may be cumulative and, therefore, may have a significant impact on energy consumption, heat generation (and corresponding cooling requirements), battery life, and other characteristics of the computing device.

[0005]

[0005] Therefore, what is needed are improved techniques for memory management in computing systems that perform mathematical operations. Summary of the Invention

[0006]

[0006] Certain aspects provide a computer-implemented method for performing an operation using a machine learning model on a processor. The method generally includes initializing at least a portion of weight data of the machine learning model in a first memory component associated with the processor. Input data is stored in a second memory component coupled to the processor. The operation using the machine learning model is performed based on at least a portion of the weight data and the input data via a functional unit associated with the processor. Results of the operation using the machine learning model are stored in the second memory component.

[0007]

[0007] Another aspect provides a processing system configured to perform the aforementioned methods and methods described herein; a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the aforementioned methods and methods described herein; a computer program product embodied on a computer-readable storage medium comprising code for performing the aforementioned methods and methods further described herein; and a processing system comprising means for performing the aforementioned methods and methods further described herein.

[0008] The following description and the annexed drawings set forth in detail certain illustrative features of the one or more embodiments. [Brief explanation of the drawings]

[0009]

[0009] The accompanying drawings illustrate some aspects of one or more embodiments and therefore should not be considered limiting of the scope of the present disclosure. [Figure 1A]

[0010] FIG. 1 illustrates an exemplary processor in which different data related to machine learning operations are stored in different types of memory associated with the processor, according to aspects of the present disclosure. [Figure 1B] FIG. 1 illustrates an exemplary processor in which different data related to machine learning operations are stored in different types of memory associated with the processor, according to aspects of the present disclosure. [Figure 1C] FIG. 1 illustrates an exemplary processor in which different data related to machine learning operations are stored in different types of memory associated with the processor, according to aspects of the present disclosure. [Figure 2]

[0011] 1 is a flowchart illustrating operations for performing a computation using a machine learning model based on weight data loaded into nonvolatile random access memory and other data in dynamic random access memory, according to aspects of the present disclosure. [Figure 3]

[0012] FIG. 1 illustrates example operations for using a machine learning model with different data related to the operations stored in different types of memory associated with a processor, according to aspects of the present disclosure. [Figure 4]

[0013] FIG. 1 illustrates an example implementation of a processing system in which operations using machine learning models may be performed with different data stored in different types of memory, according to aspects of the present disclosure.

[0010]

[0014] For ease of understanding, wherever possible, like reference numerals have been used to designate like elements common to the figures. It is contemplated that elements and features of one embodiment may be beneficially incorporated in other embodiments without further recitation. DETAILED DESCRIPTION OF THE INVENTION

[0011]

[0015] Aspects of the present disclosure provide techniques for storing data used in performing operations using machine learning models across different types of memory components associated with a processor.

[0012]

[0016] Computing systems (such as those implementing a von Neumann architecture) typically organize memory by performance and proximity to the processor that performs operations using the memory. Generally, a small amount of high-performance memory may be located near or co-located with the processor, and a larger amount of slower memory may be located further away from the processor. For example, a processor cache may be faster and located closer to the processor than main system memory, which may be faster and located closer to the processor than on-device persistent storage (e.g., solid-state drive), which may be faster and located closer to the processor than off-device persistent storage (e.g., removable media, network drive, etc.). Data currently being used by the processor may be stored in one or more registers or caches associated with the processor (e.g., co-located on the same package), while data previously used by the processor or that will be used by the processor in the future may be stored in system memory or persistent storage.

[0013]

[0017] Generally, a cache may be a relatively small memory device implemented as static random access memory (SRAM) or other high-performance memory (e.g., high bandwidth memory (HBM)), which may offer fast read and write speeds. In contrast, persistent memory, which may be located farther from the processor than the cache, may be implemented as nonvolatile random access memory (NVRAM), such as resistive random access memory (RRAM), magnetoresistive random access memory (MRAM), or other memory devices that may offer slower read and write speeds than SRAM or other high-performance memory, but may offer higher capacity at a lower cost. Furthermore, dynamic random access memory (DRAM) also offers high capacity at a low cost, but may offer slower read and write speeds than SRAM or NVRAM, consuming additional power due to memory refresh operations required to retain data in the memory. In some cases, different types of memory may be fabricated on different process nodes. Generally, high-performance memory may be fabricated on the process node used to fabricate the processor in which the high-performance memory is implemented for use as a cache, while lower-performance memory may be fabricated on an older, larger process node. For example, high-performance memory may be fabricated on a current process node such as 7 nm, while lower-performance memory may be fabricated on a process node such as 14 nm, 22 nm, or even larger (e.g., a node that is at least two generations older (larger) than the current process node).

[0014]

[0018] Machine learning models such as convolutional neural networks, recurrent neural networks, and the like can be used for a variety of tasks. For example, neural networks can be used for spatial scaling to employ artificial intelligence techniques in adjusting the resolution of an image (e.g., increasing the resolution of an input image using super-resolution techniques), for adjusting the appearance of an image (e.g., through image fusion, applying various color effects such as generating high dynamic range (HDR) images, introducing background or foreground blur (also known as "bokeh"), etc.), and for temporal interpolation to allow frames to be generated at a higher frame rate (e.g., corresponding to the refresh rate of the display on which the frames will be displayed).

[0015]

[0019] In general, operations using a machine learning model may involve multiplication and / or accumulation operations involving weights and input data defined for the machine learning model, which may be retrieved for processing from external data sources or from other parts of the machine learning model. Large amounts of data may be used when performing operations using a machine learning model, and it may be impractical or impossible to store data used in machine learning operations in a higher-performance memory device. Therefore, data may be swapped from a higher-performance memory to a lower-performance memory, and vice versa. However, the swapping process may involve both power overhead and a performance penalty. For example, a multiply-accumulate function may involve some amount of power per byte (e.g., 2.5 pJ), and a swap operation moving data from DRAM to SRAM may consume a significant amount of that power (e.g., 90 percent, or 2.25 pJ per byte). Thus, in the context of billions of bytes transferred between DRAM and SRAM during operation of a machine learning model, these swapping operations may add processing latency and impose significant power overhead on the processor. These swapping operations can therefore have a negative impact on the battery life of mobile devices on which computations using machine learning models are performed. Furthermore, because processors in computing systems implementing a von Neumann architecture may not operate on data until that data is loaded into appropriate memory, and data processing cannot be performed simultaneously with the loading of data from memory, processing cycles may be wasted in idle mode until the appropriate data is loaded into memory. The wasted processing cycles while waiting for data to be loaded into appropriate memory are sometimes referred to as a "von Neumann bottleneck."

[0016]

[0020] To improve the performance of operations using machine learning models, various techniques may locate computations near or together with memory (e.g., co-located with memory). For example, in-memory computation techniques may enable data to be stored in SRAM and analog computations to be performed in memory using modified SRAM cells. In another example, in-memory function or processing techniques may locate digital computation capabilities near memory devices (e.g., DRAM, SRAM, MRAM, etc.) where weight data and data to be processed are located. However, in each of these techniques, many data transfer operations may still need to be performed to move data in and out of memory for computation (e.g., when computation is co-located with some (but not all) of the memory in a computing system).

[0017]

[0021] During an operation using a machine learning model, such as training a machine learning model or an inference operation using the machine learning model, data to be processed and results of processing such data may be moved in and out of memory (e.g., a cache, a high-performance memory coupled to a processor, a memory in an in-memory computing system, etc.) periodically (e.g., when new inputs are received for processing and results of processing these inputs are output). However, weight data defining the machine learning model may be used repeatedly during an operation using the machine learning model. Because weight data generally remains static during inference and any given training epoch, efficiencies in power consumption, latency, etc. may be obtained by storing weight data persistently, or at least semi-persistently, in memory, since such weight data does not need to be constantly written and rewritten during an operation using the machine learning model.

[0018]

[0022] Aspects of the present disclosure provide techniques that allow different types of data to be stored in different memory components associated with a processor to take advantage of the performance characteristics of these memory components and the static or dynamic nature of the data stored in these memory components. Typically static data, such as weight data, may be written once to a first memory component, while more dynamic data, such as inputs to a machine learning model and data generated using the machine learning model, may be written to a second memory component coupled to the processor. The second memory component may provide higher write performance than the first memory component, for example, so that input data can be written to the second memory component and read from the second memory component as needed. By writing weight data or other static data to the first memory component and writing data generated using a machine learning model and input data to the second memory component, aspects of the present disclosure may reduce the amount of power used during machine learning operations to swap data in and out of memory. These techniques may therefore generally increase power efficiency, particularly extending battery life on mobile devices on which machine learning operations are performed, reducing memory latency during machine learning operations, reducing heat generation, and therefore generally improving the performance of devices on which machine learning operations are performed.

[0019] Exemplary Computing Device Architecture

[0023] 1A-1C illustrate an exemplary processing unit in which different data relevant to machine learning operations are stored in different types of memory associated with the processor.

[0020]

[0024] Generally, processing units 100A-100C include DRAM 110, NVRAM 120, functional units 130, and arbiter 140, connected via bus 150. In each of Figures 1A-1C, the layout of DRAM 110, NVRAM 120, functional units 130, and arbiter 140 is for illustrative purposes only. Note that the layout of these components may vary, and that these components may be located closer together than shown in Figures 1A-1C.

[0021]

[0025] DRAM 110 generally provides a location where input data and data generated using the machine learning model may be stored for processing by functional units 130, while NVRAM 120 provides a location where weight data for the machine learning model may be stored. Generally, DRAM 110 may provide higher performance than NVRAM 120. For example, the latency in writing and / or reading data from DRAM 110 may be lower than the latency in writing and reading data from NVRAM 120. Because the weight data for the machine learning model may have a high reuse rate during operations using the machine learning model, the weight data may be written once and read many times from NVRAM 120. In contrast, because the input data and data generated using the machine learning model are written to and read from memory by functional units 130 many times during operations using the machine learning model, the input data and data generated using the machine learning model may be written to and read from DRAM 110 many times.

[0022]

[0026] During machine learning operations, data used during computations using the machine learning model may be received at arbiter 140 from an application processor, which may be a processor that invokes the machine learning operations on processing units 100A-100C. In some aspects, as shown in FIGS. 1A and 1B, the application processor may be a die 102 or an external application processor remote from a package containing die 102. Generally, die 102 may be a monolithic semiconductor substrate (e.g., a silicon substrate, a silicon-germanium substrate, a silicon-on-insulator (SOI) substrate, etc.) on which circuits are fabricated, while a package may be a carrier on which one or more dies are placed (e.g., for installation in a larger computing system). However, in some aspects, as shown in FIG. 1C, application processor 160 may be located on the same package as die 102.

[0023]

[0027] In general, arbiter 140 may control where data received from the application processor is written. For example, when a machine learning operation is initiated, arbiter 140 may write weight data defining a machine learning model to NVRAM 120. During a machine learning operation, arbiter 140 may write input data received from the application processor (e.g., for processing using the machine learning model and functional unit 130) to DRAM 110. Additionally, arbiter 140 may output results of the machine learning model from DRAM 110 to the application processor (e.g., upon receiving a request to read data at a particular address in DRAM 110).

[0024]

[0028] During inference operations using machine learning models, data may be loaded from DRAM 110 and NVRAM 120 into registers of functional unit 130. To load data into registers, functional unit 130 may request data located at specified addresses from DRAM 110 and NVRAM 120 via bus 150 and, once received, may write the data received from DRAM 110 and NVRAM 120 to the appropriate registers. Various processing engines within functional unit 130, such as a matrix processing engine, a neural processing engine, etc., may process the data received from DRAM 110 and NVRAM 120 and generate results. Functional unit 130 may then write the results to DRAM 110 (e.g., for eventual output to an application processor) and / or retain the results in memory registers for use in subsequent operations using the machine learning model (e.g., when the results are intermediate results used as input by later portions of the machine learning model). In some aspects, such as when NVRAM 120 and functional units 130 are at least partially co-located, the functional units may process data using various process-in-memory techniques.

[0025]

[0029] In general, during operations using machine learning models, functional unit 130 may access (read data from and write data to) NVRAM 120 more frequently than functional unit 130 reads data from DRAM 110. Therefore, to manage memory access latency, processing units 100A, 100B, and 100C may be configured such that NVRAM 120 is located closer to functional unit 130 than DRAM 110. Locating NVRAM 120 closer to functional unit 130 may reduce read and write latency for input data and data generated by machine learning models, because data located in memory closer to functional unit 130 may be accessed more quickly than data located further away from functional unit 130 due to the reduced distance that signaling must travel between functional unit 130 and the more nearby memory.

[0026]

[0030] 1A illustrates a processing unit 100A in which DRAM 110, NVRAM 120, functional units 130, arbiter 140, and bus 150 are fabricated on a single die 102. In processing unit 100A, DRAM 110, NVRAM 120, functional units 130, arbiter 140, and bus 150 may be fabricated using the same process node. While processing unit 100A illustrates a linearly laid out die 102, it should be appreciated that DRAM 110, NVRAM 120, functional units 130, arbiter 140, and bus 150 may be laid out, partially or completely, in a stacked configuration. In a stacked configuration, functional unit 130 may be located at the bottom of die 102, and DRAM 110 and / or NVRAM 120 may be located on top of functional unit 130. Bus 150 may connect through vias or other vertical interconnects, DRAM 110, and NVRAM 120 to functional units 130 and / or arbiter 140, thereby reducing the size of die 102.

[0027]

[0031] 1B illustrates a processing unit 100B in which DRAM 110, NVRAM 120, functional units 130, arbiter 140, and bus 150 are integrated into a single package. As shown, NVRAM 120, functional units 130, and arbiter 140 may be located on a die 102 contained in a package 104, and DRAM 110 may be a separate unit located on package 104. By integrating DRAM 110 and die 102 as separate components on the same package 104, different fabrication techniques may be used to fabricate DRAM 110 and die 102. For example, die 102 (e.g., NVRAM 120, functional unit 130, and arbiter 140) may be fabricated using a first process node (e.g., a “leading edge” or “current” process node, such as 7 nm), while DRAM 110 may be fabricated using a second process node (e.g., on a die fabricated at an older node than the “leading edge” or “current” process node, such as 14 nm or 22 nm).

[0028]

[0032] 1C shows a processing unit 100C in which DRAM 110, NVRAM 120, functional units 130, arbiter 140, bus 150, and application processor 160 are integrated into a single package 104. In this example, DRAM 110, die 102, and application processor 160 may be fabricated separately (e.g., using process nodes appropriate for the power and performance characteristics of each of DRAM 110, die 102, and application processor 160) and included within the same package. Similar to processing unit 100B shown in FIG. 1B, die 102 may include NVRAM 120, functional units 130, and arbiter 140. Although processing unit 100C shows DRAM 110 and die 102 as separate components on the same package 104, it should be appreciated that DRAM 110 may be included within die 102 (e.g., as shown in FIG. 1A), such that DRAM 110, NVRAM 120, functional units 130, and arbiter 140 are fabricated on die 102, and application processor 160 is fabricated on a different die.

[0029]

[0033] 1A-1C illustrate various example configurations of processing units, including DRAM 110, NVRAM 120, functional units 130, arbiter 140, bus 150, and (in some aspects) application processor 160. Various other configurations of DRAM 110, NVRAM 120, functional units 130, arbiter 140, bus 150, and / or application processor 160 may also be possible.

[0030] Exemplary Machine Learning Operations Using Weight Data and Input Data in Different Types of Memory

[0034] FIG. 2 is a flowchart illustrating operations 200 that may be performed by a processing unit (e.g., processing unit 100A shown in FIG. 1A, processing unit 100B shown in FIG. 1B, or processing unit 100C shown in FIG. 1C) to perform machine learning operations based on data stored in different memory components associated with the processing unit.

[0031]

[0035] As shown, operation 200 begins at block 210 by initializing weights of a machine learning model in a first memory. The first memory may be NVRAM, which may be, for example, MRAM, RRAM, or other non-volatile memory that may provide high memory density and high access performance, particularly for read operations. As discussed, because the first memory may provide fast data access and because the weights of the machine learning model may be read frequently during machine learning operations, storing the weights in the first memory may enable low latency in retrieving the weights for machine learning operations.

[0032]

[0036] At block 220, operation 200 proceeds to determine whether a computational operation is required. Generally, a computational operation may be required when processing unit 100 receives a request from an application processor to perform an inference or training operation on input data, intermediate results of the operation, using a machine learning model pending processing by a later portion of the machine learning model. If a computational operation is not required, operation 200 may proceed to block 270, as discussed in more detail below. For example, it may be determined that a computational operation is not required when a model is preloaded into memory before a machine learning operation is invoked on the processing unit.

[0033]

[0037] Otherwise, if a computational operation is required, the operations 200 may proceed from block 220 to block 230, where the data may be stored in a second memory. The second memory may be, for example, a DRAM or other memory component having a high density (e.g., may be capable of storing a large amount of data relative to the amount of physical space the second memory occupies). Generally, the data stored in the second memory may be input data received from an external application processor, intermediate results of an operation using a machine learning model, or other (non-weight) data that may be used during an operation using a machine learning model. The second memory may be located near or co-located with a functional unit that performs an operation using the weight data stored in the first memory (e.g., NVRAM) and the data stored in the second memory to reduce the amount of time required to transfer data from the second memory to the functional unit.

[0034]

[0038] At block 240, the operations 200 proceed to loading the weights and data into registers associated with the functional units. Registers associated with the functional units are generally regions of memory that may be integrated with or otherwise accessible by the functional units for performing various operations in memory.

[0035]

[0039] In some aspects, the weights and data loaded into the registers may be selected based on the asymmetric costs of accessing data from NVRAM and accessing data from DRAM (e.g., the difference in computational cost between accessing data from NVRAM and accessing data from DRAM). In general, more data may be loaded into the registers from DRAM than from NVRAM because accessing data from NVRAM may be cheaper than accessing data from DRAM (e.g., in terms of idle processes wasted waiting to retrieve data from NVRAM).

[0036]

[0040] At block 250, the operations 200 proceed to perform an operation based on the weights and data loaded into the registers. Generally, the operation may generate a result based on various matrix or tensor operations implemented by the functional units. In some aspects, the operation may be performed using in-memory processing techniques, such as when the functional units on which the machine learning operations are performed are at least partially co-located with a first memory (e.g., NVRAM) and / or a second memory (e.g., DRAM).

[0037]

[0041] At block 260, the operations 200 proceed to store the results of performing the in-memory process operations at block 250 in a second memory.

[0038]

[0042] At block 270, the results of the in-memory process operation may be retrieved via normal array accesses in the second memory. To retrieve the results of the in-memory process operation, an arbiter (e.g., arbiter 140 shown in FIGS. 1A-1C) may receive a request from another processor to retrieve data from a specified address in the second memory that the processor may have previously output to an external source (e.g., an application processor). The arbiter may retrieve the data from the specified address in the second memory and output the retrieved data to the requesting processor.

[0039] Exemplary Method for Performing Machine Learning Operations Using Weights and Data Stored in Different Types of Memories

[0043] 3 illustrates example operations 300 that may be performed to perform an operation using a machine learning model with different data associated with the operation stored in different types of memory associated with the processor, according to aspects of the present disclosure. Operations 300 may be performed, for example, by a processor such as processing units 100A-100C shown in FIGS. 1A-1C or other processors in which weights and data may be stored in different types of memory for use during an operation using the machine learning model. As discussed above, weights that may remain static during a machine learning operation may be written once to a first memory component and repeatedly read out for use throughout an operation using the machine learning model, while data on which the machine learning operation is performed may be repeatedly written to and read out from a second memory component during an operation using the machine learning model.

[0040]

[0044] As shown, the operations 300 begin with initializing at least a portion of the weight data of the machine learning model in a first memory component associated with the processor at block 310. As discussed, the first memory component may be a non-volatile random access memory, such as MRAM or RRAM, which may provide high read performance relative to other types of memory.

[0041]

[0045] At block 320, the operations 300 proceed to store the input data in a second memory component coupled to the processor. As discussed, the second memory component may be a dynamic random access memory (DRAM). The DRAM may have a size sufficient to store the input data for use during operations using the machine learning model.

[0042]

[0046] In some aspects, the input data may include data received from a streaming data source. Generally, a streaming data source may include a data source that continuously generates data for processing using the machine learning model, such as a video camera, an audio capture device, etc. In some aspects, the input data may include data received from a non-streaming data source, such as a file or a batch of data for processing.

[0043]

[0047] At block 330, the operations 300 proceed to perform an operation based on at least a portion of the weight data and the input data using the machine learning model via a functional unit associated with the processor.

[0044]

[0048] In some aspects, the operations may be performed using in-memory processing techniques. Generally, when using in-memory processing techniques, memory components and functional units of a processor used to perform operations using a machine learning model may be co-located such that the operations are performed directly on data in the memory components. Some data used during operations using a machine learning model, such as intermediate results, do not need to be swapped between main memory and the processor when the operations are performed using in-memory processing techniques. By reducing the amount of data transferred between main memory and the processor, the techniques described herein may mitigate latency imposed by a "von Neumann bottleneck," in which data transfer operations and data processing operations cannot be performed simultaneously. In some aspects, the in-memory processing techniques may be implemented in one or more static random access memory (SRAM) components of the processor into which data is loaded.

[0045]

[0049] In some aspects, to perform an operation using the machine learning model, at least a portion of the weight data may be loaded from the first memory component into a memory register of the processor, and the functional unit generates a result of the operation using the machine learning model, the weight data loaded from the first memory component into the memory register of the processor, and the input data, and the generated result is stored in the memory register of the processor.

[0046]

[0050] In some aspects, input data from the second memory component may also be loaded into memory registers of the processor. The results of the operation may then be generated by the functional unit using the input data loaded into the memory registers of the processor.

[0047]

[0051] In some aspects, data stored in a memory register of a processor may be selected based on asymmetric costs of accessing weight data from a first memory component and input data from a second memory component. For example, because retrieving weight data from a first memory component may be computationally cheaper than retrieving input data from a second memory component (e.g., causing the processor to perform fewer no-operations or execute instructions that cause the processor to perform no action during a processing cycle while waiting for data to be transferred), the processor register may include a smaller amount of space for weight data from the first memory component than the amount of space reserved for input data from the second memory component. When selecting data stored in a memory register of a processor, the selection may be performed to maximize the amount of time operations are performed based on data in the memory register before retrieving additional data for processing from the second memory component. By maximizing the amount of time operations are performed based on data in the memory register, aspects of the present disclosure may reduce the amount of time a functional component waits for data to be loaded from the second memory component into a register for processing.

[0048]

[0052] In some aspects, the results of the operation may be read from a memory register of the processor, and the results may then be written to a second memory component to make the results available for subsequent operations that use the machine learning model, or to an external application processor that may use the results of the operation using the machine learning model for other tasks.

[0049]

[0053] At block 340, the operations 300 proceed to store the results of the operation using the machine learning model in a second memory component.

[0050]

[0054] In some aspects, the first memory component may be a memory component having a lower storage density and a lower write throughput than the second memory component.

[0051]

[0055] In some aspects, the first memory component, the second memory component, and the functional units of the processor may be integrated onto a single die, in which case the first memory component, the second memory component, and the functional units of the processor may be fabricated using the same process node.

[0052]

[0056] In some aspects, the first memory component, the second memory component, and the functional unit of the processor may be integrated into a single package. By packaging the first memory component, the second memory component, and the functional unit of the processor on a single package, different process nodes may be used to fabricate the first memory component, the second memory component, and the functional unit of the processor.

[0053]

[0057] In some aspects, a first memory component may be fabricated using a first process node, and a second memory component may be fabricated using a second process node. For example, the first memory component may be fabricated using the same process as the functional units of a processor, and the second memory component may be fabricated using a larger process than the process used to fabricate the first memory component. As an illustrative example, the first memory component may be fabricated using a 7-nanometer-class fabrication process, and the second memory component may be fabricated using a larger node size fabrication process, such as a 10-nanometer-class fabrication process, a 14-nanometer-class fabrication process, etc.

[0054]

[0058] For example, a first memory component may be implemented on a first die of a single package and fabricated using a first process node, and a second memory component and functional units of a processor may be implemented on a second die of the single package and fabricated using a second process node. In some aspects, the single package may also include an application processor. The application processor may invoke operations using the first memory component, the second memory component, and the functional units of the processor.

[0055]

[0059] In some aspects, operations 300 may be performed by an arbiter component of a processor. Generally, the arbiter component may arbitrate operations between a first memory component, a second memory component, and a functional unit of the processor. For example, to arbitrate operations, the arbiter component may route data input to the appropriate memory component and initiate processing of the data using a machine learning model based on a command received from an application processor.

[0056] Exemplary Processing System for Performing Machine Learning Operations Using Weights and Data Stored in Different Types of Memories

[0060] FIG. 4 illustrates an exemplary processing system 400 for performing operations using machine learning models with different data related to operations stored in different types of memory associated with a processor, such as those described herein with respect to FIG. 3 .

[0057]

[0061] The processing system 400 includes a central processing unit (CPU) 401, which in some examples may be a multi-core CPU. Instructions executed on the CPU 401 may be loaded, for example, from a program memory associated with the CPU 401 or from a partition in memory 424. As shown, the CPU 401 includes a first memory component 402 and a second memory component 403. The first memory component may be a memory in which weight data for a machine learning model may be stored, and the second memory component 403 may be a memory in which input data processed by the machine learning model may be stored.

[0058]

[0062] The processing system 400 also includes additional processing components adapted to specific functions, such as a graphics processing unit (GPU) 404, a digital signal processor (DSP) 406, a neural processing unit (NPU) 408, a multimedia processing unit 410, and wireless connectivity components 412.

[0059]

[0063] An NPU, such as NPU 408, is generally a dedicated circuit configured to implement control and arithmetic logic for running machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. NPUs may alternatively be referred to as neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligence processing units (IPUs), vision processing units (VPUs), or graph processing units.

[0060]

[0064] NPUs, such as NPU 408, are configured to accelerate the performance of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip, such as a system-on-chip (SoC), while in other examples, they may be part of a dedicated neural network accelerator.

[0061]

[0065] NPUs can be optimized for training or inference, or in some cases can be configured to balance performance between both. For NPUs that can perform both training and inference, the two tasks can still generally be performed independently.

[0062]

[0066] NPUs designed to accelerate training are generally configured to accelerate the optimization of new models, which is a highly computationally intensive operation that involves inputting an existing dataset (often labeled or tagged), iteratively processing the dataset, and then adjusting model parameters such as weights and biases to improve model performance. Optimization based on mispredictions generally involves backpropagating through layers of the model to determine gradients to reduce prediction error.

[0063]

[0067] NPUs designed to accelerate inference are generally configured to operate on complete models. Thus, such NPUs may be configured to input new data and rapidly process the data through already trained models to produce model outputs (e.g., inferences).

[0064]

[0068] In one implementation, the NPU 408 is part of one or more of the CPU 401, the GPU 404, and / or the DSP 406.

[0065]

[0069] Although not shown, it should be appreciated that the GPU 404, DSP 406, and / or NPU 408 may also include a first memory component in which weight data for the machine learning model may be stored, and a second memory component in which input data for the machine learning model may be stored, similar to the first memory component 402 and second memory component 403 shown and discussed above with respect to the CPU 401.

[0066]

[0070] In some examples, the wireless connectivity component 412 may include sub-components for, for example, third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G LTE), fifth generation (5G or NR) connectivity, Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity component 412 is further connected to one or more antennas 414.

[0067]

[0071] The processing system 400 may also include one or more sensor processing units 416 associated with any type of sensor, one or more image signal processors (ISPs) 418 associated with any type of image sensor, and / or a navigation processor 420, which may include satellite-based positioning system components (e.g., GPS or GLONASS), and inertial positioning system components.

[0068]

[0072] Processing system 400 may also include one or more input and / or output devices 422, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, a microphone, etc.

[0069]

[0073] In some examples, one or more of the processors of processing system 400 may be based on the ARM or RISC-V instruction set.

[0070]

[0074] Processing system 400 also includes memory 424, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, memory 424 includes computer-executable components that may be executed by one or more of the aforementioned processors of processing system 400.

[0071]

[0075] In particular, in this example, memory 424 includes a weight data initialization component 424A, an input data storage component 424B, an operation execution component 424C, a result storage component 424D, and a machine learning model component 424E. The depicted components, and other components not depicted, may be configured to implement various aspects of the methods described herein. The weight data initialization component 424A, the input data storage component 424B, the operation execution component 424C, and the result storage component 424D may correspond to means for performing various operations described herein, including means for initializing at least a portion of the weight data, means for storing input data, means for performing operations using the machine learning model, and means for storing results of operations using the machine learning model, respectively.

[0072]

[0076] In general, the processing system 400 and / or its components may be configured to perform the methods described herein.

[0073]

[0077] Notably, in other embodiments, aspects of processing system 400 may be omitted, such as when processing system 400 is a server computer. For example, multimedia processing unit 410, wireless connectivity component 412, sensor processing unit 416, ISP 418, and / or navigation processor 420 may be omitted in other embodiments. Furthermore, aspects of processing system 400, such as training models and using models to generate inferences, such as user authentication predictions, may be distributed.

[0074] Example clauses

[0078] Example implementations are described in the following numbered clauses.

[0075]

[0079] Clause 1: A computer-implemented method, comprising: initializing at least a portion of weight data of a machine learning model in a first memory component associated with a processor; storing input data in a second memory component coupled to the processor; performing an operation using the machine learning model based on at least a portion of the weight data and the input data via a functional unit associated with the processor; and storing results of the operation using the machine learning model in the second memory component.

[0076]

[0080] Clause 2: The method of clause 1, wherein performing an operation using a machine learning model includes loading at least a portion of the weight data from a first memory component into a memory register of the processor, generating a result of the operation using the machine learning model and the input data by a functional unit, and storing the generated result in the memory register of the processor.

[0077]

[0081] Clause 3: The method of clause 2, further comprising loading input data from a second memory component into a memory register of the processor, wherein a result of the operation is generated by the functional unit using the input data loaded into the memory register of the processor.

[0078]

[0082] Clause 4: The method of clause 2 or 3, further comprising reading the result of the operation from a memory register of the processor; and writing the result of the operation read from the memory register of the processor to a second memory component.

[0079]

[0083] Clause 5: The method of clause 4, wherein the data stored in the memory registers of the processor is selected to maximize the amount of time that operations are performed using data in the memory registers of the processor before retrieving additional data for processing from a second memory component.

[0080]

[0084] Clause 6: The method of clause 5, wherein the data stored in the memory register of the processor is further selected based on an asymmetry in access latency between the first memory component and the second memory component.

[0081]

[0085] Clause 7: The method of any of clauses 1 to 6, wherein the input data comprises data received from a streaming data source.

[0082]

[0086] Clause 8: The method of any of clauses 1-7, wherein the first memory component comprises a memory component having a lower storage density and a lower write throughput than the second memory component.

[0083]

[0087] Clause 9: The method of clause 8, wherein the first memory component comprises non-volatile random access memory (NVRAM) and the second memory component comprises dynamic random access memory (DRAM).

[0084]

[0088] Clause 10: The method of any of clauses 1 to 9, wherein performing the operation using the machine learning model includes performing the operation using in-memory processing techniques.

[0085]

[0089] Clause 11: The method of clause 10, wherein the in-memory processing technique includes performing operations in one or more static random access memory (SRAM) components of the processor.

[0086]

[0090] Clause 12: The method of any of clauses 1 to 11, wherein the first memory component, the second memory component, and the processor functional unit are integrated on a single die.

[0087]

[0091] Clause 13: The method of any of clauses 1 to 12, wherein the first memory component, the second memory component, and the processor functional unit are integrated within a single package.

[0088]

[0092] Clause 14: The method of clause 13, wherein the first memory component is implemented on a first die of a single package, and the second memory component and the processor functional unit are implemented on a second die of the single package.

[0089]

[0093] Clause 15: The method of clause 13 or 14, wherein the single package further comprises an application processor that invokes operations using the first memory component, the second memory component, and the processor's functional units.

[0090]

[0094] Clause 16: A method according to any of clauses 1 to 15, wherein the method is implemented by an arbitration unit of the processor configured to arbitrate operations between the first memory component, the second memory component and a functional unit of the processor.

[0091]

[0095] Clause 17: The method of any of clauses 1 to 16, wherein the first memory component comprises a component fabricated using a first process node, and the second memory component comprises a component fabricated using a second process node, and the first process node and the second process node are different process nodes.

[0092]

[0096] Clause 18: A processing system comprising a memory containing computer-executable instructions and one or more processors configured to execute the computer-executable instructions to cause the processing system to perform a method according to any of clauses 1 to 17.

[0093]

[0097] Clause 19: A processing system comprising means for carrying out the method according to any one of clauses 1 to 17.

[0094]

[0098] Clause 20: A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform the method described in any of clauses 1 to 17.

[0095]

[0099] Clause 21: A computer program product embodied on a computer-readable storage medium comprising code for carrying out the method according to any one of clauses 1 to 17.

[0096] Additional Considerations

[0100] The foregoing description is provided to enable any person skilled in the art to practice the various embodiments described herein. The examples described herein are not intended to limit the scope, applicability, or embodiments set forth in the claims. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. For example, changes may be made in the function and arrangement of elements described without departing from the scope of the disclosure. Various examples may omit, substitute, or add various procedures or components as appropriate. For example, described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Also, features described with respect to some examples may be combined in some other examples. For example, an apparatus may be implemented or a method may be practiced using any number of aspects described herein. Additionally, the scope of the disclosure is intended to encompass apparatuses or methods that are practiced using other structure, functions, or structure and functions in addition to or other than the various aspects of the disclosure described herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of a claim.

[0097]

[0101] As used herein, the word "exemplary" means "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects.

[0098]

[0102] As used herein, phrases referring to "at least one of" a list of items refer to any combination of those items, including single members. By way of example, "at least one of a, b, or c" is intended to encompass a, b, c, ab, ac, bc, and abc, as well as any combination having multiples of the same element (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other permutation of a, b, and c).

[0099]

[0103] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, looking up (e.g., searching a table, database, or another data structure), ascertaining, etc. "Determining" may also include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc. "Determining" may also include resolving, selecting, choosing, establishing, etc.

[0100]

[0104] The methods disclosed herein include one or more steps or actions for achieving the method. The steps and / or actions of those methods may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order of specific steps and / or actions and / or the use of those steps and / or actions may be modified without departing from the scope of the claims. Furthermore, the various operations of the methods described above may be performed by any suitable means capable of performing the corresponding functions. These means may include various hardware and / or software component(s) and / or various hardware and / or software module(s), including, but not limited to, circuits, application-specific integrated circuits (ASICs), or processors. Generally, when operations are illustrated in figures, these operations may have corresponding equivalent means-plus-function components that are similarly numbered.

[0101]

[0105] The following claims are not intended to be limited to the embodiments set forth herein but are to be accorded the full scope consistent with the claim language. Within the claims, reference to an element by the singular is intended to mean "one and only one," unless so expressly stated, but rather "one or more." Unless otherwise expressly stated, the term "some" refers to one or more. Claim elements are not to be construed under the provisions of 35 U.S.C. § 112(f) unless the element is expressly recited using the phrase "means for," or, in the case of a method claim, unless the element is recited using the phrase "step of." All structural and functional equivalents of the elements of the various aspects described throughout this disclosure that are known or later become known to those skilled in the art are expressly incorporated herein by reference and intended to be encompassed by the claims. Furthermore, nothing disclosed herein is intended to be made available to the public, regardless of whether such disclosure is expressly recited in the claims.

Claims

1. 1. A computer-implemented method comprising: initializing at least a portion of weight data of the machine learning model in a first memory component associated with the processor; storing the input data in a second memory component coupled to the processor; performing, via a functional unit associated with the processor, an operation using the machine learning model based on the at least a portion of the weight data and the input data; and storing a result of the operation using the machine learning model in the second memory component.

2. performing the operation using the machine learning model, loading at least the portion of the weight data from the first memory component into a memory register of the processor; generating, by the functional unit, the result of the operation using the machine learning model and the input data; and storing the generated results in the memory register of the processor.

3. 3. The method of claim 2, further comprising loading the input data from the second memory component into the memory register of the processor, wherein the result of the operation is generated by the functional unit using the input data loaded into the memory register of the processor.

4. reading the result of the operation from the memory register of the processor; 3. The method of claim 2, further comprising: writing the result of the operation read from the memory register of the processor to the second memory component.

5. 5. The method of claim 4, wherein the data stored in the memory registers of the processor is selected to maximize the amount of time that operations are performed using data in the memory registers of the processor before retrieving additional data for processing from the second memory component.

6. 6. The method of claim 5, wherein the data stored in the memory register of the processor is further selected based on an asymmetry in access latency between the first memory component and the second memory component.

7. The method of claim 1 , wherein the input data comprises data received from a streaming data source.

8. 10. The method of claim 1, wherein the first memory component comprises a memory component having a lower storage density and a lower write throughput than the second memory component.

9. 9. The method of claim 8, wherein the first memory component comprises a non-volatile random access memory (NVRAM) and the second memory component comprises a dynamic random access memory (DRAM).

10. The method of claim 1 , wherein performing operations using the machine learning model comprises performing the operations using in-memory processing techniques.

11. 11. The method of claim 10, wherein the in-memory processing techniques include performing the operations in one or more static random access memory (SRAM) components of the processor.

12. 10. The method of claim 1, wherein the first memory component, the second memory component, and the functional units of the processor are integrated on a single die.

13. The method of claim 1 , wherein the first memory component, the second memory component, and the functional units of the processor are integrated within a single package.

14. 14. The method of claim 13, wherein the first memory component is implemented on a first die of the single package, and the second memory component and the functional units of the processor are implemented on a second die of the single package.

15. 14. The method of claim 13, wherein the single package further comprises an application processor that invokes operations using the first memory component, the second memory component, and the functional units of the processor.

16. 2. The method of claim 1, wherein the method is performed by an arbitration unit of the processor configured to arbitrate operations between the first memory component, the second memory component, and the functional units of the processor.

17. 1. An apparatus comprising: a memory having executable instructions stored therein; a processor; a first memory component associated with the processor; a second memory component coupled to the processor, the processor causing the device to: initializing at least a portion of weight data of a machine learning model in the first memory component associated with the processor; storing input data in the second memory component coupled to the processor; performing, via a functional unit associated with the processor, an operation using the machine learning model based on the at least a portion of the weight data and the input data; 10. An apparatus configured to execute the executable instructions to cause a result of the operation using the machine learning model to be stored in the second memory component.

18. To perform the operation using the machine learning model, the processor causes the device to: causing at least said portion of said weight data to be loaded from said first memory component into a memory register of said processor; causing the functional unit to generate the result of the operation using the machine learning model and the input data; 18. The apparatus of claim 17, configured to store the generated results in the memory register of the processor.

19. 20. The apparatus of claim 18, wherein the processor is further configured to cause the device to load the input data from the second memory component into the memory register of the processor, and wherein the functional unit is configured to generate the result of the operation using the input data loaded into the memory register of the processor.

20. the processor: reading the result of the operation from the memory register of the processor; 20. The apparatus of claim 18, further configured to write the result of the operation read from the memory register of the processor to the second memory component.

21. 21. The apparatus of claim 20, wherein the data stored in the memory registers of the processor is selected to maximize the amount of time that operations are performed using data in the memory registers of the processor before retrieving additional data for processing from the second memory component.

22. 20. The apparatus of claim 17, wherein the first memory component comprises a memory component having a lower storage density and a lower write throughput than the second memory component.

23. 20. The apparatus of claim 17, wherein to perform operations using the machine learning model, the processor is configured to perform the operations using in-memory processing techniques.

24. 20. The apparatus of claim 17, wherein the first memory component, the second memory component, and the functional units of the processor are integrated on a single die.

25. 20. The apparatus of claim 17, wherein the first memory component, the second memory component, and the functional units of the processor are integrated within a single package.

26. 26. The apparatus of claim 25, wherein the single package further comprises an application processor configured to invoke operations using the first memory component, the second memory component, and the functional units of the processor.

27. 20. The apparatus of claim 17, wherein the first memory component comprises a component fabricated using a first process node, and the second memory component comprises a component fabricated using a second process node, the first process node and the second process node being different process nodes.

28. 20. The apparatus of claim 17, wherein the processor comprises an arbitration unit configured to arbitrate operations between the first memory component, the second memory component, and the functional units of the processor.

29. 1. An apparatus comprising: means for initializing at least a portion of weight data of the machine learning model in a first memory component associated with the processor; means for storing input data in a second memory component coupled to said processor; means for performing, via a functional unit associated with the processor, an operation using the machine learning model based on the at least a portion of the weight data and the input data; means for storing a result of the operation using the machine learning model in the second memory component.

30. 1. A non-transitory computer-readable medium containing computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to: initializing at least a portion of weight data of the machine learning model in a first memory component associated with the processor; storing the input data in a second memory component coupled to the processor; performing, via a functional unit associated with the processor, an operation using the machine learning model based on the at least a portion of the weight data and the input data; and storing results of the operation using the machine learning model in the second memory component.

Images