Storage management for arithmetical operations in computing system with heterogeneous memory architecture

Abstract

Certain aspects of the present disclosure provide techniques and apparatus for performing mathematical operations on a processor. The method generally includes initializing, in a first memory component associated with a processor, at least a portion of weight data for a machine learning model. The input data is stored in a second memory component coupled with the processor. The operation using the machine learning model is performed via a functional unit associated with the processor based on at least a portion of the weight data and the input data. The result of the operation using the machine learning model is stored in the second memory component.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. patent application Ser. No. 17 / 934,178, filed on Sept. 21, 2022, which is hereby incorporated by reference herein. Background Art

[0002] Aspects of the present disclosure relate to memory management during execution of mathematical operations on computing devices having heterogeneous memory architectures.

[0003] The von Neumann architecture implemented by many computing systems usually separates the processor and the memory so that in order to perform operations, the processor reads data from the memory and writes data to the memory via a bus. The memory of the von Neumann architecture computing system can be divided into different parts with different performance characteristics and different distances from the processor. For example, the memory can be organized in a hierarchy in which small, fast memory components can have higher performance and are closer to the processor than larger, slower memory components. For example, high-performance memory can be used as a cache for temporarily storing data, and can be located near or co-located with a processor that performs various operations with respect to the data. An off-processor memory component can be used to temporarily store other data that the processor is not currently using but may have recently been used or is expected to use in the near future. This off-processor memory component may be larger than a cache, but may have a longer latency (e.g., longer read / write time) relative to a cache. Still other memory components may be included in a computing system that are at a greater distance from the processor and have smaller performance characteristics, such as solid-state drives or hard disk drives in which data is persistently stored, network storage devices from which the processor can retrieve data via a network connection for processing, and the like.

[0004] Since the cost of high-performance memory that can be used to store data is close to the cost of the processor that performs operations on such data, a computing system can be designed to have a limited amount of high-performance memory and a large amount of lower-performance memory. In many cases, the amount of high-performance memory included in the computing system may be significantly less than the amount of memory required to perform various mathematical operations (such as mathematical operations used to perform various machine learning tasks) on the computing device. Therefore, in order to perform various mathematical operations on the computing device, the computing device may need to exchange data between high-performance memory and lower-performance memory, which may use power each time data is exchanged. This power usage may be cumulative and therefore has a significant impact on the energy consumption, heat generation (and corresponding cooling requirements), battery life, and other properties of the computing device.

[0005] Therefore, what is needed is an improved technique for memory management in computing systems that perform mathematical operations. Summary of the invention

[0006] Certain aspects provide a computer-implemented method for performing operations using a machine learning model on a processor. The method generally includes initializing at least a portion of weight data for the machine learning model in a first memory component associated with the processor. Input data is stored in a second memory component coupled to the processor. Operations using the machine learning model are performed based on at least a portion of the weight data and the input data via a functional unit associated with the processor. Results of the operations using the machine learning model are stored in the second memory component.

[0007] Other aspects provide a processing system configured to perform the above-mentioned method and those methods described herein; a non-transitory computer-readable medium comprising instructions, which, when executed by one or more processors of the processing system, cause the processor system to perform the above-mentioned method and those methods described herein; a computer program product embodied on a computer-readable storage medium, which includes code for performing the above-mentioned method and those methods further described herein; and a processing system, which includes a unit for performing the above-mentioned method and those methods further described herein.

[0008] The following description and the associated drawings set forth in detail certain illustrative features of the one or more embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The drawings depict certain aspects of one or more embodiments and therefore are not to be considered limiting of the scope of the disclosure.

[0010] Figure 1A-Figure 1CAn example processor is shown in which different data relevant to machine learning operations is stored in different types of memory associated with the processor in accordance with aspects of the present disclosure.

[0011] Figure 2 is a flow diagram illustrating operations of performing operations using a machine learning model based on weight data loaded in non-volatile random access memory and other data in dynamic random access memory in accordance with aspects of the present disclosure.

[0012] Figure 3 Example operations for using a machine learning model having different data relevant to the operations stored in different types of memory associated with a processor are shown in accordance with aspects of the present disclosure.

[0013] Figure 4 An example implementation of a processing system in accordance with aspects of the present disclosure is shown in which operations using a machine learning model may be performed with different data stored in different types of memory.

[0014] To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements and features of one embodiment may be beneficially incorporated in other embodiments without further recitation. DETAILED DESCRIPTION

[0015] Aspects of the present disclosure provide techniques for storing data used in performing operations using a machine learning model across different types of memory components associated with a processor.

[0016] Computing systems, such as those implementing the von Neumann architecture, typically organize memory by performance and proximity to the processor that performs operations using the memory. Typically, a small amount of high-performance memory may be located near or co-located with the processor, and a large amount of slower memory may be located at increasing distances from the processor. For example, a processor cache may be faster and located closer to the processor than main system memory, which in turn may be faster and located closer to the processor than on-device persistent storage (e.g., a solid-state drive), which in turn may be faster and located closer to the processor than off-device persistent storage (e.g., removable media, networked drives, etc.). Data currently being used by a processor may be stored in one or more registers or in a cache associated with the processor (e.g., co-located on the same package), while data that may have been previously used by the processor or will be used by the processor in the future may be stored in system memory or persistent storage.

[0017] Typically, a cache may be a relatively small memory device implemented as a static random access memory (SRAM) or other high-performance memory (e.g., high bandwidth memory (HBM), etc.) that may provide fast read and write speeds. In contrast, a persistent memory (which may be located farther from the processor than a cache) may be implemented as a non-volatile random access memory (NVRAM), such as a resistive random access memory (RRAM), a magnetoresistive random access memory (MRAM), or other memory device that may provide for less fast read and write speeds than SRAM or other high-performance memories, but may provide higher capacity at a lower cost. Still further, a dynamic random access memory (DRAM) also provides high capacity at a low cost, but may provide for less fast read and write speeds than SRAM or NVRAM, while consuming additional power due to the need for a memory refresh operation to retain data in the memory. In some cases, different types of memory may be manufactured at different process nodes. Typically, the higher performance memory may be manufactured on a process node used to manufacture a processor on which the higher performance memory is implemented for use as a cache, while the lower performance memory may be manufactured on an older, larger process node. For example, while the higher performance memory may be manufactured on a current process node (such as 7nm), the lower performance memory may be manufactured on a process node such as 14nm, 22nm, or even larger (e.g., a node that is at least two generations older (larger) than the current process node).

[0018] Machine learning models (such as convolutional neural networks, recurrent neural networks, etc.) can be used for a variety of tasks. For example, neural networks can be used for spatial scaling, to adjust the resolution of an image using artificial intelligence techniques (e.g., using super-resolution techniques to increase the resolution of an input image), for temporal interpolation to allow frames to be generated at a higher frame rate (e.g., corresponding to the refresh rate of the display on which the frames are to be displayed), to adjust the appearance of an image (e.g., by image fusion, applying various color effects such as generating high dynamic range (HDR) images, introducing background or foreground blur (also known as "bokeh"), etc.).

[0019] Typically, operations using machine learning models may include multiplication and / or accumulation operations involving weights and input data defined for the machine learning model, which may be retrieved from external data sources or from other parts of the machine learning model for processing. A large amount of data may be used when performing operations using machine learning models, and it may be impractical or impossible to store the data used in the machine learning operations in higher performance storage devices. Therefore, data can be exchanged from higher performance memory to lower performance memory, and vice versa. However, the exchange process may involve both power overhead and performance compensation. For example, multiplication and accumulation functions may involve a certain amount of power per byte (e.g., 2.5 pJ), and the exchange operation of moving data from DRAM to SRAM may consume a large amount of power (e.g., 90% or 2.25 pJ per byte). Therefore, in the context of transferring billions of bytes between DRAM and SRAM during the operation of the machine learning model, these exchange operations may increase processing delays and impose significant power overhead on the processor. Therefore, these exchange operations may have a negative impact on the battery life of the mobile device on which the operation using the machine learning model is performed. Further, because a processor in a computing system implementing a von Neumann architecture may not operate on data until such data is loaded into appropriate memory, and because data processing cannot be performed concurrently with loading data from memory, processing cycles may be wasted in idle mode until appropriate data is loaded into memory. The waste of processing cycles waiting for data to be loaded into appropriate memory may be referred to as a "von Neumann bottleneck."

[0020] In order to improve the performance of operations using machine learning models, various techniques can locate computations near or with memory (e.g., in the same location as memory). For example, compute-in-memory techniques can allow data to be stored in SRAM and allow analog computations to be performed in memory using modified SRAM cells. In another example, function-in-memory or processing-in-memory techniques can locate digital computing capabilities near a memory device (e.g., DRAM, SRAM, MRAM, etc.) where weight data and data to be processed are located. However, in each of these techniques, it may still be necessary to perform many data transfer operations to move data in and out of memory for computation (e.g., when computation is co-located with some but not all of the memory in the computing system).

[0021] During operations using a machine learning model, such as training of the machine learning model or inference operations using the machine learning model, data to be processed and the results of processing such data may be periodically moved into and out of memory (e.g., a cache, a high-performance memory coupled to a processor, a memory in an in-memory computing system, etc.) (e.g., as new inputs are received for processing and the results of processing these inputs are output). However, the weight data defining the machine learning model may be reused repeatedly during operations using the machine learning model. Since the weight data typically remains static during inference and any given training epoch, efficiencies in power consumption, latency, etc. may be obtained by persistently or at least semi-persistently storing the weight data in memory, because such weight data does not need to be frequently written and rewritten during operations using the machine learning model.

[0022] Various aspects of the present disclosure provide techniques that allow different types of data to be stored in different memory components associated with a processor to take advantage of the performance characteristics of these memory components and the static or dynamic nature of the data stored in these memory components. Typically, static data (such as weight data) can be written once to a first memory component, while more dynamic data (such as input to a machine learning model and data generated using a machine learning model) can be written to a second memory component coupled to the processor. For example, the second memory component can provide higher write performance than the first memory component so that input data can be written to the second memory component and read from the second memory component as needed. By writing weight data or other static data to the first memory component and input data and data generated using a machine learning model to the second memory component, various aspects of the present disclosure can reduce the amount of power used to swap data into and out of memory during machine learning operations. Therefore, these techniques can generally improve power efficiency, and particularly battery life on mobile devices on which machine learning operations are performed, reduce memory latency during machine learning operations, reduce heat generation, and thus generally improve the performance of devices on which machine learning operations are performed. Example Compute Device Architecture

[0023] Figure 1A-Figure 1C An example processing unit is shown in which different data relevant to machine learning operations is stored in different types of memory associated with the processor.

[0024] Typically, processing units 100A-100C include DRAM 110, NVRAM 120, functional units 130, and arbiter 140 connected via bus 150. Figure 1A-Figure 1CIn each of the figures, the layout of DRAM 110, NVRAM 120, functional unit 130, and arbiter 140 is for illustration purposes only. It should be noted that the layout of these components may vary, and these components may be combined with Figure 1A-Figure 1C The ones shown in are closer in position.

[0025] DRAM 110 generally provides a location where input data and data generated using a machine learning model can be stored for processing by functional unit 130, while NVRAM 120 provides a location where weight data for a machine learning model can be stored. Generally, DRAM 110 can provide higher performance than NVRAM 120. For example, the latency when writing data to and / or reading data from DRAM 110 can be lower than the latency when writing data to and reading data from NVRAM 120. Since weight data for a machine learning model may have a high reuse rate during operation using the machine learning model, the weight data may be written once and read many times from NVRAM 120. In contrast, since input data and data generated using a machine learning model are written to and read from memory many times by functional unit 130 during operation using the machine learning model, input data and data generated using a machine learning model may be written and read many times from DRAM 110.

[0026] During the machine learning operation, data used during the operation using the machine learning model may be received at the arbitrator 140 from the application processor, which may be the processor that invokes the machine learning operation on the processing unit 100A-100C. In some aspects, as Figure 1A and Figure 1B As shown, the application processor may be an external application processor remote from die 102 or a package including die 102. Generally, die 102 may be a monolithic semiconductor substrate (e.g., a silicon substrate, a silicon germanium substrate, a silicon-on-insulator (SoI) substrate, etc.) on which circuits are fabricated, and the package may be a carrier on which one or more dies are placed (e.g., for installation into a larger computing system). However, in some aspects, such as Figure 1C As shown, application processor 160 may be located on the same package as die 102 .

[0027] In general, arbitrator 140 may control where data received from an application processor is written. For example, when a machine learning operation is initiated, arbitrator 140 may write weight data defining a machine learning model to NVRAM 120. During a machine learning operation, arbitrator 140 may write input data received from an application processor (e.g., for processing using a machine learning model and functional unit 130) to DRAM 110. Further, arbitrator 140 may output the results of the machine learning model from DRAM 110 to the application processor (e.g., upon receiving a request to read data at a specific address in DRAM 110).

[0028] During inference operations using a machine learning model, data may be loaded from DRAM 110 and NVRAM 120 into registers of functional units 130. To load data into registers, functional units 130 may request data located at a specified address from DRAM 110 and NVRAM 120 via bus 150, and upon receipt, may write the received data from DRAM 110 and NVRAM 120 into appropriate registers. Various processing engines (such as a matrix processing engine, a neural processing engine, etc.) in functional units 130 may process the data received from DRAM 110 and NVRAM 120 and generate results. Functional units 130 may then write the results to DRAM 110 (e.g., for eventual output to an application processor) and / or retain the results in storage registers for use in subsequent operations using the machine learning model (e.g., when the results are intermediate results used as input by subsequent parts of the machine learning model). In some aspects, such as when NVRAM 120 and functional units 130 are at least partially co-located, the functional units may process data using various process-in-memory techniques.

[0029] Typically, during operation using a machine learning model, functional unit 130 may access NVRAM 120 (read data from and write data to NVRAM 120) more frequently than functional unit 130 reads data from DRAM 110. Therefore, to manage memory access latency, processing units 100A, 100B, and 100C may be configured such that NVRAM 120 is located closer to functional unit 130 than DRAM 110. Positioning NVRAM 120 closer to functional unit 130 may reduce read and write latency for input data and data generated by the machine learning model because data in memory located closer to functional unit 130 may be accessed faster than data located farther from functional unit 130 due to the shortened distance over which signaling travels between functional unit 130 and the closer located memory.

[0030] Figure 1A Processing unit 100A is shown in which DRAM 110, NVRAM 120, functional unit 130, arbiter 140, and bus 150 are manufactured on a single die 102. In processing unit 100A, DRAM 110, NVRAM 120, functional unit 130, arbiter 140, and bus 150 may be manufactured using the same process node. Although processing unit 100A shows die 102 arranged linearly, it should be appreciated that DRAM 110, NVRAM 120, functional unit 130, arbiter 140, and bus 150 may be arranged partially or entirely in a stacked arrangement. In a stacked arrangement, functional unit 130 may be positioned at the bottom of die 102, and DRAM 110 and / or NVRAM 120 may be positioned at the top of functional unit 130. Bus 150 may connect DRAM 110 and NVRAM 120 to functional units 130 and / or arbiter 140 through vias or other vertical interconnects, which may reduce the size of die 102 .

[0031] Figure 1BA processing unit 100B is shown in which a DRAM 110, an NVRAM 120, a functional unit 130, an arbiter 140, and a bus 150 are integrated into a single package. As shown, the NVRAM 120, the functional unit 130, and the arbiter 140 may be located on a die 102 included in a package 104, and the DRAM 110 may be a separate unit located on the package 104. By integrating the DRAM 110 and the die 102 as separate components on the same package 104, the DRAM 110 and the die 102 may be manufactured using different manufacturing technologies. For example, the die 102 (e.g., the NVRAM 120, the functional unit 130, and the arbiter 140) may be manufactured using a first process node (e.g., a "leading-edge" or "current" process node, such as 7nm), while the DRAM 110 may be manufactured using a second process node (e.g., on a die manufactured at a node earlier than the "leading-edge" or "current" process node, such as 14nm or 22nm).

[0032] Figure 1C A processing unit 100C is shown in which DRAM 110, NVRAM 120, functional unit 130, arbiter 140, bus 150, and application processor 160 are integrated into a single package 104. In this example, DRAM 110, die 102, and application processor 160 may be manufactured separately (e.g., using process nodes suitable for the power and performance characteristics of each of DRAM 110, die 102, and application processor 160) and included in the same package. Figure 1B Similar to processing unit 100B shown, die 102 may include NVRAM 120, functional units 130, and arbiter 140. Although processing unit 100C shows DRAM 110 and die 102 as separate components on the same package 104, it should be appreciated that DRAM 110 may be included in die 102 (e.g., as shown in FIG. Figure 1A ), such that DRAM 110, NVRAM 120, functional unit 130, and arbiter 140 are fabricated on die 102, and application processor 160 is fabricated on a different die.

[0033] It should be recognized that Figure 1A-Figure 1C Various example arrangements of processing units are shown including DRAM 110, NVRAM 120, functional units 130, arbiter 140, bus 150, and (in some aspects) application processor 160. Various other arrangements of DRAM 110, NVRAM 120, functional units 130, arbiter 140, bus 150, and / or application processor 160 are also possible. Example machine learning operations using weight data and input data in different types of memory

[0034] Figure 2 is to show that the processing unit (for example, Figure 1A The processing unit 100A shown in Figure 1B The processing unit 100B shown in Figure 1C Flowchart of operation 200 of a processing unit 100C) shown in FIG. 1 to perform machine learning operations based on data stored in different memory components associated with the processing unit.

[0035] As shown, operation 200 begins at box 210 by initializing weights for a machine learning model in a first memory. The first memory may be NVRAM, which may be, for example, MRAM, RRAM, or other non-volatile memory, which may provide high memory density and high access performance, particularly with respect to read operations. As discussed, since the first memory may provide high-speed data access, and since the weights for the machine learning model may be frequently read during machine learning operations, storing the weights in the first memory may allow for low latency in retrieving the weights for machine learning operations.

[0036] At block 220, operation 200 continues with a determination as to whether a computing operation is required. Typically, a computing operation may be required when a request is received from an application processor for processing unit 100 to perform an inference or training operation on input data, an intermediate result of an operation using a machine learning model to be processed by a subsequent portion of the machine learning model. If a computing operation is not required, operation 200 may continue to block 270, as discussed in further detail below. For example, when the model is preloaded into memory prior to invoking a machine learning operation on the processing unit, it may be determined that a computing operation is not required.

[0037] Otherwise, if a computational operation is required, operation 200 may proceed from box 220 to box 230 to store the data in a second memory. The second memory may be, for example, a DRAM or other memory component having a high density (e.g., a large amount of data may be stored for the amount of physical space occupied by the second memory). Typically, the data stored in the second memory may be input data received from an external application processor, an intermediate result of an operation using a machine learning model, or other (non-weight) data that may be used during an operation using a machine learning model. The second memory may be located near or in the same location as a functional unit that performs operations using weight data stored in a first memory (e.g., NVRAM) and data stored in the second memory to reduce the amount of time required to transfer data from the second memory to the functional unit.

[0038] Operation 200 continues with loading weights and data into registers associated with the functional unit at block 240. Registers associated with a functional unit are typically areas of memory that may be integrated with the functional unit or otherwise accessible by the functional unit to perform various operations in memory.

[0039] In some aspects, the weights and data loaded into the registers may be selected based on the asymmetric cost of accessing data from NVRAM and accessing data from DRAM (e.g., the difference in computational expense between accessing data from NVRAM and accessing data from DRAM). Typically, because accessing data from NVRAM may be cheaper than accessing data from DRAM (e.g., in terms of idle processes wasted while waiting to retrieve data from NVRAM), more data may be loaded into registers from DRAM than from NVRAM.

[0040] At block 250, operations 200 continue with performing operations based on the weights and data loaded into the registers. In general, the operations may generate results based on various matrix or tensor operations implemented by the functional units. In some aspects, the operations may be performed using in-memory processing techniques, such as when the functional units on which the machine learning operations are performed are at least partially co-located with the first memory (e.g., NVRAM) and / or the second memory (e.g., DRAM).

[0041] At block 260, the operation 200 continues by storing the results of the in-memory processing operation performed at block 250 in a second memory.

[0042] At block 270, the result of the in-memory processing operation may be retrieved via a conventional array access at the second memory. To retrieve the result of the in-memory processing operation, an arbitrator (e.g., Figure 1A-Figure 1C The arbitrator 140 shown in FIG. 1 may receive a request from another processor to retrieve data from a specified address in the second memory that the processor may have previously output to an external source (e.g., an application processor). The arbitrator may retrieve the data from the specified address in the second memory and output the retrieved data to the requesting processor. Example methods for performing machine learning operations using weights and data stored in different types of memory

[0043] Figure 3 Example operations 300 are shown that can be implemented for performing operations using a machine learning model when different data related to the operations are stored in different types of memory associated with a processor in accordance with various aspects of the present disclosure. Operations 300 can be performed, for example, by a processor such as Figure 1A-Figure 1CThe processors of the processing units 100A-100C shown, or other processors in which weights and data may be stored in different types of memory for use during operations using the machine learning model, are executed. As discussed above, weights (which may remain static during machine learning operations) may be written once to a first memory component and repeatedly read for use during operations using the machine learning model, while data on which machine learning operations are performed may be repeatedly written to and read from a second memory component during operations using the machine learning model.

[0044] As shown, operations 300 begin by initializing at least a portion of weight data for a machine learning model in a first memory component associated with a processor at block 310. As discussed, the first memory component may be a non-volatile random access memory, such as MRAM or RRAM, which may provide high read performance relative to other types of memory.

[0045] At block 320, operation 300 continues by storing the input data in a second memory component coupled to the processor. As discussed, the second memory component may be a dynamic random access memory. The DRAM may be of sufficient size to store the input data for use during operation using the machine learning model.

[0046] In some aspects, the input data may include data received from a streaming data source. Typically, a streaming data source may include a data source that continuously generates data for processing using a machine learning model, such as a camera, an audio capture device, etc. In some aspects, the input data may include data received from a non-streaming data source, such as a file or a batch of data for processing.

[0047] At block 330 , the operations 300 continue by performing operations using the machine learning model based on at least a portion of the weight data and the input data via a functional unit associated with the processor.

[0048] In some aspects, the operations may be performed using in-memory processing techniques. Typically, when using in-memory processing techniques, the memory components and functional units of the processor used to perform operations using the machine learning model may be in the same location, so that the operations are performed directly on the data in the memory components. When performing operations using in-memory processing techniques, some data (such as intermediate results) used during the operation using the machine learning model do not need to be exchanged between the main memory and the processor. By reducing the amount of data transferred between the main memory and the processor, the techniques described herein can alleviate the delay caused by the "von Neumann bottleneck" in which data transfer and data processing operations cannot be performed simultaneously. In some aspects, the in-memory processing techniques may be performed in one or more static random access memory (SRAM) components of the processor in which the data is loaded.

[0049] In some aspects, to perform an operation using the machine learning model, at least a portion of the weight data may be loaded from the first memory component into a storage register of the processor. The functional unit generates a result of the operation using the machine learning model, the weight data read from the first memory component into the storage register of the processor, and the input data, and the generated result is stored in the storage register of the processor.

[0050] In some aspects, the input data from the second memory component may also be loaded into the storage registers of the processor.The result of the operation may then be generated by the functional unit using the input data loaded into the storage registers of the processor.

[0051] In some aspects, the data stored in the storage registers of the processor can be selected based on the asymmetric cost of accessing the weight data from the first memory component and the input data from the second memory component. For example, since retrieving the weight data from the first memory component may be computationally cheaper than retrieving the input data from the second memory component (e.g., causing the processor to perform fewer no-ops, or instructions that cause the processor to not perform actions during a processing cycle while waiting for data to be transmitted), the processor registers may include less space for the weight data from the first memory component than the amount of space reserved for the input data from the second memory component. When selecting the data stored in the storage registers of the processor, the selection can be performed to maximize the amount of time during which operations are performed based on the data in the storage registers before retrieving additional data for processing from the second memory component. By maximizing the amount of time during which operations are performed based on the data in the storage registers, various aspects of the present disclosure can reduce the amount of time during which functional components wait for data to be loaded from the second memory component to the registers for processing.

[0052] In some aspects, the result of the operation can be read from a storage register of the processor. The result can then be written to a second memory component to make the result available for subsequent operations using the machine learning model, or to an external application processor, which can use the result of the operation using the machine learning model to perform other tasks.

[0053] At block 340 , the operations 300 continue by storing the results of the operations using the machine learning model in a second memory component.

[0054] In some aspects, the first memory component may be a memory component having a lower storage density and a lower write throughput than the second memory component.

[0055] In some aspects, the first memory component, the second memory component, and the functional unit of the processor can be integrated on a single die. In such a case, the first memory component, the second memory component, and the functional unit of the processor can be manufactured using the same process node.

[0056] In some aspects, the first memory component, the second memory component, and the functional unit of the processor can be integrated into a single package. By packaging the first memory component, the second memory component, and the functional unit of the processor on a single package, different process nodes can be used to manufacture the first memory component, the second memory component, and the functional unit of the processor.

[0057] In some aspects, the first memory component may be manufactured using a first process node, and the second memory component may be manufactured using a second process node. For example, the first memory component may be manufactured using the same process as a functional unit of a processor, and the second memory component may be manufactured using a larger process than the process used to manufacture the first memory component. As an illustrative example, the first memory component may be manufactured using a 7 nanometer class manufacturing process, and the second memory component may be manufactured using a manufacturing process with a larger node size, such as a 10 nanometer class manufacturing process, a 14 nanometer class manufacturing process, etc.

[0058] For example, a first memory component may be implemented on a first die in a single package and may be manufactured using a first process node, and a second memory component and a functional unit of a processor may be implemented on a second die in a single package and may be manufactured using a second process node. In some aspects, the single package may also include an application processor. The application processor may invoke operations using the first memory component, the second memory component, and the functional unit of the processor.

[0059] In some aspects, operation 300 may be performed by an arbitrator component of a processor. Typically, the arbitrator component may arbitrate operations between a first memory component, a second memory component, and a functional unit of the processor. For example, to arbitrate operations, the arbitrator component may route data input to an appropriate memory component and may initiate processing of data using a machine learning model based on a command received from an application processor. Example processing system for performing machine learning operations using weights and data stored in different types of memory

[0060] Figure 4 Depicting an example processing system 400 for performing operations using a machine learning model when different data related to the operations are stored in different types of memory associated with a processor, such as described herein, for example, with respect to Figure 3 Described.

[0061] The processing system 400 includes a central processing unit (CPU) 401, which in some examples may be a multi-core CPU. Instructions executed at the CPU 401 may be loaded, for example, from a program memory associated with the CPU 401, or may be loaded from a partition in the memory 424. As shown, the CPU 401 includes a first memory component 402 and a second memory component 403. The first memory component may be a memory in which weight data for a machine learning model may be stored, and the second memory component 403 may be a memory in which input data processed by the machine learning model may be stored.

[0062] The processing system 400 also includes additional processing components customized for specific functions, such as a graphics processing unit (GPU) 404, a digital signal processor (DSP) 406, a neural processing unit (NPU) 408, a multimedia processing unit 410, a multimedia processing unit 410, and a wireless connectivity component 412.

[0063] An NPU, such as NPU 408, is typically a dedicated circuit configured to implement control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. An NPU may also sometimes be referred to as a neural signal processor (NSP), a tensor processing unit (TPU), a neural network processor (NNP), an intelligence processing unit (IPU), a vision processing unit (VPU), or a graphics processing unit.

[0064] NPUs, such as NPU 408, are configured to accelerate the performance of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single die, such as a system on a chip (SoC), while in other examples, they may be part of a dedicated neural network accelerator.

[0065] The NPU may be optimized for training or inference, or in some cases configured to balance performance between the two. For an NPU that can perform both training and inference, the two tasks can still typically be performed independently.

[0066] NPUs designed to accelerate training are typically configured to accelerate the optimization of new models, which is a highly compute-intensive operation that involves inputting an existing dataset (usually marked or labeled), iterating over the dataset, and then adjusting model parameters such as weights and biases to improve model performance. Typically, optimization based on error predictions involves passing back through the layers of the model and determining the gradient to reduce the prediction error.

[0067] NPUs designed to accelerate inference are typically configured to operate on complete models. Thus, such NPUs can be configured to input new data slices and quickly process them through an already trained model to generate model output (e.g., inference).

[0068] In one implementation, the NPU 408 is part of one or more of the CPU 401 , the GPU 404 , and / or the DSP 406 .

[0069] Although not shown, it should be appreciated that the GPU 404, DSP 406 and / or NPU 408 may also include a first memory component (in which weight data for the machine learning model may be stored) and a second memory component (in which input data for the machine learning model may be stored), similar to the first memory component 402 and the second memory component 403 shown with respect to the CPU 401 and discussed above.

[0070] In some examples, wireless connectivity component 412 may include subcomponents, such as for third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G LTE), fifth generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. Wireless connectivity component 412 is also connected to one or more antennas 414.

[0071] The processing system 400 may also include one or more sensor processing units 416 associated with any manner of sensors, one or more image signal processors (ISPs) 418 associated with any manner of image sensors, and / or a navigation processor 420, which may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0072] The processing system 400 may also include one or more input and / or output devices 422, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, speakers, microphones, and the like.

[0073] In some examples, one or more of the processors of processing system 400 may be based on the ARM or RISC-V instruction set.

[0074] Processing system 400 also includes memory 424 , which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, memory 424 includes computer-executable components that can be executed by one or more of the aforementioned processors of processing system 400 .

[0075] In particular, in this example, memory 424 includes weight data initialization component 424A, input data storage component 424B, operation execution component 424C, result storage component 424D and machine learning model component 424E. The components depicted, as well as other components not described, can be configured to perform various aspects of the method described herein. Weight data initialization component 424A, input data storage component 424B, operation execution component 424C and result storage component 424D can correspond to the unit for performing various operations described herein, including a unit for initializing at least a portion of weight data, a unit for storing input data, a unit for performing operations using a machine learning model, and a unit for storing the result of the operation using the machine learning model.

[0076] Generally, processing system 400 and / or components thereof may be configured to perform the methods described herein.

[0077] Note that in other embodiments, various aspects of the processing system 400 may be omitted, such as where the processing system 400 is a server computer, etc. For example, the multimedia processing unit 410, the wireless connectivity component 412, the sensor processing unit 416, the ISP 418, and / or the navigation processor 420 may be omitted in other embodiments. Further, various aspects of the processing system 400 may be distributed, such as training a model and using the model to generate inferences, such as user-verified predictions. Sample Clauses

[0078] Examples of implementations are described in the following numbered clauses:

[0079] Item 1: A computer-implemented method comprising: initializing at least a portion of weight data for a machine learning model in a first memory component associated with a processor; storing input data in a second memory component coupled to the processor; performing an operation using the machine learning model based on at least a portion of the weight data and the input data via a functional unit associated with the processor; and storing a result of the operation using the machine learning model in the second memory component.

[0080] Clause 2: A method according to clause 1, wherein performing an operation using a machine learning model comprises: loading at least a portion of weight data from a first memory component into a storage register of a processor; generating a result of the operation by a functional unit using the machine learning model and input data; and storing the generated result in a storage register of the processor.

[0081] Clause 3: The method of clause 2, further comprising loading input data from the second memory component into a storage register of the processor, wherein the result of the operation is generated by the functional unit using the input data loaded into the storage register of the processor.

[0082] Clause 4: The method of clause 2 or clause 3, further comprising: reading a result of the operation from a storage register of the processor; and writing the result of the operation read from the storage register of the processor to the second memory component.

[0083] Clause 5: The method of clause 4, wherein the data stored in the storage register of the processor is selected to maximize an amount of time during which operations are performed using the data in the storage register of the processor before additional data is retrieved from the second memory component for processing.

[0084] Clause 6: The method of clause 5, wherein the data stored in the storage register of the processor is further selected based on an asymmetry in access latency between the first memory component and the second memory component.

[0085] Clause 7: A method according to any of clauses 1 to 6, wherein the input data comprises data received from a streaming data source.

[0086] Clause 8: The method of any of clauses 1 to 7, wherein the first memory component comprises a memory component having a lower storage density and a lower write throughput than the second memory component.

[0087] Clause 9: The method of clause 8, wherein the first memory component comprises non-volatile random access memory (NVRAM) and the second memory component comprises dynamic random access memory (DRAM).

[0088] Clause 10: A method according to any of clauses 1 to 9, wherein performing the operation using the machine learning model includes performing the operation using in-memory processing techniques.

[0089] Clause 11: The method of clause 10, wherein the in-memory processing technique comprises performing operations in one or more static random access memory (SRAM) components of the processor.

[0090] Clause 12: The method of any of clauses 1 to 11, wherein the first memory component, the second memory component, and the functional unit of the processor are integrated on a single die.

[0091] Clause 13: The method of any of clauses 1 to 12, wherein the first memory component, the second memory component, and the functional unit of the processor are integrated into a single package.

[0092] Clause 14: The method of clause 13, wherein the first memory component is implemented in a single packaged first die, and wherein the second memory component and the functional unit of the processor are implemented in a single packaged second die.

[0093] Clause 15: The method of clause 13 or clause 14, wherein the single package further comprises an application processor that invokes operations using the first memory component, the second memory component, and the functional unit of the processor.

[0094] Clause 16: A method according to any of clauses 1 to 15, wherein the method is performed by an arbitration unit of a processor, the arbitration unit being configured to arbitrate operations between the first memory component, the second memory component and the functional unit of the processor.

[0095] Clause 17: A method according to any of clauses 1 to 16, wherein the first memory component includes a component manufactured using a first process node, wherein the second memory component includes a component manufactured using a second process node, and wherein the first process node and the second process node are different process nodes.

[0096] Clause 18: A processing system comprising: a memory including computer executable instructions, and one or more processors configured to execute the computer executable instructions and cause the processing system to perform the method according to any of clauses 1 to 17.

[0097] Clause 19: A processing system comprising: means for performing the method according to any of clauses 1 to 17.

[0098] Clause 20: A non-transitory computer readable medium comprising computer executable instructions which, when executed by one or more processors of a processing system, cause the processing system to perform a method according to any of clauses 1 to 17.

[0099] Clause 21: A computer program product embodied on a computer readable storage medium, comprising code for performing a method according to any of clauses 1 to 17. Additional Notes

[0100] The foregoing description is provided to enable any person skilled in the art to practice the various embodiments described herein. The examples discussed herein do not limit the scope, applicability or embodiments set forth in the claims. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. For example, without departing from the scope of the present disclosure, the functions and arrangements of the elements discussed may be changed. Each example may omit, replace or add various processes or components as appropriate. For example, the described method may be performed in an order different from that described, and various steps may be added, omitted or combined. In addition, the features described with respect to some examples may be combined in some other examples. For example, a device or a method of practice may be implemented using any number of aspects set forth herein. In addition, the scope of the present disclosure is intended to cover such devices or methods practiced using other structures, functions, or structures and functions other than or different from the various aspects of the disclosure set forth herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of the claims.

[0101] As used herein, the word “exemplary” means “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.

[0102] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single members. As an example, "at least one of a, b, or c" is intended to cover a, b, c, ab, ac, bc, and abc, as well as any combination with multiples of the same elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc or any other ordering of a, b, and c).

[0103] As used herein, the term "determine" encompasses a wide variety of actions. For example, "determine" may include computing, calculating, processing, deriving, investigating, searching (e.g., searching in a table, a database, or another data structure), ascertaining, etc. In addition, "determine" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc. In addition, "determine" may include resolving, selecting, choosing, establishing, etc.

[0104] The method disclosed herein includes one or more steps or actions for implementing the method. Without departing from the scope of the claims, the method steps and / or actions can be interchanged with each other. In other words, unless the specific order of steps or actions is specified, the order and / or purpose of specific steps and / or actions can be modified without departing from the scope of the claims. Further, the various operations of the method described above can be performed by any appropriate unit that can perform the corresponding function. The unit can include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs) or processors. Typically, in the presence of the operations shown in the figure, those operations can have corresponding paired functional module components with similar numbers.

[0105] The attached claims are not intended to be limited to the embodiments shown herein, but are given the full scope consistent with the language of the claims. Within the claims, unless explicitly stated as such, references to singular elements are not intended to mean "one and only one", but "one or more". Unless otherwise specifically stated, the term "some" refers to one or more. No claim element is to be interpreted according to the provisions of 35 U.S.C. § 112 (f), unless the element is explicitly recorded using the phrase "unit for...", or in the case of a method claim, the element is recorded using the phrase "step for...". All structural and functional equivalents of the elements of the various aspects described throughout the present disclosure, which are known or will be known later to those of ordinary skill in the art, are expressly incorporated herein by reference, and are intended to be covered by the claims. In addition, nothing disclosed herein is intended to be dedicated to the public, regardless of whether such disclosure is explicitly recorded in the claims.

Claims

1. A computer-implemented method comprising: Initializing at least a portion of weight data for a machine learning model in a first memory component associated with the processor; storing input data in a second memory component coupled to the processor; performing, via a functional unit associated with the processor, an operation using the machine learning model based on at least the portion of the weight data and the input data; as well as Results of the operations using the machine learning model are stored in the second memory component.

2. The method according to claim 1, wherein: Using the machine learning model to perform the operations includes: loading at least the portion of the weight data from the first memory component into a storage register of the processor; generating, by the functional unit, the result of the operation using the machine learning model and the input data; and The generated result is stored in the storage register of the processor.

3. The method of claim 2, further comprising loading the input data from the second memory component into the storage register of the processor, wherein: The result of the operation is generated by the functional unit using the input data loaded into the storage register of the processor.

4. The method according to claim 2, further comprising: reading the result of the operation from the storage register of the processor; as well as The result of the operation read from the storage register of the processor is written to the second memory component.

5. The method according to claim 4, wherein: The data stored in the storage registers of the processor is selected to maximize an amount of time during which operations are performed using the data in the storage registers of the processor before additional data is retrieved from the second memory component for processing.

6. The method according to claim 5, wherein: The data stored in the storage register of the processor is also selected based on an asymmetry in access latency between the first memory component and the second memory component.

7. The method according to claim 1, wherein: The input data includes data received from a streaming data source.

8. The method according to claim 1, wherein: The first memory component includes a memory component having a lower storage density and a lower write throughput than the second memory component.

9. The method according to claim 8, wherein: The first memory component comprises non-volatile random access memory (NVRAM), and wherein the second memory component comprises dynamic random access memory (DRAM).

10. The method according to claim 1, wherein: Performing operations using the machine learning model includes using in-memory processing techniques to perform the operations.

11. The method according to claim 10, wherein: The in-memory processing technique includes performing the operations in one or more static random access memory (SRAM) components of the processor.

12. The method according to claim 1, wherein: The first memory component, the second memory component, and the functional unit of the processor are integrated on a single die.

13. The method according to claim 1, wherein: The first memory component, the second memory component, and the functional unit of the processor are integrated into a single package.

14. The method according to claim 13, wherein: The first memory component is implemented in a first die of the single package, and wherein the second memory component and the functional unit of the processor are implemented in a second die of the single package.

15. The method according to claim 13, wherein: The single package also includes an application processor that invokes the operations using the first memory component, the second memory component, and the functional unit of the processor.

16. The method according to claim 1, wherein: The method is performed by an arbitration unit of the processor, wherein the arbitration unit is configured to arbitrate operations between the first memory component, the second memory component, and the functional unit of the processor.

17. An apparatus comprising: a memory having executable instructions stored thereon; processor; a first memory component associated with the processor; A second memory component is coupled to the processor, and the processor is configured to execute executable instructions that cause the apparatus to: Initializing at least a portion of weight data for a machine learning model in the first memory component associated with the processor; storing input data in the second memory component coupled to the processor; performing, via a functional unit associated with the processor, an operation using the machine learning model based on at least the portion of the weight data and the input data; as well as Results of the operations using the machine learning model are stored in the second memory component.

18. The device according to claim 17, wherein: In order to perform the operations using the machine learning model, the processor is configured to cause the device to perform the following operations: loading at least the portion of the weight data from the first memory component into a storage register of the processor; generating, by the functional unit, the result of the operation using the machine learning model and the input data; as well as The generated result is stored in the storage register of the processor.

19. The device according to claim 18, wherein: The processor is further configured to cause the apparatus to load the input data from the second memory component into the storage register of the processor, wherein the functional unit is configured to generate the result of the operation using the input data loaded into the storage register of the processor.

20. The device according to claim 18, wherein The processor is further configured to: reading the result of the operation from the storage register of the processor; and The result of the operation read from the storage register of the processor is written to the second memory component.

21. The device according to claim 20, wherein: The data stored in the storage registers of the processor is selected to maximize an amount of time during which operations are performed using the data in the storage registers of the processor before additional data is retrieved from the second memory component for processing.

22. The device according to claim 17, wherein: The first memory component includes a memory component having a lower storage density and a lower write throughput than the second memory component.

23. The device according to claim 17, wherein: In order to perform operations using the machine learning model, the processor is configured to perform the operations using in-memory processing techniques.

24. The device according to claim 17, wherein: The first memory component, the second memory component, and the functional unit of the processor are integrated on a single die.

25. The device according to claim 17, wherein: The first memory component, the second memory component, and the functional unit of the processor are integrated into a single package.

26. The device according to claim 25, wherein The single package also includes an application processor that invokes the operations using the first memory component, the second memory component, and the functional unit of the processor.

27. The device according to claim 17, wherein: The first memory component includes components fabricated using a first process node, wherein the second memory component includes components fabricated using a second process node, and wherein the first process node and the second process node are different process nodes.

28. The device according to claim 17, wherein: The processor includes an arbitration unit configured to arbitrate operations between the first memory component, the second memory component, and the functional unit of the processor.

29. An apparatus comprising: means for initializing at least a portion of weight data for a machine learning model in a first memory component associated with the processor; means for storing input data in a second memory component coupled to said processor; means for performing, via a functional unit associated with the processor, an operation using the machine learning model based on at least the portion of the weight data and the input data; as well as Means for storing results of said operation using said machine learning model in said second memory component.

30. A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform a method comprising: Initializing at least a portion of weight data for a machine learning model in a first memory component associated with the processor; storing input data in a second memory component coupled to the processor; performing, via a functional unit associated with the processor, an operation using the machine learning model based on at least the portion of the weight data and the input data; as well as Results of the operations using the machine learning model are stored in the second memory component.

Images