Optimal Allocation Method and System for Hybrid Memory-Based Data Structures

Abstract

A method and system for optimized placement of data structures in memory in a computing environment is provided. One or more data structures can be dynamically allocated in a hybrid memory according to weights and activations of the one or more data structures in a deep neural network ("DNN"). The hybrid memory includes at least a first memory and a second memory that differ according to write endurance attributes.

Description

[Technical field]

[0001] The present invention relates generally to computing systems, and more particularly to various embodiments for optimal placement of data structures within a hybrid memory-based inference engine / computing platform in a processor-based computing environment. [Background technology]

[0002] In today's society, consumers, business people, educators, and others use various computing network systems more and more frequently in a variety of situations. Computer systems may be found at work, at home, or at school. Computer systems may include data storage systems or disk storage systems to process and store data. In recent years, both software and hardware technologies have made incredible advances. With new technologies, more features are being added and greater convenience is being realized for use with these computing systems. Summary of the Invention

[0003] Various embodiments are provided for optimal placement of data structures in a hybrid memory-based inference engine / computing platform using one or more processors in a computing environment. In one embodiment, by way of example only, a method is provided for optimal placement of data structures in memory in a computing environment, also by a processor. One or more data structures may be dynamically allocated in the hybrid memory according to weights and activations of one or more data structures of a deep neural network ("DNN"). The hybrid memory includes at least a first memory and a second memory that differ according to write attributes. The first memory may be a non-volatile memory ("NVM")-based memory and the second memory may be a CMOS-based memory.

[0004] So that the advantages of the present invention may be readily understood, a more particular description of the invention briefly described above will be rendered by reference to specific embodiments which are illustrated in the accompanying drawings, in which: The invention will be described and explained with additional specificity and detail through the use of the accompanying drawings, with the understanding that such drawings illustrate only typical embodiments of the invention and therefore should not be considered as limiting of the invention, and in which: [Brief description of the drawings]

[0005] [Figure 1] FIG. 2 is a block diagram illustrating an exemplary computing node according to one embodiment of the present invention. [Diagram 2] FIG. 13 is an additional block diagram illustrating an exemplary cloud computing environment according to an embodiment of the present invention. [Diagram 3] FIG. 4 is an additional block diagram illustrating abstraction model layers according to one embodiment of the present invention. [Figure 4A] FIG. 13 is an additional block diagram illustrating a deep neural network architecture in which aspects of the present invention are implemented. [Figure 4B] FIG. 13 is an additional block diagram illustrating a deep neural network architecture in which aspects of the present invention are implemented. [Diagram 5] 1 is an additional block illustrating an optimal arrangement of data structures within a hybrid memory-based inference engine / computing platform in which aspects of the present invention are implemented. [Figure 6] 1A-1C are additional block diagrams illustrating exemplary functional relationships and operations between various aspects of the present invention. [Figure 7] FIG. 11 is a flow chart diagram illustrating an additional exemplary method for optimally placing data structures in a hybrid memory in a computing environment by a processor, in which aspects of the present invention may also be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] The present invention relates generally to the field of artificial intelligence ("AI"), such as machine learning and / or deep learning. "Machine learning" is used loosely to describe the primary function of electronic systems that learn from data. In machine learning and cognitive science, neural networks are a family of statistical learning models inspired by the biological neural networks of animals, particularly the brains. Neural networks depend on a large number of inputs and can be used to infer and approximate generally unknown systems and functions.

[0007] Neural networks use a class of algorithms based on the concept of interconnected "neurons." In a typical neural network, neurons have a given activation function that acts on the input. By determining the appropriate connection weights (a process also called "training"), the neural network achieves efficient recognition of a desired pattern, for example an image or a character. Often, these neurons are grouped into "layers" to make the connections between groups, and for each computation of a value, more explicit.

[0008] Deep learning refers to a class of machine learning algorithms based on learning multiple levels of features or representations of a set of data. Current deep learning methods involve using a cascade of multiple layers of nonlinear processing devices for feature extraction and transformation. Feature extraction refers to the process of taking an initial set of measured data and constructing derived values ​​(or features) to facilitate subsequent learning and generalization steps. In many cases, higher level features are derived from lower level features, resulting in a hierarchical representation of the measured data and the derived features.

[0009] Furthermore, deep learning algorithms are based on distributed representations. Distributed representations operate under the assumption that the observed (or measured) data is the result of the interaction of one or more elements organized in one or more layers. Conceptually, deep learning introduces the additional assumption that the layers of elements that interact to provide the measured data represent levels of abstraction or composition. Under this assumption, multiple layers and layer sizes correspond to different amounts of abstraction.

[0010] Overall, deep learning has led to significant advances in optimizing decision making and extracting business value from unstructured data, e.g., images, audio, video, text, etc. However, while deep learning achieves improved computing performance for a wide variety of data types and tasks, it still faces several challenges, namely data structure placement and memory allocation issues for deep neural networks (DNNs).

[0011] For example, DNNs place significant demands on memory subsystems as model sizes and datasets grow. An increasing proportion of processor chip area and energy in DNN accelerators is spent on memory, which faces challenges with technology scaling. Thus, there is considerable interest in emerging non-volatile memory (NVM) technologies that may improve the efficiency of memory systems while overcoming the challenges of complementary metal-oxide semiconductor ("CMOS") memory. NVM offers several unique attributes, such as near-zero leakage and high density, that are beneficial for memory designs.

[0012] However, there are many different challenges associated with NVM, some of which include, for example, that NVM is limited by 1) high energy requirements, 2) high latency incurred during write operations, 3) limited endurance, or 4) a limited number of write cycles. Therefore, novel solutions are required that can leverage the strengths of NVM while overcoming its weaknesses, thereby achieving an energy-efficient memory system in DNN accelerators.

[0013] Therefore, to overcome this challenge, the present invention provides a novel solution by distinguishing memory traffic related to the model data structures (e.g., weights and biases) of a DNN from memory traffic resulting from activation data structures for each DNN layer during inference. That is, the present invention can distinguish between weight memory traffic and activation memory traffic. In other words, the present invention can distinguish between weight memory data structures and activation memory data structures.

[0014] In an additional aspect, the present invention may dynamically allocate one or more data structures in a hybrid memory according to weights and activations of one or more data structures in a deep neural network ("DNN"). The hybrid memory includes at least a first memory and a second memory that differ according to write attributes, characteristics, or properties, or combinations thereof, such as differences in write energy, write latency, or write endurance, or combinations thereof. In one aspect, a non-volatile memory ("NVM") based memory may be the first memory and a CMOS based memory may be the second memory. In one aspect, the present invention may store (e.g., read) model parameters in (from) a first memory (e.g., a first NVM based memory) and store output activations using a second memory (e.g., a complementary metal oxide semiconductor ("CMOS") based memory), such as a static random access memory ("SRAM"). The memory partitioning achieves significant energy efficiency by exploiting the varying strength of read / write operations in each DNN data structure.

[0015] Thus, the present invention may mitigate the performance (or energy) penalty due to the difference in read and write latency (or energy) and may also improve durability (i.e., extend the life of the memory). In one aspect, as used herein, "durability" may be expressed by how many write operations can write to the memory. Thus, the present invention may be extended to NVM (or any other memory) that may have write / read performance differences or that may have durability differences.

[0016] In an additional aspect, the present invention provides for optimal placement of data structures in memory. One or more data structures may be dynamically allocated among one or more separate and / or separate memories, which may include 1) non-volatile memory ("NVM")-based memory and CMOS-based memory, 2) hybrid memory including NVM and CMOS memory (e.g., hybrid memory including NVM and CMOS memory that is not separate and separate), or 3) one or more memory types, or combinations thereof, that differ in various attributes (e.g., write attributes), such as, for example, write energy, write latency, write endurance, or combinations thereof. In one aspect, the first memory (e.g., NVM) may include, but is not limited to, memristor, magnetoresistive RAM ("MRAM"), resistive RAM ("RRAM"), conductive bridging RAM ("CBRAM"), or phase change memory ("PCM") elements. The second memory (eg, a CMOS memory) may include SRAM or other types of memory with reduced write energy, write latency, write endurance, or a combination thereof.

[0017] In other words, a hybrid memory may be composed of NVM and CMOS that are not separate and separated (e.g., a single memory that includes both NVM and CMOS). Alternatively, a hybrid memory may be composed of NVM and CMOS that are separate and separated (e.g., multiple memories that include NVM and CMOS).

[0018] For example, in one aspect, one or more data structures may be dynamically allocated among one or more types of memory configurations / combinations (e.g., separate and / or separate memories that may include 1) non-volatile memory ("NVM")-based memory and CMOS-based memory, or NVM and CMOS, 2) hybrid memory) according to the weights and activations of one or more data structures in a deep neural network ("DNN"). That is, the weights in a neural network represent the strength of the connections between different neurons. Similarly, the activation represents the output generated by a neuron after going through a process called an "activation function." An activation function may be a function that takes an input and generates some output value based on some threshold. In one aspect, the hybrid memory may include both non-volatile memory ("NVM")-based memory and CMOS-based memory. In another aspect, the hybrid memory may include different types of memory or memory combinations that differ according to the write endurance attributes of the particular memory.

[0019] In another aspect, the present invention may take advantage of the different read / write characteristics of DNN data structures, such as by distinguishing between activations from weights when reading from / writing to memory. Note that weights and activations have different characteristics. Weights during DNN inference are read-intensive, and activations are both read-intensive and write-intensive. To achieve and improve energy efficiency, the number of writes to one or more separate memories and / or separate memories may be minimized. The present invention may also mitigate performance penalties.

[0020] For example, consider a scenario where there is only non-volatile memory ("NVM") based memory. In this case, both weights and activations may be written / read from the NVM based memory. Writes into NVM may incur significant latency (e.g., five times ("5X") more than CMOS based SRAM). Thus, by using the mechanisms of the illustrated embodiments to avoid writes to NVM for dynamic (read / write intensive) activations and redirect such writes to CMOS based SRAM, the write energy, write latency, write difficulty, or a combination thereof is much lower, minimizing the impact on overall system performance.

[0021] Additionally, as used herein, a computing system may include large-scale computing referred to as "cloud computing" in which resources interact and / or are accessed through a communication system such as a computer network. The resources may be software-rendered simulations and / or emulations of computing devices, storage devices, applications, and / or other computer-related devices, and / or services running on one or more computing devices such as servers. For example, multiple servers may communicate and / or share information that may scale across the servers, depending on the amount of processing power, storage space, or other computing resources, or a combination thereof, required to accomplish a requested task. The term "cloud" connotes the cloud-shaped appearance of a diagram of interconnectivity between computing devices, computer networks, or other computer-related devices interacting in such a configuration.

[0022] It should be noted that one or more calculations or computations may be performed using various mathematical operations or functions that may include one or more mathematical operations (e.g., analytically or computationally solving differential or partial differential equations, addition, subtraction, division, multiplication, using standard deviations, averages, percentages, statistical modeling using statistical distributions, finding minimum, maximum, or similar thresholds for combined variables, etc.).

[0023] As merely an example, "best," "appropriate," or "optimize," or combinations thereof may be used interchangeably herein to refer to or define, or both, "maximize," "minimize," or achieve one or more particular targets, objectives, goals, or intents. "Best," "appropriate," or "optimize," or combinations thereof may also refer to maximizing benefit to a user (e.g., maximizing a workflow pipeline). "Best," "appropriate," or "optimize," or combinations thereof may also refer to making the most effective or functional use of a situation, opportunity, or resource.

[0024] Furthermore, "best", "appropriate", or "optimize", or combinations thereof, may not necessarily refer to the best solution or result, but may refer to a solution or result that is "good enough" for a particular application, for example. In some implementations, the objective is to suggest the "best combination" of sections / items / defined objects, but there may be various factors from which alternative suggestions of section / item / defined object / activity / application combinations that yield better results are obtained. Thus, some changes to various factors may result in a jump from one minimum / maximum to another minimum / maximum. In either case, the resulting suggestion of the section / item / defined object / activity / application combination of the optimal / best performer node may be determined by what "function" is considered to be "good enough", "near optimal", or "good enough", or combinations thereof. As used herein, the terms "best", "appropriate", or "optimize", or combinations thereof, may refer to such results based on minimum (or maximum, depending on what parameters are considered in the optimization problem) to suggest the optimal / best application, program, and / or computing component combination.

[0025] In additional aspects, the terms "optimize" and / or "optimizing" may refer to operations performed to achieve improved results, such as reduced execution costs or improved resource utilization, regardless of whether an optimal result is actually achieved. Similarly, the term "optimize" may refer to components for performing such improvement operations, and the term "optimized" may be used to describe the results of such improvement operations.

[0026] Other examples of various aspects of the illustrated embodiments, and the corresponding benefits, are described further herein.

[0027] Although this disclosure includes detailed descriptions of cloud computing, it should be understood that implementation of the teachings described herein is not limited to cloud computing environments and / or computing systems associated with one or more vehicles. Rather, embodiments of the present invention may be implemented in conjunction with any other type of computing environment now known or later developed.

[0028] Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with the service provider. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0029] The characteristics are as follows:

[0030] On-Demand Self-Service: Cloud consumers may unilaterally provision computing capacity, such as server time or network storage, automatically as needed, without the need for human interaction with the provider of the service.

[0031] Pervasive Network Access: Functionality is available over the network and is accessed through standard mechanisms that facilitate use by heterogeneous thin- or thick-client platforms (e.g., cell phones, laptops, and PDAs).

[0032] Resource Pooling: Provider computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. There is some location independence in that consumers generally have no control or knowledge of the exact location of the resources provided, but may be able to specify location at a higher level of abstraction (e.g., country, state, or data center).

[0033] Rapid Elasticity: Capabilities can be rapidly and elastically provisioned, in some cases automatically, to scale out quickly, release quickly and scale in quickly. To the consumer, the capabilities available for provisioning often appear to be unlimited, with any amount available to be purchased at any time.

[0034] Metering Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage is monitored, controlled, and reported, enabling transparency for both providers and consumers of the services being utilized.

[0035] The service model is as follows:

[0036] Software as a Service (SaaS): The functionality provided to the consumer is the use of the provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application functions, with the possible exception of limited user-specific application configuration settings.

[0037] Platform as a Service (PaaS): The functionality offered to the consumer is the deployment of applications created or acquired by the consumer, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does have control over the applications deployed and, in some cases, the application hosting environment configuration.

[0038] Infrastructure as a Service (IaaS): The functionality provided to the consumer is the provisioning of processing, storage, network, and other basic computing resources onto which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating systems, storage, applications being deployed, and possibly limited control over selected networking components (e.g., host firewalls).

[0039] The deployment model is as follows:

[0040] Private Cloud: The cloud infrastructure is operated solely for the organization. The cloud infrastructure is managed by the organization or a third party and can exist on-premise or off-premise.

[0041] Community Cloud: The cloud infrastructure is shared by several organizations to support a particular community with shared concerns (e.g., mission, security requirements, policy, and compliance considerations). The cloud infrastructure may be managed by the organization or a third party and may exist on-premise or off-premise.

[0042] Public Cloud: The cloud infrastructure is made available to the general public or large industry groups and is owned by an organization that sells cloud services.

[0043] Hybrid Cloud: The cloud infrastructure is a composition of two or more clouds (private, community, or public) that remain unique entities but are tied together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0044] A cloud computing environment is a service oriented approach that focuses on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0045] 1, a simplified diagram of an example cloud computing node is shown. Cloud computing node 10 is merely one example of a suitable cloud computing node and is not intended to suggest any limitation as to the scope of use or functionality of the inventive embodiments described herein. Regardless, cloud computing node 10 may implement and / or perform any of the functionality described above.

[0046] Cloud computing node 10 includes computer system / server 12, which is operable with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, or configurations, or combinations thereof, that may be suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0047] The computer system / server 12 may be described in the general context of computer system executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system / server 12 may be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0048] 1, a computer system / server 12 in a cloud computing node 10 is shown in the form of a general-purpose computing device. Components of computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including system memory 28 to processor 16.

[0049] Bus 18 represents any one or more of several types of bus structures including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example and not limitation, such architectures include Industry Standard Architecture (ISA) bus, MicroChannel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnects (PCI) bus.

[0050] Computer system / server 12 typically includes a variety of computer system readable media, which may be any available media that is accessible by computer system / server 12 and includes both volatile and nonvolatile media, removable and non-removable media.

[0051] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer system / server 12 may further include other removable / non-removable volatile / non-volatile computer system storage media. By way of example only, a storage system 34 may be provided for reading from and writing to a non-removable non-volatile magnetic medium (not shown, typically referred to as a "hard drive"). Although not shown, a magnetic disk drive may be provided for reading from and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive may be provided for reading from and writing to a removable non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical medium. In such a case, each may be connected to the bus 18 by one or more data media interfaces. As further illustrated and described below, system memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to implement the functions of an embodiment of the present invention.

[0052] By way of example and not limitation, a program / utility 40 having a set (at least one) of program modules 42, as well as an operating system, one or more application programs, other program modules, and program data, may be stored within system memory 28. Each of the operating system, one or more application programs, other program modules, and program data, or any combination thereof, may include an implementation of a networking environment. The program modules 42 generally implement the functions and / or methods of embodiments of the present invention as described herein.

[0053] The computer system / server 12 may also communicate with one or more external devices 14, such as a keyboard, pointing device, display 24, one or more devices that allow a user to interact with the computer system / server 12, or any device that allows the computer system / server 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.), or combinations thereof. Such communication occurs through an input / output (I / O) interface 22. Additionally, the computer system / server 12 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or combinations thereof, through a network adapter 20. As shown, the network adapter 20 may communicate with other components of the computer system / server 12 through a bus 18. Although not shown, it is understood that other hardware and / or software components may be used with the computer system / server 12. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data archive storage systems, and the like.

[0054] 2, an exemplary cloud computing environment 50 is shown. As shown, the cloud computing environment 50 comprises one or more cloud computing nodes 10 with which local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or cellular phone 54A, a desktop computer 54B, a laptop computer 54C, or an automobile computer system 54N, or combinations thereof, may communicate. The nodes 10 may communicate with each other. The nodes 10 are physically or virtually grouped into one or more networks, such as a private cloud, a community cloud, a public cloud, or a hybrid cloud as previously described, or combinations thereof (not shown). This enables the cloud computing environment 50 to provide infrastructure, platform, and / or software as a service without the cloud consumer having to maintain resources therefor on the local computing device. It should be understood that the types of computing devices 54A-N in FIG. 2 are illustrative only, and that the computing node 10 and cloud computing environment 50 may communicate with any type of computerized device over any type of network and / or network-addressable connections (e.g., using a web browser).

[0055] Referring now to Figure 3, a set of functional abstraction layers provided by cloud computing environment 50 (Figure 2) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 3 are merely exemplary, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0056] The device layer 55 includes physical and / or virtual devices and / or standalone electronics, sensors, actuators, and other objects incorporated with electronics, sensors, actuators, and other objects for performing various tasks in the cloud computing environment 50. Each of the devices in the device layer 55 incorporates networking capabilities to other functional abstraction layers, such that information obtained from the device may be provided to other functional abstraction layers, information from other abstraction layers may be provided to the device, or both. In one embodiment, the various devices comprising the device layer 55 may incorporate a network of entities collectively referred to as the "Internet of Things" (IoT). Such a network of entities allows for intercommunication, collection, and distribution of data to accomplish a wide variety of purposes, as one skilled in the art will appreciate.

[0057] The illustrated device layer 55, as shown, includes sensors 52, actuators 53, a "learning" thermostat with integrated processing 56, sensor and networking electronics, a camera 57, a controllable home outlet / receptacle 58, and a controllable electrical switch 59. Other possible devices may include, without limitation, various additional sensor devices, networking devices, electronics devices (such as remote control devices), additional actuator devices, so-called "smart" appliances such as refrigerators and washers / dryers, and a wide variety of other possible interconnected objects.

[0058] Hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframes 61, RISC (reduced instruction set computing) architecture based servers 62, servers 63, blade servers 64, storage devices 65, and networks and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0059] The virtualization layer 70 provides an abstraction layer at which the following examples of virtual entities are provided: virtual servers 71 , virtual storage 72 , virtual networks including virtual private networks 73 , virtual applications and operating systems 74 , and virtual clients 75 .

[0060] As an example, the management layer 80 may provide the functions described below. Resource provisioning 81 provides dynamic procurement of computing and other resources utilized to perform tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of such resources. As an example, such resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, and protection for data and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides cloud computing resource allocation and management such that required service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides proactive coordination and procurement of cloud computing resources for anticipated future needs in accordance with SLAs.

[0061] The workload layer 90 provides examples of functions for which a cloud computing environment may be utilized. Examples of workloads and functions provided from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom education delivery 93, data analytics processing 94, transaction processing 95, and, in the context of the illustrated embodiment of the present invention, various workloads and functions for optimized placement of data structures in hybrid memory 96. Additionally, the workloads and functions for optimized placement of data structures in hybrid memory 96 may include operations such as data analysis, data parsing, and memory allocation functions as will be further described. Those skilled in the art will appreciate that the workloads and functions for optimized placement of data structures in hybrid memory 96 may also work with other portions of the various abstraction layers (e.g., data analytics processing 94, etc.) within the hardware and software 60, virtualization 70, management 80, and other workloads 90 to accomplish various objectives of the illustrated embodiment of the present invention.

[0062] 4A, a block diagram illustrates the architecture of a deep neural network ("DNN") 400 in which exemplary embodiments are implemented. In one aspect, one or more of the components, modules, services, applications, or functions described in FIGS. 1-3, or combinations thereof, may be used in FIGS. 4A-4B.

[0063] For example, as shown in Figure 4A, the DNN 400 may be a component that is embedded, controlled, accessed, managed, or included, or combinations thereof, within the computer system / server 12 of Figure 1. In one aspect, repetitive descriptions of similar elements, components, modules, services, applications, or functions, or combinations thereof, utilized in other embodiments described herein (e.g., Figures 1-3) are omitted for brevity.

[0064] The DNN 400 may include on-chip memory 404 (e.g., NVM-CMOS hybrid memory) to populate all different types / kinds of storage and computation cores 406. The DNN 400 may be trained using one or more operations, such as, for example, backpropagation ("BP"). During training, a number of data structures, such as, for example, weights and activations, associated with each layer (e.g., convolutional layers such as Conv1, Conv2_1, Conv2_2, ..., Conv5_9, and FC1) may be generated. That is, the DNN 400 may have a layered architecture, and each layer (e.g., Conv1, Conv2_1, Conv2_2, ..., Conv5_9, and FC1) may include several nodes, each of which effectively performs some mathematical operation on some input to compute an output.

[0065] As shown, the DNN 400 may include one or more layers ("N layers"), such as, for example, layer 402 (e.g., convolutional layers such as Conv1, Conv2_1, Conv2_2, ..., Conv5_9, and a fully connected "FC1"), where some may be convolutional layers and other layers may be fully connected.

[0066] In one aspect, layer 402 illustrates an exemplary DNN layer having multiple layers, such as Conv1, Conv2_1, Conv2_2, ..., Conv5_9, and a fully connected "FC1."

[0067] The on-chip memory 404 and the computation core 406 represent a typical DNN accelerator with an engine that executes the layer 402. In one aspect, the on-chip memory 404 and the computation core 406 may include, by way of example only, a CMOS-based and / or NVM-based memory / hybrid memory for capturing all the different types / kinds of storage that captures the weights associated with the different layers (Conv1, Conv2_1, etc.) of the layer 402 and the input image for the first layer. The on-chip memory 404 and the computation core 406 engine then uses the computation core 406 to operate on each layer of the layer 402 in turn, generating outputs (called activations) that are stored in memory for a given layer. To execute the subsequent or next layer, the engine then uses those stored activations along with the weights for that layer, and execution continues.

[0068] Furthermore, during the process of using BP to train the DNN 400, one or more computation passes may occur per layer, such as, for example, Conv1, Conv2_1, Conv2_2, ..., Conv5_9, and FC1. The computation passes may include a forward pass (e.g., forward propagation "FP"), a BP, and a weight update pass. For example, initially, during FP, the weights of each layer, such as, for example, Conv1, Conv2_1, Conv2_2, ..., Conv5_9, and FC1, are used to generate activations (e.g., output activations). As time progresses, at each time step, an FP operation is completed at each of the layers, such as, for example, Conv1, Conv2_1, Conv2_2, ..., Conv5_9, and FC1.

[0069] The input to a given node is a weighted sum of the outputs of the previous layer (plus a bias term, usually equal to one or zero). As illustrated (via dashed lines), some outputs of some layers are needed as inputs to other layers, e.g., the output of Conv1 is also used as input to Conv2_2, and vice versa. It is these weights that are learned during training. To learn these parameters, the outputs of a training run are compared with the values, the error is backpropagated through the network, and the weights are updated.

[0070] It is noted that current accelerators for running DNNs can only address smaller models (e.g., ResNet-50) and are limited by on-chip SRAM capacity limitations, e.g., power and silicon area budgets. Therefore, the present invention targets larger models for inference, e.g., ResNet-101 (with approximately 39.8 million "M" weights). A data structure during inference provides activations and weights, and the mini-batch size may be equal to 1 (e.g., latency-constrained inference). Furthermore, activation and weight precision may be 8 bits.

[0071] Note that merely by way of example, inference may mean that a trained DNN model is used to infer or predict a given input based on an image shown to the trained DNN model (e.g., classifying the input as "X", such as cat, or "Y", such as dog), or to translate text from a first language to a second language based on the inputted text. Inference may be a generative phase, and the model may be used to predict real-world data.

[0072] Thus, as illustrated, the model size of the DNN 400 may be 40 megabytes ("MB") generating a maximum ("max") activation storage for layer 402, such as 13 megabytes ("MB") of activation storage. Note that the model size may be the total capacity of the model, i.e., weights (and biases), and the maximum activation storage represents what is the maximum size of the activations that need to be stored in memory. Thus, by applying the mechanisms of the illustrated embodiment, a higher density on-chip memory 404 allows for a larger on-chip buffer capacity. However, the current state of the art stores weights and activations on NVM-based on-chip buffers 404 (e.g., compute cores 406). Non-hybrid NVM improves write energy and latency.

[0073] It should be noted that non-hybrid memories (e.g., NVM only) are less efficient than hybrid NVMs (e.g., NVM and SRAM as shown in FIG. 5). For example, the data mapping (e.g., data placement) of the data structures does not depend on the characteristics of the DNN data structures. For example, weights are static (e.g., data is written once and read several times). Activations are dynamic (e.g., data structures are read / written several times). Thus, without the placement of hybrid memories (see FIG. 5), DNNs utilizing non-hybrid memories create increased energy inefficiencies due to the high write energy of activations. For example, NVM write energy is less efficient than hybrid memories (e.g., NVM / SRAM read energy) of the illustrated embodiment. NVM write energy is several times higher (e.g., 3-5 times higher) because activations are frequently written to (and read from) the NVM / CMOS buffer. Furthermore, the high write energy and performance penalty associated with slow write operations outweighs the benefit of low leakage of NVM. This is because off-chip memory latency overlaps with computation latency (i.e., computation latency is hidden), reducing the opportunity for latency to be hidden by slow writes.

[0074] 4B, a diagram 415 illustrates a DNN 410, which may include one or more layers ("N layers"), such as, for example, layers 1-5 (e.g., some may be convolutional layers and others may be fully connected). The DNN 410 may be trained using one or more operations, such as, for example, backpropagation ("BP"). During training, a number of data structures may be generated that are associated with each layer (e.g., layers 1-5), such as, for example, weights ("WEI") 440, activations ("ACT") 420, deactivations ("DACT") 430, and weight gradients ("WGRAD" or "WG") 450.

[0075] The DNN 410 may include a sequence of convolutional layers and fully connected layers. DNN inference uses forward propagation (FP), which includes two data structures: weights ("WEI") and activations ("ACT"). In one embodiment, two exemplary types of data structures may be ACT / WEI, which is equivalent to approximately 2N data structures for N layers, where "N" is a positive integer or a defined value. Furthermore, there may be one computational pass per layer, which may be equivalent to approximately N computational passes for N layers.

[0076] During the process of using backpropagation ("BP") to train the DNN 410, one or more computational passes may occur per layer, e.g., layer 1 ("L1"), layer 2 ("L2"), layer 3 ("L3"), layer 4 ("L4"), or layer 5 ("L5") (or layer N ("LN"), where "N" is a positive integer), or combinations thereof. The computational passes may include a forward pass (e.g., forward propagation "FP"), a backward pass (e.g., backpropagation "BP"), and a weight update pass (e.g., weight gradient). By way of example only, while training a DNN 410 using backpropagation ("BP"), various dependencies that arise during each time step (illustrated in FIG. 4B, which is for illustrative purposes only and should not be construed as being of a limiting nature) are shown for weights ("WEI") 440, activations ("ACT") 420, deactivations ("DACT") 430, and weight gradients ("WGRAD" or "WG") 450.

[0077] For example, initially, during a forward pass ("FP"), the weights (WEI 440) of each layer are used to generate activations (ACT 420). As time progresses, at each time step, FP operations are completed at each of the layers, and a series of alternating steps are performed between a weight upgrade step (e.g., WGRAD or WG 450) and a backward pass (e.g., backpropagation), which also include various dependencies between each other. For example, a weight upgrade step (e.g., WGRAD or WG 450) may require one or more of the activations (e.g., ACT 420) generated from the forward pass FP, and then one or more of the deactivations (e.g., DACT 430) may be generated from the BP.

[0078] Referring now to FIG. 5, a block diagram illustrates an optimized DNN architecture 500 having a hybrid memory for optimized placement of data structures in the hybrid memory with non-volatile deep neural network inference operations, in which an exemplary embodiment is implemented. In an aspect, one or more of the components, modules, services, applications, or functions described in FIGS. 1-4, or combinations thereof, may be used in FIG. 5. For example, the optimized DNN architecture 500 may be a component that is incorporated, controlled, accessed, managed, or included in the computer system / server 12 of FIG. 1, or combinations thereof. In an aspect, repetitive descriptions of similar elements, components, modules, services, applications, or functions, or combinations thereof, utilized in other embodiments described herein (e.g., FIGS. 1-4) are omitted for brevity.

[0079] As shown, the optimized DNN architecture 500 may include one or more types of memory, such as a first memory 510 (e.g., NVM) and a second memory 520, such as a CMOS-based memory (e.g., SRAM), a hybrid controller 540, and a computational core 530, each of which may be associated with and / or in communication with each other. In one aspect, the first memory 510 (e.g., NVM) and the second memory 520 (e.g., SRAM) form a hybrid memory. In one aspect, the hybrid memory may include an on-chip NVM-based buffer as well as an on-chip CMOS-based buffer. In one aspect, the DNN 500 may partition the memory (e.g., the first memory 510 (e.g., NVM) and the second memory 520 (e.g., SRAM)) to form a hybrid memory for storage of model parameters and output activations.

[0080] It should be noted that the first memory 510 (e.g., NVM) and the second memory 520 (e.g., SRAM) may form a hybrid memory. A hybrid memory may be a single memory that includes both the first memory 510 and the second memory 520, and the first memory 510 and the second memory 520 may be located on the same computing system. Alternatively, a hybrid memory may be two or more memories that include both the first memory 510 and the second memory 520, and the first memory 510 and the second memory 520 may be separate, separated, and located on different or remote computing systems.

[0081] In one aspect, the "0" above the first memory 510 (e.g., NVM) and the "1" below the second memory 520 (e.g., SRAM) may be or represent a flag or indicator (indicating whether it is an activation ("ACT") or a weight ("W"). When the value is "0", it means that the value represents a weight data structure, and when the value is "1", it means that the value represents an activation. This indicator may help the controller to distinguish the weight from the activation and thus redirect / reroute it to the correct destination, i.e., NVM or CMOS.

[0082] In operation, each read ("RD") and each write ("WR") may be controlled by the hybrid controller 540. That is, the hybrid controller 540 may determine and distinguish between the weights and activations of the data structure ("DS"). For example, the hybrid controller 540 may store (read from) the model parameters in the NVM-based memory or "NVM" 510 for each RD. The hybrid controller 540 may use a CMOS-based memory (e.g., the second memory 520 (e.g., SRAM)) to store the output activation in the CMOS-based memory (e.g., the second memory 520 (e.g., SRAM)). In an additional aspect, RD and WR represent read and write operations, indicating the operation of reading and writing to the memory (e.g., NVM is always read, SRAM can be read or written). That is, the output activation ("ACT") is both read and write, and the weight ("W") is only read.

[0083] In one aspect, a hybrid memory (e.g., first memory 510 and second memory 520) may include memory types that form a single memory, such as a single hybrid memory in a single computing system. Alternatively, a hybrid memory (e.g., first memory 510 and second memory 520) may include one or more memory types that are distinct and separate from one another, such as in different computing systems.

[0084] In one aspect, the hybrid memory may be comprised of elements such as, for example, SRAM. The first memory 510 (e.g., an NVM-based memory) may include elements including, but not limited to, magnetoresistive random access memory ("MRAM"), memristor, resistive RAM ("RRAM"), conductive bridging RAM ("CBRAM"), or phase change memory ("PCM") elements. Thus, the hybrid controller 540 may provide dynamic (programmable) allocation and partitioning of the hybrid memory (e.g., the first memory 510 (e.g., NVM) and the second memory 520 (e.g., SRAM)) for model parameters and output activation. In an additional aspect, one or more computational cores, such as, for example, the computational core 530, may provide a DNN with dynamic (programmable) allocation and partitioning of the hybrid memory for model parameters and output activation between one or more computational cores, such as, for example, the computational core 530.

[0085] Thus, the DNN 500 with hybrid memory may exploit the various read / write characteristics of the DNN data structures, such as by distinguishing activations (“ACT”) from weights (“W”) when reading from / writing to memory, and the number of writes (“WR”) to the first and second memories (e.g., the second memory having a higher write energy) is minimized, achieving and improving energy efficiency.

[0086] Referring now to FIG. 6, a block diagram illustrating exemplary functional components of a system 600 according to various features of the illustrated embodiment is shown. FIG. 6 illustrates a system 600 for optimally arranging data structures in a hybrid memory in a computing environment. In one aspect, one or more of the components, modules, services, applications, or functions, or combinations thereof, described in FIGS. 1-5 may be used in FIG. 6. In one aspect, repetitive descriptions of similar elements, components, modules, services, applications, or functions, or combinations thereof, utilized in other embodiments described herein (e.g., FIGS. 1-5) are omitted for brevity.

[0087] With the above in mind, the modules / component blocks of system 600 may also be incorporated into various hardware and software components of a system for protecting data compression in accordance with the present invention. Many of the functional blocks of system 600 may be executed as background processes within distributed computing components, or on user devices, or on various components elsewhere.

[0088] The computer system / server 12 of Figure 1 is shown incorporating a DNN inference engine service 610. The DNN inference engine service 610 may incorporate the processing unit 16 ("processor") and memory 28 of Figure 1 and perform various computations, data processing, and other functions in accordance with various aspects of the present invention. The DNN inference engine service 610 may be provided by the computer system / server 12 of Figure 1.

[0089] As one of ordinary skill in the art will appreciate, the illustration of various functional units within the DNN inference engine service 610 is for illustrative purposes only, as the functional units may be located within the DNN inference engine service 610 or may be located elsewhere within or between the distributed computing components, or both.

[0090] In one aspect, the computer system / server 12 and / or the DNN inference engine service 610 may provide virtualized computing services (i.e., virtualized computing, virtualized storage, virtualized networking, etc.). More specifically, the DNN inference engine service 610 may provide and / or be included within virtualized computing, virtualized storage, virtualized networking, and other virtualized services running on the hardware substrate.

[0091] The DNN inference engine service 610 may include a differentiation component 640, a data placement component 650, a partitioning component 660, and a memory component 670, each of which may be in communication with each other.

[0092] In one aspect, the differentiation component 640 can differentiate between weights and activations of one or more data structures for placing the one or more data structures in the hybrid memory. The differentiation component 640 can also differentiate between weights and activations of one or more data structures according to characteristics of a read operation, a write operation, or a combination thereof.

[0093] The data placement component 650 may dynamically allocate one or more data structures in a memory component (e.g., a hybrid memory) according to weights and activations of one or more data structures in a deep neural network ("DNN"). In one aspect, the memory component 670 is a hybrid memory that includes both non-volatile memory ("NVM")-based memory and CMOS-based memory (e.g., SRAM).

[0094] The partitioning component 660 may partition a memory component (e.g., a hybrid memory) for weights and activations of one or more data structures. The data placement component 650 may store the weights in an NVM-based memory of the hybrid memory. The data placement component 650 may store the activations in a CMOS-based memory of the hybrid memory. In an additional aspect, the data placement component 650 may minimize write operations of one or more data structures to a non-volatile memory based on distinguishing between the weights and activations.

[0095] 7, an additional method 700 for optimally placing data structures in a hybrid memory by a processor in a computing environment in which various aspects of the illustrated embodiments may be implemented is shown. The function 700 may be implemented as a method executed as instructions to a machine, the instructions being contained on at least one computer readable medium or one non-transitory machine readable storage medium. The function 700 may begin at block 702.

[0096] As in block 704, model parameters and activations of the data structures may be differentiated to place the data structures in the hybrid memory. As in block 706, the hybrid memory may be partitioned for the model parameters and activations of one or more data structures. As in block 708, the data structures may be dynamically allocated in the hybrid memory according to the model parameters and activations of one or more data structures in a deep neural network ("DNN"). The hybrid memory includes at least a first memory and a second memory that differ according to write attributes (e.g., write energy, write latency, write durability, or a combination thereof). The function 700 may end as in block 710.

[0097] In one aspect, in conjunction with or as part of at least one block of FIG. 7, or both, operations of method 700 may include each of the following: The operations of method 700 may dynamically allocate one or more data structures in a hybrid memory according to weights and activations of one or more data structures in a DNN. The one or more write attributes may include write energy, write latency, write endurance, or a combination thereof. The hybrid memory may include non-volatile memory ("NVM")-based memory and CMOS-based memory. The operations of method 700 may partition the hybrid memory for weights and activations of one or more data structures.

[0098] The operations of method 700 may distinguish between weights and activations of one or more data structures to place the one or more data structures in a hybrid memory. The operations of method 700 may distinguish between weights and activations of one or more data structures according to characteristics of a read operation, a write operation, or a combination thereof.

[0099] The operations of method 700 store the weights in a first memory (e.g., an NVM-based memory) of the hybrid memory and store the activations in a second memory (e.g., a CMOS-based memory) of the hybrid memory. The operations of method 700 minimize write operations of one or more data structures to the hybrid memory based on distinguishing between the weights and the activations. Thus, the hybrid memory may include a first memory and a second memory that differ in attributes, properties, or characteristics, or combinations thereof, such as, for example, write endurance, write energy, write latency, etc. In one aspect, the first memory may be NVM and the second memory may be CMOS.

[0100] The present invention may be a system, a method, and / or a computer program product. The computer program product may include a computer readable storage medium having computer readable program instructions thereon that cause a processor to implement aspects of the present invention.

[0101] A computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or ridges in a groove structure with instructions recorded on them, and any suitable combination of the above. As used herein, computer-readable storage media should not be interpreted as signals that are transitory in nature, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through wires.

[0102] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to the respective computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in the respective computing / processing device.

[0103] The computer readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk®, C++, and traditional procedural programming languages ​​such as the “C” programming language and similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or the connection may be made to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer readable program instructions by utilizing state information of the computer readable program instructions to individualize the electronic circuitry to implement aspects of the invention.

[0104] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, are implemented by computer readable program instructions.

[0105] Such computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, produce means for implementing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams to create a machine. Such computer-readable program instructions may also be stored in a computer-readable storage medium that may direct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture including instructions that implement aspects of the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0106] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device such that the instructions, which execute on the computer, other programmable apparatus, or other device, implement the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams, causing the computer, other programmable apparatus, or other device to perform a series of operational steps to create a computer-implemented process.

[0107] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions that includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions described in the blocks may occur in an order other than that described in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It will also be noted that each block in the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, are implemented by a dedicated hardware-based system that performs the specified functions or operations, or a combination of dedicated hardware and computer instructions.

Claims

1. 1. A method for optimized placement of data structures in memory in a processor-based computing environment, comprising: Dynamically allocating one or more data structures in a hybrid memory according to weights and activations of one or more data structures in a deep neural network ("DNN"), the hybrid memory including at least a first memory and a second memory that differ according to one or more write attributes; distinguishing between the weights and the activations of the one or more data structures for placing the one or more data structures in the hybrid memory; and minimizing write operations of the one or more data structures to the hybrid memory based on distinguishing between the weights and the activations. The method includes:

2. 2. The method of claim 1, further comprising partitioning the hybrid memory for the weights and activations of one or more data structures, wherein the one or more write attributes include write energy, write latency, write endurance, or a combination thereof.

3. The method of claim 1 , further comprising: differentiating between the weights and the activations of the one or more data structures according to characteristics of read operations, write operations, or a combination thereof.

4. The method of claim 1 , further comprising storing the weights in the first memory of the hybrid memory.

5. The method of claim 1 , further comprising storing the activation in the second memory of the hybrid memory.

6. 1. A system for optimized placement of data structures in memory in a computing environment, comprising: When executed, the system comprises: Dynamically allocating one or more data structures in a hybrid memory according to weights and activations of one or more data structures in a deep neural network ("DNN"), the hybrid memory including at least a first memory and a second memory that differ according to one or more write attributes; distinguishing between the weights and the activations of the one or more data structures for placing the one or more data structures in the hybrid memory; and minimizing write operations of the one or more data structures to the hybrid memory based on distinguishing between the weights and the activations. One or more computers having executable instructions to A system comprising:

7. 7. The system of claim 6, wherein the executable instructions, when executed, cause the system to partition the hybrid memory for the weights and activations of one or more data structures, and the one or more write attributes include write energy, write latency, write endurance, or a combination thereof.

8. 7. The system of claim 6, wherein the executable instructions, when executed, cause the system to distinguish between the weights and the activations of the one or more data structures according to characteristics of a read operation, a write operation, or a combination thereof.

9. 7. The system of claim 6, wherein said executable instructions, when executed, cause said system to store said weights in said first memory of said hybrid memory.

10. 7. The system of claim 6, wherein said executable instructions, when executed, cause said system to store said activation in said second memory of said hybrid memory.

11. A computer program product causing one or more computers to carry out the method according to any one of claims 1 to 5.

12. A computer readable storage medium having the computer program according to claim 11 stored therein.

Images