Neural network training using dataflow graphs and dynamic memory management

Abstract

A neural network dataflow graph having a set of nodes and a set of edges is processed. An insertion point for a memory reduction or memory restoration operation is determined. The determination is based on computing tensor timing slack (TTS) for a set of input tensors; using input tensors having respective TTS values greater than a threshold (thTTS), editing a candidate list (SI) of input tensors in the set of input tensors; filtering the SI to retain input tensors whose size satisfies a threshold (thS); and using the filtered SI to determine the insertion point for the operation. A new dataflow graph is generated or an existing dataflow graph is modified using the process.

Description

Technical Field

[0001] This invention generally relates to processing large machine learning and neural network models in environments with limited memory capacity. More specifically, this invention relates to modifications to computational data flow graphs that reduce their memory footprint, i.e., require less memory during the evaluation of the graph, thus enabling the processing of larger graphs or larger data samples (or processing multiple samples at once). Background Technology

[0002] Neural networks are artificial neural networks (ANNs) that mimic the function of the human brain, featuring weighted connections between their nodes or "neurons." Deep neural networks (DNNs) are artificial neural networks with multiple "hidden" layers between their input and output layers. The hidden layers of a DNN allow it to model complex nonlinear relationships characterized by higher abstract representations of the data, where each hidden layer determines a nonlinear transformation of the previous layer. Training large deep neural networks is a challenging computational problem. The introduction of graphics processing units (GPUs) with their extraordinary performance has allowed progress from recognizing handwritten digits (MNIST) and tiny images (CIFAR 10) to classifying high-resolution images (ImageNet). A key factor in GPU computational performance is its high-speed memory. Unfortunately, GPU memory is much smaller than a computer's main memory (CPU memory). The size of the GPU memory limits both the depth of a DNN and the size of the data samples. As a result, there are often constraints on image size and resolution, which in turn affect the accuracy of the DNN. Memory constraints can be mitigated by reducing the memory footprint of the data. In an illustrative embodiment, the GPU may send data from its memory to the CPU to free up additional memory space, and then retrieve the data from the CPU after using the additional memory space for its intended function. Summary of the Invention

[0003] Embodiments of the present invention provide methods, systems, and computer program products for reducing overall GPU memory usage by automatically determining the insertion points of memory save operations in an underlying data flow graph comprising a set of nodes and a set of edges while processing neural networks.

[0004] In an embodiment, a method for reducing overall GPU memory usage as described above computes a tensor timing relaxation (TTS) of a set of input tensors, edits a candidate list (SI) of input tensors in the set of input tensors using input tensors with corresponding TTS values ​​greater than a threshold (thTTS), filters the SIs to retain input tensors whose size meets the threshold (thS), and uses the filtered SIs to determine an insertion point for operation.

[0005] In an embodiment, the operation is a combination of a reduction operation and a restoration operation, wherein the reduction operation includes either or both of a copy operation to the central processing unit (CPU) memory and a compression operation, and wherein the restoration operation includes either or both of a copy operation from the CPU memory and a decompression operation.

[0006] In an embodiment, calculating the TTS includes performing timing analysis using the input tensor.

[0007] In this embodiment, the timing analysis includes initializing the Tensor Arrival Time (TAT), Tensor Requirement Time (TRT), and Tensor Timing Relaxation (TTS) values ​​of the input tensors; for a set of input tensors Q, when Q is not empty, the following steps are performed: selecting a node q to be excluded from Q; excluding q from Q; and for each subsequent input tensor s of q, setting the TAT of s to be equal to the TAT of q.

[0008] In an embodiment, initialization includes initializing the TAT, TRT, and TTS values ​​to unknown values ​​or zero.

[0009] In an embodiment, all inputs to node n have known TAT values, where n is a node with s as an input tensor, and the method further includes performing the following steps for the set of input tensors Q: setting the TRT of n to the maximum value of the TAT of n's input s; setting the TTS of each input i of n to the difference between n's TRT and s's TAT; setting the TAT of n based on n's TRT incremented by node delay; and including n back in Q.

[0010] In an embodiment, the data flow graph includes a pair (N, E), where N includes a set of nodes, E includes a set of hyperedges, and a given node in N implements one or more operators on one or more tensors.

[0011] In an embodiment, a given node comprises a triple (f, I, O), where f is a function of an operator implemented by the given node, I is a set of input tensors of the given node, and O is a set of outputs of the given node generated based on the operator.

[0012] In an embodiment, a hyperedge defines how a tensor is passed from the operator that generates the tensor to the operator that uses the tensor as an argument.

[0013] In an embodiment, the hyperedge comprises a pair (s, H), where s is a node output and H is a set of node inputs.

[0014] In an embodiment, the method further includes: inserting a subgraph node corresponding to the operation into the data flow graph.

[0015] In an embodiment, the insertion generates a new data flow graph or modifies an existing data flow graph.

[0016] In one embodiment, the insertion generates a new data flow graph, wherein the new data flow graph includes a complete data flow graph or a partial data flow graph.

[0017] In one embodiment, the method further includes: processing the set of input tensors using a data flow graph based on the insertion; and generating an output based on the processing.

[0018] In one embodiment, the operation is a memory reduction operation, and the method further includes inserting a memory reduction subgraph node corresponding to the memory reduction operation.

[0019] In an embodiment, the insertion further includes: connecting a first node to a memory-reduced subgraph node via a first hyperedge, the first node corresponding to a source node; and connecting a memory-reduced subgraph node to a second node via a second hyperedge, wherein the second hyperedge includes a serialized hyperedge, and the second node corresponds to an intermediate node or a destination node.

[0020] In an embodiment, the insertion further includes connecting the first node to the second node via a third hyperedge.

[0021] In this embodiment, the operation is a memory recovery operation, and the method further includes: inserting a memory recovery subgraph node corresponding to the memory recovery operation.

[0022] In an embodiment, the insertion further includes: connecting a first node to a memory recovery subgraph node via a first hyperedge, wherein the second hyperedge includes a serialization hyperedge or a prefetch hyperedge; connecting a memory recovery subgraph node to a second node via the second hyperedge, the second node corresponding to a destination node; and connecting a memory reduction subgraph node to a memory recovery subgraph node via a third hyperedge.

[0023] In an embodiment, the method further includes connecting two nodes of a data flow graph via a hyperedge, wherein the connection includes any of the following: a direct connection via a hyperedge between the two nodes; and an indirect connection via one or more additional nodes and a hyperedge between the two nodes.

[0024] In an embodiment, the steps of the method are performed iteratively to insert a set of subgraph nodes into a data flow graph for performing at least one memory reduction operation and at least one memory recovery operation.

[0025] In an embodiment, the step of iteratively performing the method includes inserting a set of hyperedges into the data flow graph, the set of hyperedges including at least one serialized hyperedge, or at least one prefetched hyperedge, or at least one serialized hyperedge and at least one prefetched hyperedge.

[0026] In an embodiment, the above method is stored as programming instructions stored on one or more tangible storage media of one or more computer systems and executed by one or more processors of the one or more computer systems that execute programming instructions. The programming instructions may be stored as a computer program product.

[0027] According to one aspect, a method is provided for reducing overall GPU memory usage while processing neural networks, the method comprising: automatically determining the insertion point of a memory save operation in a basic data flow graph comprising a set of nodes and a set of edges.

[0028] According to another aspect, a method is provided, comprising: selecting an insertion point for an operation in a data flow graph based on a set of Tensor Timing Relaxations (TTS) and a candidate list of input tensors (SI); calculating the Tensor Timing Relaxations (TTS) of the set of input tensors; editing the candidate list (SI) of input tensors in the set of input tensors using input tensors having corresponding TTS values ​​greater than a threshold (thTTS); filtering the SIs to retain input tensors whose size satisfies a threshold (thS); and using the filtered SIs to determine the insertion point for a reduction and recovery operation.

[0029] According to another aspect, a method is provided, comprising: determining, by identifying the memory requirements of output data to be generated by node P, to apply a memory reduction action to subgraph nodes A and P; sending data from a source node to subgraph nodes A and P; enabling subgraph node A to perform the memory reduction action before execution of node P, the memory reduction action making additional memory space available for a graphics processing unit (GPU) to execute one or more functions; delaying a memory recovery action for subgraph node B until node Q has completed its execution; and sending the data from the central processing unit (CPU) of subgraph node B to the GPU to recover the data to the GPU.

[0030] According to another aspect, a computer program including program code means is provided, wherein when the program is run on a computer, the program code means is adapted to perform the method described in any of the paragraphs

[0027] ,

[0028] or

[0029] .

[0031] According to another aspect, a system for reducing overall GPU memory usage when processing neural networks is provided, the system being operable to automatically determine the insertion point of a memory save operation in a basic data flow graph comprising a set of nodes and a set of edges.

[0032] According to another aspect, a system is provided that can perform the following operations: selecting an insertion point for an operation in a data flow graph based on a set of Tensor Timing Relaxations (TTS) and a candidate list of input tensors (SI); calculating the Tensor Timing Relaxations (TTS) of a set of input tensors; editing the candidate list (SI) of input tensors in the set of input tensors using input tensors with corresponding TTS values ​​greater than a threshold (thTTS); filtering the SIs to retain input tensors whose size satisfies a threshold (thS); and using the filtered SIs to determine the insertion point for a reduction and recovery operation.

[0033] According to another aspect, a system is provided capable of performing the following operations: determining, by identifying the memory requirements of output data generated by node P, to apply a memory reduction action to child node A and node P; sending data from the source node to child node A and node P; enabling child node A to perform the memory reduction action before the execution of node P, the memory reduction action making additional memory space available for the graphics processing unit (GPU) to execute one or more functions; delaying the memory recovery action of child node B until node Q has completed its execution; and

[0034] The data is sent from the CPU of subgraph node B to the GPU to restore the data to the GPU. Attached Figure Description

[0035] Preferred embodiments of the invention will now be described by way of example only and with reference to the following figures:

[0036] Figure 1 A block diagram of a data processing system network in which illustrative embodiments can be implemented is shown;

[0037] Figure 2 A block diagram of a data processing system in which illustrative embodiments can be implemented is shown;

[0038] Figure 3 A data flow diagram according to an illustrative embodiment is shown;

[0039] Figure 4 A data flow diagram with a memory reduction subgraph according to an illustrative embodiment is shown;

[0040] Figure 5 A data flow diagram with prefetching and serialization is shown according to an illustrative embodiment;

[0041] Figure 6 A data flow subgraph according to an illustrative embodiment is shown;

[0042] Figure 7 A data flow subgraph according to an illustrative embodiment is shown;

[0043] Figure 8 A data flow subgraph according to an illustrative embodiment is shown;

[0044] Figure 9 A data flow subgraph according to an illustrative embodiment is shown;

[0045] Figure 10A A flowchart illustrating an exemplary process for determining the insertion point for reduction and recovery operations, according to an illustrative embodiment, is shown;

[0046] Figure 10B A flowchart illustrating an exemplary process for performing timing analysis according to an illustrative embodiment is shown;

[0047] Figure 11 A memory profile graph according to an illustrative embodiment is shown;

[0048] Figure 12 A schematic diagram of a memory according to an illustrative embodiment is shown; and

[0049] Figure 13 A flowchart illustrating an exemplary process for memory reduction in a graphics processor according to an illustrative embodiment is shown. Detailed Implementation

[0050] GPUs have only specific memory space used for computation or performing various functions. Illustrative embodiments recognize that GPUs can efficiently move data to the CPU or compress their own data to free up additional memory space needed to perform computations and / or functions. Swap-out nodes move their input tensors from the GPU to the CPU. Swap-in nodes move data from the CPU back to the GPU. Data recovery to the GPU is delayed until the data is needed for further computation.

[0051] The illustrative embodiments recognize that serialization and tensor swapping can minimize GPU memory requirements. Tensor swapping and data compression are used for reduction. Since the GPU compresses data faster than sending data to the CPU, data compression significantly reduces computation time. Applying data compression alone can save memory space with minimal runtime loss. Compressing the tensors to be swapped reduces the time lost due to data transfer.

[0052] The illustrative embodiments illustrate how memory reduction can be improved through proper serialization of data flows. These illustrative embodiments recognize that data flow graphs with memory management are an effective technique for training deep neural networks. A data flow graph illustrates the flow of tensors or data from the GPU to the CPU within a node of a data processing system, as well as the flow of data from one data node to another. Memory management refers to the GPU's ability to efficiently free up additional memory space. Typically, a data flow graph has computational nodes that take input data and produce output data. The GPU is responsible for allocating and deallocating these tensors or data in a timely manner. However, insufficient GPU memory can severely limit the size of the data flow graph and the corresponding tensors.

[0053] The illustrative embodiments illustrate how a GPU can free up additional memory space by transferring data to the CPU. However, a problem with data exchange between the GPU and CPU (where the GPU transfers some of its data to the CPU to free up additional memory space) is how to perform this process in a timely manner as data flows through the data processing system and is sent through various nodes. Swapping large amounts of data from the GPU to the CPU can be time-consuming. The swapping-out node moves data or input tensors from the GPU to CPU memory. The swapping-in node moves the data or input tensors back to GPU memory during memory recovery. Thus, large data flow graphs can be executed with limited GPU memory. Tensor swapping between the GPU and CPU, as well as data compression, can be efficiently used to free up additional memory space required to execute large data flow graphs. The illustrative embodiments recognize that the more tensors involved in memory conservation, the greater the memory savings, and that larger data flow graphs can be processed in GPU memory due to the additional memory space available in the GPU. Furthermore, the GPU can also decide to move a larger number of smaller or medium-sized tensors to free up memory space, which can be less time-consuming than moving large amounts of data to the CPU.

[0054] The illustrative embodiments recognize that the memory reduction process can be improved through appropriate serialization and prefetching performed via the data flow graph. Due to tensor swapping with the CPU, the GPU's memory requirements are significantly reduced. Prefetching delays data recovery to the GPU until the data is actually needed.

[0055] The illustrative embodiments recognize that data transfers from one device to another (e.g., from a GPU to a CPU and vice versa) can be time-consuming. In these embodiments, since memory stores the time incurred for data transfer, compression, and decompression (the terms decompress and other forms are used interchangeably with uncompress and other forms), it should only be applied to tensors that occupy GPU memory for a sufficiently long time without participating in any computation. Therefore, the GPU can send data (tensors) to the CPU and receive the same data from the CPU that is not involved in computation or functionality. Serialization allows additional structural modifications to the dataflow graph to be added in the form of control dependency edges rather than regular data transmission edges. The GPU can swap or send tensors (data) to the CPU to free up additional memory space required to perform a series of functions or computations.

[0056] In the illustrative embodiment, memory recovery can be delayed. Additional structural modifications can be added to the dataflow graph in the form of control dependency edges. The GPU can reclaim data that it initially transferred to the CPU to perform functions and / or computations. Nodes can apply control dependency edges to subgraph nodes to delay memory recovery until the node has completed its execution.

[0057] In illustrative embodiments, the GPU may also decide not to exchange tensors with the CPU or another device. Instead, the GPU may decide to compress its tensors internally to free up additional memory space. Serialization allows the GPU to begin compressing data before data transfer between nodes resumes, freeing up additional memory space. Prefetching allows the GPU to delay decompressing data, allowing nodes to transfer their data before the GPU decompresses the compressed data.

[0058] The illustrative embodiments provide memory reduction and memory recovery actions that allow the GPU to send data to the CPU to free up additional memory space for computation and / or functions, and then retrieve its data from the CPU after the additional memory space has been used. The illustrative embodiments also provide a program product and computer system that includes the above-described memory reduction and memory recovery procedures for transferring data to the CPU to free up additional memory space and then retrieving its data from the CPU after the additional memory space has been used.

[0059] The illustrative embodiments used to describe the present invention generally relate to and solve the aforementioned problems of memory management in GPUs within data processing systems of deep learning systems. The illustrative embodiments provide a method, computer program product, and computer system for memory management in GPUs of deep learning systems.

[0060] In this embodiment, timing analysis calculates the timing of events occurring in the data flow graph during its execution. Timing is a criterion for selecting tensors (data) for memory reduction. Timing analysis also assumes that computation of data at all outputs of a given node is completed simultaneously.

[0061] Timing variables used in timing analysis include Tensor Arrival Time (TAT), Tensor Requirement Time (TRT), and Tensor Timing Relaxation (Relaxation Time). TAT is the time it takes for data computation to complete and for that data to be available to the operator / node. TRT is the time it takes for data from the node's input to be needed for the node's execution. TTS is the difference between TRT and TAT.

[0062] The timing analysis algorithm begins by initializing TAT, TRT, and TTS in the data flow graph to unknowns. For each source node in the data flow graph, TAT, TRT, and TTS = 0. Any node in the data flow graph can be selected to perform timing analysis involving TAT, TRT, and TTS.

[0063] Some basic criteria for selecting candidate tensors for memory storage include tensor size, tensor timing relaxation, and the total number of tensors. Small tensors may not be the best candidates for data exchange. Conversely, larger tensor timing relaxations can remain in CPU memory or in a compressed state for a longer period. Furthermore, a larger number of tensors involved in memory storage can free up more GPU memory for other computations.

[0064] In the illustrative embodiment, timing analysis can be used to identify one or more nodes in a data flow group. One or more nodes in the data flow graph that produce output data with the largest memory requirements can be identified. Structural modifiers (such as "serialization" and "prefetch") can be inserted into one or more nodes in the data flow graph. Serialization of a data flow graph node is performed by defining a control dependency of one node on another. Execution of node B does not begin until execution of node A is fully completed. This means that the output tensor of node B is not allocated until execution of node A is complete. Prefetching is performed by calling control dependency edges from one node to another to delay tensor reconstruction or memory recovery. The node with the applied control dependency edge completes its execution before tensor reconstruction or memory recovery occurs.

[0065] One embodiment provides a method for memory management within a GPU, in which the GPU transfers data to the CPU to free up additional memory space for performing one or more functions or computations. The method also includes memory recovery, wherein the CPU can return the recovered data to the GPU. Another embodiment provides a computer program product having program instructions for the GPU to send data to the CPU to free up additional memory space. The computer program product also includes program instructions for memory recovery to allow the GPU to receive data back from the CPU. Another embodiment provides a computer system having program instructions for the GPU to transfer data to the CPU. The computer system also includes program instructions for using memory recovery to transfer data back to the GPU.

[0066] Completion time refers to the time it takes for a tensor or data computation to be completed at a node and become available to other nodes. By definition, relaxation time is the amount of time an input can be delayed without causing overall operational delay, or the amount of time a tensor waits at a node's input before being used for computation.

[0067] In this embodiment, a memory device (such as a GPU) is added to a data processing system in which nodes are transmitting and receiving data. The GPU attempts to execute a function or computation that will result in the acquisition of additional memory space. To execute the function or computation, the GPU decides to move data to the CPU within the data processing system to free up the additional memory space. During memory reduction, a subgraph node applies control dependency edges to the node to allow the GPU to send data to the CPU and free up the additional memory space. This node is a dependent node, and the subgraph node is the control node during serialization routing. The subgraph node is serialized with the node, wherein the subgraph node performs its execution before the node performs its execution. When a node sends its data to other nodes, another structural modification can be added to the data flow graph to serialize the execution of the node. The illustrative embodiment may have multiple structural modifications added to the data flow graph, and wherein the GPU performs its execution of sending data to the CPU before the node sends its output data to another node.

[0068] Once the GPU has finished using the additional memory space for its intended function, it can retrieve the data it initially sent to the CPU, thus enabling memory recovery. Another node can apply a control dependency edge to another subgraph node. The result of applying the control dependency edge is that the recovered data is delayed until the destination node that received the recovered data needs it, and until the controlling node to which the control dependency edge was applied has completed its execution. Once the node to which the control dependency edge was applied has completed its execution, the CPU can send the data to be recovered to the GPU to the destination node.

[0069] In one embodiment, the GPU may also fractionally compress its data to provide additional memory space for performing desired functions or computations without sending any data to the CPU. The GPU may compress data from 32 bits to 16 bits to create the additional memory space. In other illustrative embodiments, the GPU may compress its data at different ratios (i.e., from 32 bits to 24 bits, or from 32 bits to 8 bits). Subgraph nodes apply control dependency edges to nodes that receive data from the source node. The GPU can then complete the execution of its data compression. The node is serialized and sends the received data after the GPU has finished compressing its data. Control dependency edges may delay other subgraph nodes from decompressing the data until those nodes have completed their execution. The GPU cannot receive decompressed data until the node to which the control dependency edge was applied has completed its execution.

[0070] As used herein, the term data flow graph may be defined according to one of the following exemplary and non-restrictive definitions.

[0071] Definition 1: A data flow graph is a pair (N, E), where N is a set of nodes and E is a set of hyperedges. Nodes implement operators on tensors. Each node is a triple (f, I, O), where f is a function of the operator implemented by the node, I is a set of node inputs corresponding to the operator's arguments, and O is a set of node outputs corresponding to the tensors computed by the operator. Hyperedges define how tensors are passed from operators that compute tensors to operators that use tensors as arguments. Each hyperedge is a pair (s, H), where s is the node output and H is a set of node inputs. s and H = {h1, h2, ...} are the source and slot sets of the hyperedge, respectively. Hypergraphs differ from regular graphs. Edges in a regular graph always have a sink. Hyperedges can have multiple sinks: the same tensor is used as arguments by multiple operators. Hypergraph nodes have a more complex structure than nodes in a regular graph. Different operator arguments and results can have different meanings, sizes, and dimensions. Therefore, the inputs and outputs of hypergraph nodes can be identified and uniquely labeled. Nodes in a regular graph can only describe operators with a single argument and a single result. It is convenient to discuss the source and slot nodes of hyperedges, in addition to defining the source and slot as node inputs and outputs. They are defined as follows.

[0072] Definition 2: The nodes corresponding to the source and slot of a hyperedge are called the source and slot nodes of the hyperedge, respectively.

[0073] Definition 3: A superedge connected to the input of a node is called the input superedge of that node, and a superedge connected to the output of that node is called the output superedge of that node.

[0074] Definition 4: The successor input of a node is the slot of the output superedge of that node.

[0075] Definition 5: The successor node of a node is the slot node of the output superedge of that node. Note that we distinguish between the successor input and the successor node. The former is the node input, and the latter is the node itself.

[0076] Accordingly, one or more embodiments provide methods for implementing memory management. Various embodiments provide a method for implementing memory management in a deep learning system.

[0077] For clarity of description and without implying any limitation thereof, some exemplary configurations are used to describe illustrative embodiments. Based on this disclosure, those skilled in the art will be able to conceive of many changes, adaptations, and modifications to the described configurations to achieve the described purposes, and the same changes, adaptations, and modifications are conceivable within the scope of the illustrative embodiments.

[0078] Furthermore, simplified diagrams of the data processing environment are used in the accompanying drawings and illustrative embodiments. In a real computing environment, other structures or components not shown or described herein, or structures or components that function similarly to those shown but described herein, may exist without departing from the scope of the illustrative embodiments.

[0079] Furthermore, illustrative embodiments are described merely as examples, with respect to specific actual or hypothetical components. The steps described in the different illustrative embodiments can be adapted using various components that can be purposed or modified to provide the described functionality within a data processing environment, and such adaptations are contemplated within the scope of the illustrative embodiments.

[0080] The illustrative embodiments described are merely examples of certain types of steps, applications, and data processing environments. Any particular manifestation of these and other similar products is not intended to limit the invention. Any suitable manifestation of these and other similar products may be chosen within the scope of the illustrative embodiments.

[0081] The examples in this disclosure are for clarity of description only and are not intended to limit the illustrative embodiments. Any advantages listed herein are merely examples and are not intended to limit these illustrative embodiments. Additional or different advantages may be achieved through specific illustrative embodiments. Furthermore, certain illustrative embodiments may or may not have some or all of the advantages listed above.

[0082] Please refer to the attached diagram for details. Figure 1 and Figure 2 These figures are example diagrams of a data processing environment that can implement illustrative embodiments. Figure 1 and 2 This is merely an example and is not intended to assert or imply any limitation regarding the environment in which different embodiments may be implemented. Specific implementations may be modified in many ways based on the environment depicted in the following description.

[0083] Figure 1 A block diagram of a data processing system network in which illustrative embodiments can be implemented is shown. Data processing environment 100 is a computer network in which illustrative embodiments can be implemented. Data processing environment 100 includes network 102. Network 102 is a medium for providing communication links between different devices and computers connected together within data processing environment 100. Network 102 may include connections such as wired, wireless communication links, or fiber optic cables.

[0084] The client or server is merely an example role of certain data processing systems connected to network 102 and is not intended to exclude other configurations or roles of these data processing systems. Server 104 is coupled to network 102. Software applications can execute on any quantum data processing system in data processing environment 100. (The diagram shows...) Figure 1 Any software application executing in server 104 can be configured to execute in another data processing system in a similar manner. Figure 1 Any data or information stored or generated in server 104 can be stored or generated in another data processing system in a similar manner. A classic data processing system such as server 104 may contain data and may have software applications or software tools that perform classic computational processes thereon.

[0085] Server 106 is coupled to network 102 along with storage unit 108. Storage unit 108 includes neural network training data configured to store data as described herein with respect to different embodiments. Server 106 is a conventional data processing system. Software applications can execute on any quantum data processing system in data processing environment 100. Any software application described can be configured to execute in a similar manner on another data processing system.

[0086] Clients 110, 112, and 114 are also coupled to network 102. A conventional data processing system, such as server 106 or clients 110, 112, or 114, may contain data and may have software applications or software tools on which conventional computational processes are performed.

[0087] This is merely an example and does not imply any limitations on such an architecture. Figure 1 Certain components available in the example implementations of the embodiments are depicted. For example, server 106 and clients 110, 112, and 114 are depicted as servers and clients only by way of example and are not intended to imply a limitation on the client-server architecture. As another example, the embodiments may be distributed across several conventional data processing systems, quantum data processing systems, and data networks shown, while within the scope of the illustrative embodiments, another embodiment may be implemented on a single conventional data processing system or a single quantum data processing system. Conventional data processing systems 106, 110, 112, and 114 also represent example nodes in clusters, partitions, and other configurations suitable for implementing the embodiments.

[0088] Device 132 is an example of a conventional computing device described herein. For example, device 132 may take the form of a smartphone, tablet, laptop, fixed or portable client 110, wearable computing device, or any other suitable device. Figure 1 Any software application running in another conventional data processing system can be configured to run in device 132 in a similar manner. Figure 1 Any data or information stored or generated in another conventional data processing system can be configured to be stored or generated in device 132 in a similar manner.

[0089] Server 106, storage unit 108, classical processing system 104, quantum processing system 140, and clients 110, 112, and 114, as well as device 132, can be connected to network 102 using wired connections, wireless communication protocols, or other suitable data connections. Clients 110, 112, and 114 can be, for example, personal computers or network computers.

[0090] In the depicted example, server 106 can provide clients 110, 112, and 114 with data such as boot files, operating system images, and applications. In this example, clients 110, 112, and 114 can be clients of server 106. Clients 110, 112, 114, or some combination thereof, can include their own data, boot files, operating system images, and applications. Data processing environment 100 can include additional servers, clients, and other devices not shown.

[0091] Data processing environment 100 may include additional memory, a quantum processor, and other devices not shown. In the depicted example, data processing environment 100 may be the Internet. Network 102 may represent a collection of networks and gateways that communicate with each other using Transmission Control Protocol / Internet Protocol (TCP / IP) and other protocols. The core of the Internet is the skeleton of data communication links between master nodes or master computers (including thousands of commercial, government, educational, and other computer systems that route data and messages). Of course, data processing environment 100 can also be implemented as several different types of networks, such as, for example, intranets, local area networks (LANs), or wide area networks (WANs). Figure 1 This is intended as an example, not as an architectural limitation for different illustrative embodiments.

[0092] Among other uses, the data processing environment 100 can be used to implement a client-server environment in which illustrative embodiments can be implemented. The client-server environment enables software applications and data to be distributed across a network, allowing applications to function by interacting with a conventional client data processing system and a conventional server data processing system. The data processing environment 100 can also employ a service-oriented architecture, where interoperable software components distributed across a network can be packaged together into a consistent business application. The data processing environment 100 can also take the form of a cloud and employ a service-delivered cloud computing model to enable convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services), which can be rapidly provisioned and released with minimal management effort or interaction with the service provider.

[0093] See Figure 2The figure illustrates a block diagram of a data processing system in which illustrative embodiments may be implemented. The data processing system 200 is an example of a conventional computer, such as... Figure 1 The classic processing system 104, server 106, or client 110, 112, and 114, or another type of device in which computer-usable program code or instructions for implementing the illustrative embodiments may reside.

[0094] Data processing system 200 also represents a conventional data processing system or a configuration therein, in which computer-usable program code or instructions for implementing the processing of the illustrative embodiments may be located. Data processing system 200 is described as a computer by way of example only, but is not limited thereto. Implementations in other device forms may modify data processing system 200, such as by adding a touch interface, or even by removing certain depicted components from data processing system 200, without departing from the general description of the operation and function of data processing system 200 described herein.

[0095] In the depicted example, the data processing system 200 employs a central architecture including a Northbridge and Memory Controller Center (NB / MCH) 202 and a Southbridge and Input / Output (I / O) Controller Center (SB / ICH) 204. A processing unit 206, main memory 208, and a graphics processor 210 are coupled to the Northbridge and Memory Controller Center (NB / MCH) 202. The processing unit 206 may contain one or more processors and may be implemented using one or more heterogeneous processor systems. The processing unit 206 may be a multi-core processor. In some implementations, the graphics processor 210 may be coupled to the NB / MCH 202 via an Accelerated Graphics Port (AGP).

[0096] In the depicted example, a local area network (LAN) adapter 212 is coupled to the Southbridge and I / O controller hub (SB / ICH) 204. An audio adapter 216, a keyboard and mouse adapter 220, a modem 222, a read-only memory (ROM) 224, a universal serial bus (USB) and other ports 232, and a PCI / PCIe device 234 are coupled to the Southbridge and I / O controller hub 204 via bus 238. A hard disk drive (HDD) or solid-state drive (SSD) 226 and a CD-ROM 230 are coupled to the Southbridge and I / O controller hub 204 via bus 240. The PCI / PCIe device 234 may include, for example, an Ethernet adapter, an insert card, and a PC card for a notebook computer. PCI uses a card bus controller, while PCIe does not. The ROM 224 may be, for example, a flash binary input / output system (BIOS). Hard disk drive 226 and CD-ROM 230 can use, for example, integrated drive electronics (IDE), Serial Advanced Technology Attachment (SATA) interface, or variants such as external SATA (eSATA) and micro SATA (mSATA). Super I / O (SIO) device 236 can be coupled to the southbridge and I / O controller hub (SB / ICH) 204 via bus 238.

[0097] Memory such as main memory 208, ROM 224, or flash memory (not shown) are some examples of computer-usable storage devices. Hard disk drives or solid-state drives 226, CD-ROMs 230, and other similar available devices are some examples of computer-usable storage devices that include computer-usable storage media.

[0098] The operating system runs on processing unit 206. The operating system coordinates and provides... Figure 2 The data processing system 200 controls various components within it. The operating system can be a commercially available operating system for any type of computing platform, including but not limited to server systems, personal computers, and mobile devices. Object-oriented or other types of programming systems can operate in conjunction with the operating system and provide calls to the operating system from programs or applications executing on the data processing system 200.

[0099] Operating system, object-oriented programming system, and instructions for applications or programs, such as code 226A on hard disk drive 226, reside on a storage device and can be loaded into at least one of one or more memories (such as main memory 208) for execution by processing unit 206. Processing in exemplary embodiments can be executed by processing unit 206 using computer-implemented instructions that may reside in memory, such as, for example, main memory 208, read-only memory 224, or one or more peripheral devices.

[0100] Furthermore, in one scenario, code 226A can be downloaded from a remote system 201B via network 201A, where similar code 201C is stored on storage device 201D. In another scenario, code 226A can be downloaded to a remote system 201B via network 201A, where the downloaded code 201C is stored on storage device 201D.

[0101] Figures 1-2 The hardware within can vary depending on the implementation method. (Except for or replacing...) Figures 1-2 The hardware described herein can be replaced with other internal hardware or peripheral devices, such as flash memory, equivalent non-volatile memory, or optical disc drives. Furthermore, the processes of the illustrative embodiments can be applied to multiprocessor data processing systems.

[0102] In some illustrative examples, the data processing system 200 may be a personal digital assistant (PDA), which is typically configured with flash memory to provide non-volatile memory for storing operating system files and / or user-generated data. The bus system may include one or more buses, such as a system bus, I / O bus, and PCI bus. Of course, the bus system can be implemented using any type of communication structure or architecture that provides data transfer between different components or devices attached to the structure or architecture.

[0103] The communication unit may include one or more means for sending and receiving data, such as a modem or network adapter. Memory may be, for example, main memory 208 or a cache, such as the cache found in the Northbridge and memory controller center 202. The processing unit may contain one or more processors or CPUs.

[0104] Figures 1-2 The examples depicted and those described above are not intended to imply architectural limitations. For example, the data processing system 200 could take the form of a tablet, laptop, or telephone device, in addition to being a mobile or wearable device.

[0105] When a computer or data processing system is described as a virtual machine, virtual device, or virtual component, the virtual machine, virtual device, or virtual component operates using virtualized representations of some or all of the components depicted in the data processing system 200, in a manner consistent with data processing system 200. For example, in a virtual machine, virtual device, or virtual component, processing unit 206 represents a virtualized instance of all or some of the hardware processing units 206 available in the host data processing system. Main memory 208 represents a virtualized instance of all or some portions of main memory 208 available in the host data processing system. Disk 226 represents a virtualized instance of all or some portions of disk 226 available in the host data processing system. In this case, the host data processing system is represented by data processing system 200.

[0106] See Figure 3 This diagram illustrates a data flow diagram with data transfer between nodes within the data processing system 200. Figure 3 It is shown that there are no modifications to the memory-saving features proposed by the present invention as will be described in the following figures. Figure 3 The nodes shown are some of the nodes that can be encountered in the practice of neural networks. In other embodiments, the data flow graph may consist of more than 1,000 nodes. Figure 3 In the diagrams described below, nodes take data items (tensors) as input, manipulate the data, and produce new data at their outputs. A node can execute whenever all its input data is available. Nodes that produce output data have already executed accordingly. Source node 300 can receive data and can send its output data to multiple nodes within the data processing system 200. For example, source node 300 can send its output data to a second node 302. Second node 302 can also send its output data to a third node 304. Further, third node 304 sends its output data to a fourth node 306. Fourth node 306 can send the output data it receives from third node 304 to a fifth node 308. Source node 300 also sends its output data to fifth node 308 and destination node 310. Fifth node 308 also sends its output data received from fourth node 306 and source node 300 to destination node 310. The process of data transfer between nodes in the data flow diagram can be repeated as needed. Nodes sending data have completion times. Completion time refers to the time it takes for data computation to be completed at a node and made available to other nodes. Relaxation time refers to the amount of time an input can be delayed without causing overall operational delay, or the amount of time data must wait at a node's input before it can be used for computation.

[0107] See Figure 4 The figure shows a diagram with... Figure 3A similar data flow diagram. Additionally... Figure 4 Memory shrinking and memory recovery subgraph insertion are introduced. The GPU within the data processing system 200 may often lack the necessary memory to perform certain computations or functions requiring additional memory space. As a result, the GPU may, for example, move data occupying the required memory space to the CPU. Memory shrinking node 301 illustrates the GPU transferring data to the CPU to perform necessary functions or computations requiring additional memory space. A second node 302 can execute immediately after node 300, allocating storage space for its own output data, while simultaneously, node 301 may be busy copying data to CPU memory. After node 301 completes its copy and releases the memory held by its input tensor, nothing prevents subgraph node 312 from immediately beginning its own memory recovery operation, thus bringing the data just saved from the CPU back to the GPU.

[0108] See also Figure 4 In the Figure 4 During the second call to the memory shrink / restore subgraph on the right, the same series of events occurs, involving the third node 304, subgraph nodes 311 and 322, and the fourth node 306. The third node 304 sends its output data to the subgraph node 311. The subgraph node 311 transfers the data to the subgraph node 322. The subgraph node 322 transfers the data to the fourth node 306. The fourth node 306 sends its output data to the fifth node 308. The fifth node 308 receives data from the subgraph node 312. The fifth node 308 sends its output data to the destination node 310. In parallel execution, while the fifth node 308 is sending its output data to the destination node 310, the subgraph node 312 continues to transfer data to the destination node 310.

[0109] See Figure 5 It has been determined that the use of data at nodes 416 and 418 is not urgent, and therefore the memory reduction action latency can be mitigated by the memory represented by subgraph node 401. This action requires a corresponding action in the form of a memory recovery action represented by subgraph node 413. The intention of the insertion subgraph including subgraph nodes 401 and 413 is to relieve the GPU memory of the need to store the output tensor data of node 400 for a long time. Once node 402 consumes the tensor, it is not needed until node 416 is executed. There are also three additional provisions that constitute part of the insertion subgraph, namely, additional control dependency edges marked with dashed arrows in the data flow graph with the words "serialization" and "prefetch". The first is the edge from subgraph node 401 to node 402.

[0110] See also Figure 5The control dependency edge from node A to node B means that the execution of node B depends on node A. In this embodiment, node B must wait until node A has completed its operation before it can begin its operation. Therefore, the control dependency edge can be viewed as the data output from node A to the input of the subordinate node B, where no actual data is transmitted, but instead a signal of completion is sent. The reason for inserting the control dependency edge from subgraph node 401 to the second node 402 is to ensure that the execution of node 402 is delayed until the memory reduction of subgraph node 401 has been completed. Otherwise, node 402 would execute immediately after the source node 400 has executed and would allocate additional GPU memory for its own output tensor. This is avoided by the "serialization" edge from subgraph node 401 to node 402. Subgraph node 401 and node 402 are ensured to execute serially. Subgraph node 401 will complete its execution before node 402. Node 402 can send its output data to node 404. Node 404 can send its output data to nodes 406 and 414. Subgraph nodes 411 and 408 serve the same function as subgraph nodes 401 and 402.

[0111] See also Figure 5 Regarding memory recovery node 413, unless otherwise specified otherwise, it will execute immediately after subgraph node 401 completes execution. However, this would defeat the purpose of memory reduction. In an embodiment, the execution of subgraph node 413 can be forcibly delayed until nodes (nodes 416 and 418) that require data as input are ready to execute. Therefore, control dependency edges are needed as input to subgraph node 413, causing subgraph node 413 to wait for the completion of the control node. The control node cannot be nodes 416 or 418, as additional dependency edges would create disallowed cycles in the graph. The control node should be some node upstream of node 416. Node 410 sends its output data to node 412. Node 412 sends its output data to node 414. Node 414 is selected accordingly to apply the "prefetch" edge to subgraph node 413. Once node 414 completes its operation, it outputs its tensor to node 416 and simultaneously signals subgraph node 413 via a "prefetch" edge, notifying subgraph node 413 that it can begin its recovery action. The initiation of the recovery action should ensure that, when it is complete, the nodes awaiting data recovery are ready to execute, meaning that other input data to those nodes has arrived.

[0112] about Figure 5 , Figure 5The diagram shows the third control dependency edge from node 412 to subgraph node 413 (the second control dependency edge is labeled "serialization"). It provides a second condition for subgraph node 413 to begin its recovery action. The purpose is to serialize subsequent recovery actions, which in this case involve the recovery actions of subgraph nodes 423 and 413. Subgraph node 423 recovers the tensor consumed by node 412. Once node 412 has completed its execution (and therefore the memory used for its input tensors has been freed), it allows subgraph node 413 to begin its execution, but only if all other inputs are equally available. Node 413 must wait a longer period for its "prefetch" signal to arrive from node 414. Once received, subgraph node 413 recovers its tensor and sends its data to nodes 416 and 418. Node 416 now has all inputs available, so it executes, producing the output tensor that node 418 needs as its input. Node 418 now also has all input data available, so it executes.

[0113] See also Figure 5 The two control dependencies for node 413 play different roles. The "prefetch" dependency ensures that subgraph node 413 does not immediately recover the data reduced by node 401. There will be a delay until the nodes waiting for the recovered data are ready at their other inputs. The "serialization" dependency ensures that two recoveries do not occur simultaneously, as doing so would cause all recovered tensors to use excessive memory concurrently. In the illustrative embodiment, it can be said that the "prefetch" signal dominates the "serialization" signal because the "prefetch" is initiated later than the "serialization". In other illustrative embodiments, the "serialization" signal may dominate the "prefetch" signal.

[0114] See Figure 6 This illustrates an embodiment where the GPU moves data to the CPU to free up memory space to perform one or more functions or computations. (See example...) Figure 5 As shown, it utilizes serialization and prefetching.

[0115] See also Figure 6The diagram illustrates specific calls in the insertion subgraph responsible for memory shrinking and restoring operations. Here, a memory shrinking operation is represented by a single node, whose operation is to copy its input tensor from GPU memory to CPU memory. Data swapped out from GPU memory to CPU memory can be an swap-out operation. The GPU memory initially holding the data may now be released after the copying operation has been completed. Node 501, representing this operation, is marked "Copy to CPU". The corresponding operation is established by the restoring node 507, marked "Copy from CPU". Its purpose is to copy the tensor from CPU memory back to GPU memory. Before copying begins, GPU memory must be allocated to hold the restored tensor. Afterward, the CPU memory holding that tensor can be released.

[0116] See Figure 6 Source node 500 receives data and sends it to node 501 and second node 502. Memory reduction begins when node 501 (the control node) inserts a control dependency edge (shown as "serialization" with a dashed line and arrow) onto second node 502 or a slave node. Execution of second node 502 must wait until node 501 completes its execution. The execution of nodes 501 and second node 502 is serialized accordingly. Second node 502 transmits its output data to another node. Another node (not shown) can apply a control dependency edge (shown as "serialization") to node 507 to ensure that node 507 performs the transmission of the data to be recovered after the other node has performed the transmission of its output data. Third node 504 can send its output data to fourth node 506. Fourth node 506 can apply a control dependency edge (shown as "prefetch") to node 507. Thus, node 507 will be delayed in transmitting the data to be recovered to the GPU until fourth node 506 has already sent its output data to destination node 508. After the execution of the fourth node 506, node 507 sends the data to the destination node 508.

[0117] See Figure 7 This illustrates another data flow diagram illustrating data transfer and memory management. The GPU can decide more than just to move data to the CPU to free up memory space needed to perform one or more large functions or computations. The GPU can also decide to compress data proportionally (i.e., from 32-bit to 16-bit, from 32-bit to 8-bit, etc.) to obtain the necessary memory space without moving the data to the CPU. In other illustrative embodiments, the GPU can compress its data in various ways. In this illustrative embodiment, the GPU compresses large tensors of data from 32-bit floating-point numbers to 16-bit floating-point numbers (see [reference]). Figure 11 In other words, a GPU can compress data from 32 bits to 16 bits, freeing up memory space to perform functions and / or computations.

[0118] See also Figure 7 Source node 600 receives input data and sends it to node 601 and second node 602. Node 601 is the controlling node applying the control dependency edge (“serialization”), and second node 602 is the subordinate node. Second node 602 waits until node 601 completes its execution. Second node 602 sends its output data after node 601 has compressed the data to free up memory space. Another node (not shown) may apply the control dependency edge (“serialization”) on node 606. Node 606 waits to decompress the data until other nodes have completed their execution. Third node 609 receives data and sends its output data to fourth node 605. Fourth node 605 may apply the control dependency edge (“prefetch”) on subgraph node 606. Node 606 waits to decompress and recover the data until fourth node 605 has completed its own execution and sent its output data to destination node 607. After fourth node 605's execution, node 606 may complete its execution. The data can be decompressed and sent to the destination node 607.

[0119] about Figure 8 The GPU can compress data and send the compressed data to the CPU to free up extra memory space. Memory shrinking can be performed where the GPU compresses data and sends it to the CPU before another node sends it to the next node. Memory recovery can be performed where the CPU delays decompressing the data and sending it back to the GPU until the destination node needs the data.

[0120] Still about Figure 8 Source node 700 receives data and sends it to node 701 and second node 702. Node 701 applies control dependency edges to second node 702. Execution of second node 702 must wait until node 701 has compressed the data, sent the compressed data, and copied the compressed data to the CPU. Node 707 is part of the recovery structure. The execution of subgraph nodes 701 and second node 702 is serialized accordingly. The GPU can use additional memory space to perform one or more functions or computations. Second node 702 can send its output data to the next node (not shown). A node (not shown) can apply control dependency edges to node 707, which contains compressed data on the CPU. In this embodiment, node 707 must wait to copy the compressed data back to the GPU until the control dependency is activated by another node. After the execution of another node is complete, node 707 completes its execution of copying the compressed data back to the GPU.

[0121] See Figure 8The third node 704 receives the data and transmits its output data to the fourth node 706. The fourth node 706 executes a control dependency edge on node 707. Node 707 waits to copy the data back and sends the data to the node marked "Decompress to FP32". Once both of its control dependency inputs are satisfied, node 707 copies the data from CPU memory back to GPU memory and sends the data to be decompressed. After decompression, the recovered data is made available to node 708.

[0122] about Figure 9 There may be situations where recovery actions should be replicated to satisfy different destination nodes. Although the memory reduction action represented by node 801 is single, experimental data indicates that in this case, having two separate data recovery actions represented by nodes 813 and 807 is beneficial. In the illustrative embodiment, the GPU can retrieve the same data from the CPU multiple times as needed to free up additional memory space for computation and / or other functions.

[0123] See Figure 9 Source node 800 sends data to node 801 and second node 802. Serialization is implemented by adding control dependency edges from node 801 to second node 802. Node 801 copies the data to CPU memory. The execution of nodes 801 and second node 802 is serialized, with second node 802 sending its output data to the next node (not shown) after node 801 has executed. The figure further illustrates an example of two memory recovery subgraphs in the form of nodes 807 and 813. Each of the two nodes 807 and 813 has two incoming control dependencies that establish the serialization and prefetching rules as described in the previous figures.

[0124] See also Figure 9 The third node 804 receives data and sends its output data to the fourth node 806. The fourth node 806 can apply control dependency edges to node 807. Node 807 waits to retrieve data from the CPU and send it to the destination node 808 until the fourth node 806 has completed its execution. Node 807 then sends the recovered data to the destination node 808.

[0125] about Figure 9The same recovery scenario applies to nodes 813, 814, 812, and 810. After node 813 (not shown) performs a concurrent signaling notification of the "serialization" edge and node 812 also performs a concurrent signaling notification of the "prefetch" edge, the same data stored in the CPU memory by the execution of node 801 is brought back to the GPU once its control dependency is satisfied. The fifth node 810 can also receive data and send its output data to the sixth node 812. The sixth node 812 is the source of the "prefetch" control dependency for node 813. The recovery of CPU data to the GPU is delayed until the execution of the sixth node 812 has completed. Accordingly, node 813 sends the data to the destination node 814.

[0126] about Figure 10A The diagram illustrates a flowchart of an example process 1100 for determining the insertion point for shrinking and restoring operations. Method 1100 can be executed using a processor (e.g., in conjunction with...) stored... Figure 1 and 2 The programming instructions on the tangible storage medium are used in practice.

[0127] In this embodiment, some user-configurable information can be assumed. First, a threshold thTTS can be defined for the timing relaxation (TTS) of tensors; only tensors with relaxation values ​​greater than the threshold will be considered as candidates for insertion points. Second, a threshold thS can be defined for the size of tensors; only tensors (e.g., in bytes) with a size exceeding the threshold will be considered as candidates for insertion points. Third, a maximum value N can be defined for the number of insertions to be considered.

[0128] Based on these assumptions, in box 1102, tensor timing relaxation (TTS) is computed for a set of input tensors (e.g., all input tensors of all data flow graph nodes) through timing analysis.

[0129] In box 1104, edit the candidate list SI of the input tensors for the data flow graph node, where the TTS value is greater than thTTS and the size of the tensor is greater than thS.

[0130] In decision box 1106, determine whether the size of the candidate list SI is greater than N.

[0131] In box 1108 (the "Yes" branch of decision box 1106), the candidate list SI is sorted in descending order based on tensor size. Only the first N elements in the list are retained, and other tensors are removed from the candidate list SI. Other sorting or thresholding mechanisms can also be used.

[0132] In box 1110 (the "No" branch of decision box 1106 or the completion of box 1108), return to candidate list S1 as a list of candidate nodes for insertion points of memory reduction or memory recovery operations according to embodiments of the invention described in conjunction with other figures.

[0133] about Figure 10B The diagram illustrates a flowchart of an example procedure 1200 for performing timing analysis. Method 12000 can execute data stored in a processor (e.g., in conjunction with...) Figure 1 and 2 The programming instructions on the tangible storage medium are used in practice.

[0134] See now Figure 10B Procedure 1200 begins according to the illustrated embodiment. In step 1202, procedure 1200 initializes the TAT, TRT, and TTS values ​​of all data flow graph nodes to unknown values. In step 1204, procedure 1200 defines the dataset Q as follows: Let Q be the set of all data flow graph source nodes. Procedure 1200 also sets the TAT, TRT, and TTS values ​​of the elements of Q to zero (0).

[0135] In step 1206, process 1200 checks whether Q is empty. If yes, the process ends (step 1218). However, if no (i.e., when Q is not empty), process 1200 selects (step 1208) any element q of Q and excludes q from Q.

[0136] In step 1210, process 1200 determines whether all slot inputs s of q have been processed. If yes, process 1200 considers the next q (the check performed in step 1206). However, if no, process 1200 processes s as provided in step 1212, whereby process 1200 sets the TAT of s to be equal to the TAT of node q. Process 1200 further (in step 1212) defines n as the node to which the node inputs s belong. Process 1200 determines (in step 1214) whether all inputs i of node n have known TAT values. If no, process 1200 considers the next s and iterates by a while loop to process the remaining slot inputs s of q as identified in step 1210.

[0137] However, if so (as determined in step 1214), process 1200 continues to step 1216, wherein process 1200 sets the TRT of node n to the maximum value of the TAT of its input; sets the TTS of each input I to the difference between the TRT of n and the TAT of i; sets the TAT of node n to its TRT, decrements the corresponding node delay; and includes n in Q.

[0138] If Q is empty (determined in step 1206), process 1200 ends (step 1218), thus the TAT, TRT, and TTS values ​​of all nodes have been determined through the above steps.

[0139] In an embodiment, a timed analysis is performed, as described in process 1200, traversing the data flow graph from source to slot, and TAT, TRT, and TSS are calculated according to the definitions of TAT, TRT, and TSS implemented through this process. This calculation is performed in a time-linear manner within the size of the data flow graph.

[0140] See now Figure 11 The diagram illustrates a schematic representation of the GPU's memory as described in the illustrative embodiments above. The GPU in the illustrative embodiment has 16 gigabytes (GB) of memory. In other illustrative embodiments, the GPU may have a larger memory space. As the GPU performs functions and computations, more and more memory is allocated to the intended functions and computations until a peak is reached where no memory space remains. Once the peak is reached, memory is released, with the most recently allocated memory being released first.

[0141] See Figure 12 This illustrates a schematic diagram of the GPU's memory as described in the above illustrative embodiment. When the GPU decides to compress the data within it as described above... Figure 7 and 8 As shown, when additional memory space is created, large data tensors within the GPU are compressed from 32-bit floating-point numbers to 16-bit floating-point numbers, or from 32 bits to 16 bits. Therefore, half of the memory space becomes idle, and the total data batch size in memory may double before the GPU's memory reaches its peak.

[0142] See Figure 13 A flowchart of an example process 1000 for performing different functions according to an embodiment is shown. At block 1002, by identifying the memory requirements of the output data to be generated by node P, the subgraph nodes A and P (e.g., in...) are... Figure 5In the context of this, node A can refer to subgraph 401, and node P can refer to node 402. A memory reduction action is applied to make this determination. In box 1004, the source node sends data to subgraph nodes A and P. In box 1006, subgraph node A applies a control dependency edge to node P. The execution of subgraph nodes A and P is serialized. Subgraph node A executes before node P. Here, the operation of subgraph node A is to copy its input tensor from GPU memory to CPU memory. Data swapped out from GPU memory to CPU memory can be an swap-out action. The GPU memory that initially held this data can now be released after the copying action is complete. The GPU can use the additional memory space to perform one or more functions or computations. In box 1008, node P sends its output data to node Q. In box 1010, node Q applies a control dependency edge to subgraph node B. This control dependency edge ensures that subgraph node B delays its memory recovery action until node Q has completed its execution. There will be a delay until the nodes waiting for recovery data are ready at their other inputs. In box 1012, node Q sends its output data to the destination node. Further, in box 1014, subgraph node B sends the recovered data to the destination node.

[0143] Some embodiments of the invention will now be described more generally. In the following description, reference may be made directly to one or more of the accompanying drawings as needed, but it should be understood that the embodiments are not limited to any particular drawing.

[0144] According to embodiments of the present invention, a method is provided for processing a neural network dataflow graph comprising (e.g., as defined above in Definitions 1-5) a set of nodes and a set of edges. For example, in... Figure 10A and 10B Perform the steps of this method as described in [the document] to generate a data flow graph, or, for example, combine [the method with other methods]. Figure 3-9 The data flow graph is modified as described. The steps of this method are stored in a computer system such as... Figure 1-2 The physical storage devices described (e.g., as computer program products).

[0145] The method computes a tensor timing relaxation (TTS) of a set of input tensors; edits a candidate list (SI) of input tensors in the set of input tensors with corresponding TTS values ​​greater than a threshold (thTTS); filters the SIs to retain input tensors whose size satisfies a threshold (thS); and uses the filtered SIs to determine the insertion point of the operation.

[0146] According to an embodiment of the method, the operation is a combination of a shrinking operation and a recovery operation. The shrinking operation includes either a copying operation to the central processing unit (CPU) memory or a compression operation. The recovery operation includes either a copying operation from the CPU memory or a decompression operation.

[0147] According to an embodiment of the method, calculating the TTS involves performing timing analysis using an input tensor.

[0148] According to an embodiment of the method, timing analysis includes: initializing the tensor arrival time (TAT), tensor demand time (TRT), and tensor timing relaxation (TTS) values ​​of the input tensors; for a set of input tensors Q, when Q is not empty, performing the following steps: selecting nodes q to be excluded in Q; excluding q from Q; and for each subsequent input tensor s of q, setting the TAT of s to be equal to the TAT of q.

[0149] Preferably, the initialization includes: initializing the TAT, TRT, and TTS values ​​to unknown values ​​or zero.

[0150] According to an embodiment of the method, all inputs to node n have known TAT values, where n is a node for which s is an input tensor, and wherein the method further includes performing the following steps for a set of input tensors Q: setting the TRT of n to the maximum value of the TAT of n's input s; setting the TTS of each input i of n to the difference between the TRT of n and the TAT of s; and including n back in Q.

[0151] According to an embodiment of the method, the data flow graph includes a pair (N, E), where N includes a set of nodes and E includes a set of hyperedges, and where a given node in N implements one or more operators on one or more tensors.

[0152] According to an embodiment of the method, the given node comprises a triple (f, I, O), where f is a function of an operator implemented by the given node, I is a set of input tensors of the given node, and O is a set of outputs of the given node generated based on the operator.

[0153] According to an embodiment of the method, the hyperedge defines how a tensor is passed from the operator that generates the tensor to the operator that uses the tensor as an argument.

[0154] According to an embodiment of the method, the hyperedge comprises a pair (s, H), where s is a node output and H is a set of node inputs.

[0155] According to an embodiment of the method, the method further includes inserting a subgraph node corresponding to the operation into the data flow graph.

[0156] According to an embodiment of the method, the insertion generates a new data flow graph or modifies an existing data flow graph.

[0157] According to an embodiment of the method, the insertion generates a new data flow graph, wherein the new data flow graph includes a complete data flow graph or a partial data flow graph.

[0158] According to an embodiment of the method, the method further includes: processing the set of input tensors using a data flow graph based on the insertion; and generating an output based on the processing.

[0159] According to an embodiment of the method, the operation is a memory reduction operation, and the method further includes: inserting a memory reduction subgraph node corresponding to the memory reduction operation.

[0160] According to an embodiment of the method, the insertion further includes: connecting a first node to a memory-reduced subgraph node via a first hyperedge, the first node corresponding to a source node; and connecting a memory-reduced subgraph node to a second node via a second hyperedge, wherein the second hyperedge includes a serialized hyperedge, and the second node corresponds to an intermediate node or a destination node.

[0161] According to an embodiment of the method, the insertion further includes connecting the first node to the second node via a third hyperedge.

[0162] According to an embodiment of the method, the operation is a memory recovery operation, and the method further includes: inserting a memory recovery subgraph node corresponding to the memory recovery operation.

[0163] According to an embodiment of the method, the insertion further includes: connecting a first node to a memory recovery subgraph node via a first superedge, wherein the second superedge includes a serialization superedge or a prefetch superedge; connecting the memory recovery subgraph node to a second node via the second superedge, the second node corresponding to the destination node; and connecting a memory reduction subgraph node to the memory recovery subgraph node via a third superedge.

[0164] According to an embodiment of the method, the method further includes connecting two nodes of a data flow graph via a hyperedge, wherein the connection includes any of the following: a direct connection via a hyperedge between the two nodes; and an indirect connection via one or more additional nodes and a hyperedge between the two nodes.

[0165] According to an embodiment of the method, the steps of the method are performed iteratively to insert a set of subgraph nodes for performing at least one memory shrinking operation and at least one memory recovery operation into the data flow graph.

[0166] According to an embodiment of the method, iteratively performing the steps of the method includes inserting a set of superedges into the data flow graph, the set of superedges including at least one serialized superedge, or at least one prefetched superedge, or at least one serialized superedge and at least one prefetched superedge.

[0167] According to another embodiment of the present invention, a method is provided. This method selects the insertion point of an operation in a data flow graph based on a candidate list of tensor timing relaxations (TTS) and input tensors (SI).

[0168] In an embodiment, the method further includes: calculating a tensor timing relaxation (TTS) for a set of input tensors; editing a candidate list (SI) of input tensors in the set of input tensors using input tensors with corresponding TTS values ​​greater than a threshold (thTTS); filtering the SIs to retain input tensors whose size satisfies a threshold (thS); and using the filtered SIs to determine an insertion point for shrinking and restoring operations.

[0169] In an embodiment, calculating the TTS includes performing timing analysis using the input tensor.

[0170] In an embodiment, the method further includes inserting a subgraph node corresponding to the operation into the data flow graph.

[0171] In an embodiment, the insertion generates a new data flow graph or modifies an existing data flow graph.

[0172] In one embodiment, the insertion generates a new data flow graph, wherein the new data flow graph includes a complete data flow graph or a partial data flow graph.

[0173] In one embodiment, the method further includes: processing the set of input tensors using a data flow graph based on the insertion; and generating an output based on the processing.

[0174] According to an embodiment of the present invention, a method is provided. The method includes: determining, by identifying memory requirements for output data to be generated by node P, to apply a memory reduction action to subgraph nodes A and P; sending data from a source node to subgraph nodes A and P; enabling subgraph node A to perform the memory reduction action before execution of subgraph node P, the memory reduction action making additional memory space available for a graphics processing unit (GPU) to execute one or more functions; delaying a memory recovery action for subgraph node B until node Q has completed its execution; and sending the data from the central processing unit (CPU) of subgraph node B to the GPU to recover the data to the GPU.

[0175] In one embodiment, the method further includes applying control dependency edges to subgraph node P so that subgraph node A can be executed before subgraph node P.

[0176] In an embodiment, the method further includes determining the insertion position of the control dependency edge by identifying a node P that generates output data with a memory requirement greater than that of the output data generated by node Q.

[0177] In one embodiment, the method includes applying control dependency edges to subgraph node B to delay the memory recovery action until node Q has completed its execution.

[0178] In one embodiment, the method includes delaying the execution of node P until the memory reduction action is complete.

[0179] In one embodiment, the method includes the GPU using additional memory space to perform a series of computations.

[0180] In one embodiment, the method includes sending data from subgraph node B to the destination node after node Q has completed its execution.

[0181] In one embodiment, the method includes compressing data to obtain additional memory space for the GPU to perform one or more functions.

[0182] In one embodiment, the method includes decompressing the data before completing the memory recovery operation.

[0183] In one embodiment, the method includes delaying the memory recovery action until the destination node requires the data.

[0184] In one embodiment, the method includes decompressing the data after node Q has completed its execution.

[0185] In one embodiment, the method includes restoring data to the GPU after execution has been completed on node Q.

[0186] In one embodiment, the method includes determining the insertion position to apply control dependency edges to allow additional memory recovery actions to be performed.

[0187] In one embodiment, the method includes performing additional memory recovery actions at time intervals different from the memory recovery actions between node Q and subgraph node B.

[0188] In one embodiment, the method includes: compressing data to create additional memory space for the GPU to perform one or more functions; copying the compressed data to the CPU before execution at node P; sending the received data from subgraph node P after the CPU receives the compressed data; and using the created additional memory space to perform the one or more functions in the GPU.

[0189] In one embodiment, the method includes determining the time interval required to delay memory recovery actions to prevent the GPU from prematurely receiving data back from the CPU.

[0190] According to another embodiment of the present invention, a computer-usable program product may be provided. The computer program product includes one or more computer-readable storage devices and program instructions stored on at least one of the one or more storage devices, the stored program instructions including program instructions for performing steps of the methods according to embodiments provided in this disclosure.

[0191] According to another embodiment of the present invention, a computer may be provided. The system includes one or more processors, one or more computer-readable memories and one or more computer-readable storage devices, and program instructions stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, the stored program instructions including steps for performing methods according to embodiments provided in this disclosure.

[0192] In the illustrative embodiment, in the data flow graph, tensors computed earlier occupy GPU memory until other tensors required for node execution are ready. Prematurely computed tensors can be moved to larger CPU memory, compressed, or even removed from GPU memory. When these tensors are needed again, they are returned to GPU memory, decompressed, or recomputed. The more tensors involved in memory storage, the greater the memory savings and the larger the data flow graph that can be processed within the same GPU memory.

[0193] In one embodiment, a method determines to apply a memory reduction action to subgraph nodes A and P by identifying the memory requirements of the output data generated by node P. The method sends data from the source node to subgraph nodes A and P. The method enables subgraph node A to perform a memory reduction action before the execution of node P, the memory reduction action making additional memory space available for the graphics processing unit (GPU) to execute one or more functions. The method causes subgraph node B to delay a memory recovery action until node Q has completed its execution. The method sends data from the central processing unit (CPU) of subgraph node B to the GPU to recover the data to the GPU.

[0194] In one embodiment, the method applies control dependency edges to node P so that subgraph node A can be executed before node P.

[0195] In an embodiment, the method determines the insertion position of the control dependency edge by identifying the node P that generates output data with a memory requirement greater than that generated by node Q.

[0196] In one embodiment, the method applies control dependency edges to subgraph node B to delay memory recovery actions until node Q has completed its execution.

[0197] In this embodiment, the method delays the execution of node P until the memory reduction action is completed.

[0198] In this embodiment, the GPU uses additional memory space to perform a series of calculations.

[0199] In one embodiment, the method sends data from subgraph node B to the destination node after node Q completes its execution.

[0200] In one embodiment, the method compresses data to obtain additional memory space for the GPU to perform one or more functions.

[0201] In one embodiment, the method decompresses the data before completing the memory recovery operation.

[0202] In this embodiment, the method delays the memory recovery action until the destination node needs the data.

[0203] In this embodiment, the method decompresses the data after node Q has completed its execution.

[0204] In one embodiment, the method restores the data to the GPU after node Q has completed its execution.

[0205] In one embodiment, the method determines the insertion position to apply control dependency edges to allow additional memory recovery actions to be performed.

[0206] In one embodiment, the method performs an additional memory recovery action at a time interval different from the memory recovery action between node Q and subgraph node B.

[0207] In one embodiment, the method compresses data to create additional memory space for the GPU to perform one or more functions. The method copies the compressed data to the CPU before execution at node P. After the CPU receives the compressed data, the method sends the received data from node P. The method uses the created additional memory space to execute the one or more functions in the GPU.

[0208] In an embodiment, the method determines the time interval required to delay memory recovery actions to prevent the GPU from prematurely receiving data back from the CPU.

[0209] In one embodiment, a computer-usable program product is provided, including one or more computer-readable storage devices, and program instructions stored on at least one of the one or more storage devices. The stored program instructions include instructions for performing the following operations: determining to apply a memory reduction action to subgraph nodes A and P by identifying memory requirements for output data to be generated by node P; sending data from a source node to subgraph nodes A and P; enabling subgraph node A to perform the memory reduction action prior to the execution of subgraph node P, the memory reduction action making additional memory space available for a graphics processing unit (GPU) to execute one or more functions; causing subgraph node B to delay a memory recovery action until node Q has completed its execution; and sending the data from the central processing unit (CPU) of subgraph node B to the GPU to recover the data to the GPU.

[0210] In an embodiment, the computer system includes one or more processors, one or more computer-readable memories, and one or more computer-readable storage devices, and program instructions stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories. The stored program instructions include instructions for performing the following operations: determining to apply a memory reduction action to subgraph nodes A and P by identifying memory requirements for output data to be generated by node P; sending data from the source node to subgraph nodes A and P; enabling subgraph node A to perform the memory reduction action before the execution of subgraph node P, the memory reduction action making additional memory space available for the graphics processing unit (GPU) to execute one or more functions; delaying a memory recovery action for subgraph node B until node Q has completed its execution; and sending the data from the central processing unit (CPU) of subgraph node B to the GPU to recover the data to the GPU.

[0211] The following definitions and abbreviations will be used to interpret the claims and description. As used herein, the terms “comprising,” “including,” “containing,” “having,” “having,” or “containing,” or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a composition, mixture, process, method, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such composition, mixture, process, method, article, or apparatus.

[0212] Furthermore, the term "illustrative" is used herein to mean "serving as an example, illustration, or illustration." Any embodiment or design described herein as "illustrative" is not necessarily to be construed as preferred or advantageous over other embodiments or designs. The terms "at least one" and "one or more" should be understood to include any integer greater than or equal to one, i.e., one, two, three, four, etc. The term "multiple" should be understood to include any integer greater than or equal to two, i.e., two, three, four, five, etc. The term "connection" can include both indirect "connection" and direct "connection."

[0213] References to "an embodiment," "embodiment," "exemplary embodiment," etc., in the specification indicate that the described embodiment may include a particular feature, structure, or characteristic; however, each embodiment may or may not include a particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same implementation. Moreover, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is believed that the influence of other embodiments (whether explicitly described or not) on such feature, structure, or characteristic is within the knowledge of those skilled in the art.

[0214] The terms “about,” “substantially,” “roughly,” and their variations are intended to include the degree of error associated with a measurement based on a specific quantity of equipment available at the time of application submission. For example, “about” could include a range of ±8%, 5%, or 2% of a given value.

[0215] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements over those found in the market, or to enable those skilled in the art to understand the embodiments described herein.

[0216] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements over those found in the market, or to enable those skilled in the art to understand the embodiments described herein.

[0217] Therefore, illustrative embodiments provide computer-implemented methods for managing participation and other related features, functions, or operations in online communities. When embodiments or portions thereof are described with respect to a class of devices, computer-implemented methods, systems, or devices, computer program products, or portions thereof are adapted or configured for a representation suitable and comparable to that class of devices.

[0218] Users can access the application through a variety of client devices using thin client interfaces such as web browsers (e.g., web-based email) or other lightweight client applications. Users do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, or storage of the cloud infrastructure.

[0219] This invention can be a method, apparatus, and / or computer program product at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to execute aspects of the invention.

[0220] Computer-readable storage media can be tangible means for retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or protrusions in slots having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0221] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or to an external computer or external storage device. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0222] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by utilizing state information from the computer-readable program instructions to personalize the electronic circuitry in order to perform aspects of this invention.

[0223] The present invention has been described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0224] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions comprises an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0225] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, such that the instructions executed on the computer, other programmable apparatus, or other device perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0226] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than indicated in the figures. For example, depending on the functions involved, two consecutively shown blocks may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

Claims

1. A method for reducing overall GPU memory usage while processing a neural network, the method comprising: automatically determining an insertion point for a memory saving operation in a base dataflow graph comprising a set of nodes and a set of hyper-edges, wherein the automatically determining an insertion point comprises: computing tensor timing slack (TTS) values for a set of input tensors by performing a timing analysis; editing a candidate list (SI) of input tensors in the set of input tensors using input tensors having respective TTS values greater than a threshold value (thTTS); filtering the SI to retain input tensors whose size satisfies a threshold value (thS); and determining an insertion point for the operation using the filtered SI.

2. The method of claim 1, wherein, the operation is a combination of a shrink operation and a restore operation, wherein the shrink operation comprises either or both of a copy operation to a central processing unit (CPU) memory and a compression operation, and wherein the restore operation comprises either or both of a copy operation from the CPU memory and a decompression operation.

3. The method of claim 1, wherein, the timing analysis comprises: initializing tensor arrival time (TAT), tensor required time (TRT), and tensor timing slack (TTS) values for the input tensors; for a set of input tensors Q, when Q is not empty, performing the following steps: selecting a node q to exclude in Q; excluding q from Q; and for each successor input tensor s of q, setting the TAT of s equal to the TAT of q.

4. The method of claim 3, wherein, the initializing comprises: initializing the TAT, TRT, and TTS values to unknown values for all graph nodes, and initializing the TAT, TRT, and TTS values to zero for all graph inputs collected in Q.

5. The method of claim 3, wherein, all inputs of a node n have known TAT values, where n is a node for which s is an input tensor, and wherein the method further comprises performing the following further steps for the set of input tensors Q: setting the TRT of n equal to the maximum of the TATs of the inputs i of n; setting the TTS of each input i of n equal to the difference between the TRT of n and the TAT of s; setting the TAT of n based on the TRT of n incremented by a node delay; and including n back into Q.

6. The method of claim 1, wherein, a dataflow graph comprises a pair N and E, where N comprises a set of nodes, E comprises a set of hyper-edges, and where a given node in N implements one or more operators on one or more tensors.

7. The method of claim 6, wherein, the given node comprises a triple f, I, and O, where f is a function of an operator implemented by the given node, I is a set of input tensors of the given node, and O is a set of outputs of the given node generated based on the operator.

8. The method of claim 6, wherein, a hyper-edge defines how a tensor is passed from an operator that generates the tensor to an operator that uses the tensor as an argument.

9. The method of claim 6, wherein, a hyper-edge comprises a pair s and H, where s is a node output, and H is a set of node inputs.

10. The method of claim 1, further comprising: inserting a subgraph node corresponding to the operation in the dataflow graph.

11. The method of claim 10, wherein, the inserting generates a new dataflow graph or modifies an existing dataflow graph.

12. The method of claim 11, wherein, the inserting generates a new dataflow graph, wherein the new dataflow graph comprises a complete dataflow graph or a partial dataflow graph.

13. The method of claim 10, further comprising: processing the set of input tensors using a dataflow graph based on the insertion; and generating an output based on the processing.

14. The method of claim 1, wherein, the operation is a memory reduction operation, the method further comprising: inserting a memory reduction subgraph node corresponding to the memory reduction operation.

15. The method of claim 10, wherein, the insertion further comprises: connecting a first node to the memory reduction subgraph node by a first hyperedge, the first node corresponding to a source node; and connecting the memory reduction subgraph node to a second node by a second hyperedge, the second hyperedge comprising a serialization hyperedge, and the second node corresponding to an intermediate node or a destination node.

16. The method of claim 10, wherein, the insertion further comprises: connecting the first node to the second node by a third hyperedge.

17. The method of claim 1, wherein, the operation is a memory restoration operation, the method further comprising: inserting a memory restoration subgraph node corresponding to the memory restoration operation.

18. The method of claim 17, wherein, the insertion further comprises: connecting a first node to the memory restoration subgraph node by a first hyperedge, the second hyperedge comprising a serialization hyperedge or a prefetch hyperedge; connecting the memory restoration subgraph node to a second node by a second hyperedge, the second node corresponding to a destination node; and connecting the memory reduction subgraph node to the memory restoration subgraph node by a third hyperedge.

19. The method of claim 1, further comprising connecting two nodes of the hyperedge by a dataflow graph, wherein, the connection comprises any of: a direct connection through the hyperedge between the two nodes; and an indirect connection through one or more additional nodes and hyperedges between the two nodes. the steps of the method are iteratively performed to insert a set of subgraph nodes for performing at least one memory reduction operation and at least one memory restoration operation into the dataflow graph.

20. The method of claim 1, wherein, iteratively performing the steps of the method comprises inserting a set of hyperedges into the dataflow graph, the set of hyperedges comprising at least one serialization hyperedge, or at least one prefetch hyperedge, or at least one serialization hyperedge and at least one prefetch hyperedge.

21. The method of claim 20, wherein, the operation is a memory reduction action, the method comprising:

22. The method of claim 1, wherein, determining to apply a memory reduction action with subgraph node A and subgraph node P by identifying a memory requirement for output data to be produced by node P; sending data from a source node to subgraph node A and subgraph node P; enabling subgraph node A to perform a memory reduction action prior to execution of node P, the memory reduction action making additional memory space available to a graphics processing unit (GPU) to perform one or more functions; delaying a memory restoration action by subgraph node B until node Q has completed its execution; and sending the data from a central processing unit (CPU) of subgraph node B to the GPU to restore the data to the GPU.

23. A method comprising: selecting an insertion point of an operation in a dataflow graph based on a set of tensor timing slack (TTS) and a candidate list SI of input tensors; computing a TTS of a set of input tensors; editing the candidate list SI of input tensors in the set of input tensors using input tensors having a respective TTS value greater than a threshold value thTTS; filtering the SI to retain input tensors whose size satisfies a threshold value thS; and ​ ​ determining an insertion point for the operation using the filtered SI.

24. A method comprising: determining to apply a memory reduction action with subgraph node A and node P by identifying memory requirements of output data to be produced by node P; sending data from a source node to subgraph node A and node P; enabling subgraph node A to perform the memory reduction action prior to execution of node P, the memory reduction action making additional memory space available to a graphics processing unit (GPU) to perform one or more functions; delaying a memory restoration action with subgraph node B until node Q has completed its execution; and sending the data from a central processing unit (CPU) of subgraph node B to the GPU to restore the data to the GPU.

25. A computer program product comprising instructions for performing the method of any one of claims 1 to 24.

26. A system for reducing overall GPU memory usage while processing a neural network, the system operable to: automatically determine an insertion point for a memory saving operation in a base dataflow graph comprising a set of nodes and a set of edges; automatically determining the insertion point comprises: wherein, computing tensor timing slack (TTS) for a set of input tensors; editing a candidate list SI of input tensors in the set of input tensors using input tensors having respective TTS values greater than a threshold value thTTS; filtering the SI to retain input tensors whose size satisfies a threshold value thS; and determining the insertion point for the operation using the filtered SI.

27. A system operable to: select an insertion point for an operation in a dataflow graph based on a set of tensor timing slacks (TTS) and a candidate list SI of input tensors; computing TTS for a set of input tensors; editing a candidate list SI of input tensors in the set of input tensors using input tensors having respective TTS values greater than a threshold value thTTS; filtering the SI to retain input tensors whose size satisfies a threshold value thS; and determining an insertion point for the operation using the filtered SI.

28. A system operable to: determining to apply a memory reduction action with subgraph node A and node P by identifying memory requirements of output data to be produced by node P; sending data from a source node to subgraph node A and node P; enabling subgraph node A to perform the memory reduction action prior to execution of node P, the memory reduction action making additional memory space available to a graphics processing unit (GPU) to perform one or more functions; delaying a memory restoration action with subgraph node B until node Q has completed its execution; and sending the data from a central processing unit (CPU) of subgraph node B to the GPU to restore the data to the GPU. ​ ​ ​

Images