GPU memory optimization method and system for DNN training under limited resources

Abstract

The invention discloses a GPU memory optimization method and system for DNN training under limited resources, and the method comprises the steps: S1, carrying out the traversal selection of a current operator op from an operator sequence O of an input DNN model, and skipping to S2 if the traversal is not completed; otherwise, ending and exiting; s2, executing the current operator op, and recording runtime information of a tensor for executing the current operator op; s3, if the memory usage amount of the GPU is larger than or equal to the preset threshold value, tensors in part of the GPU memory are selected, and the selected tensors are directly expelled or exchanged to the CPU memory to release the GPU memory; and skipping to the step S1. The invention aims to realize GPU memory optimization oriented to DNN training under limited resources, so that the same GPU can support a larger DNN model and a larger training batch, meanwhile, the influence on the training efficiency is reduced, and the overhead during operation is reduced.

Description

Technical Field

[0001] The present invention relates to the fields of computer edge computing and artificial intelligence, and in particular to a GPU memory optimization method and system for DNN (deep neural network) training under limited resources. Background Art

[0002] Driven by massive data, huge computing power and advanced algorithms, artificial intelligence technologies represented by deep learning have achieved great success in computer vision, natural language processing, recommendation systems and other fields. In order to improve the performance of deep neural networks (DNNs), researchers use increasingly larger data sets to train deeper and wider DNN models. In recent years, the parameter scale of basic models has grown at an exponential rate, and this trend has posed a severe challenge to the memory resources of computing devices. The training process of DNN models involves a large number of high-dimensional matrix operations, which is a computationally intensive task and is usually processed using dedicated acceleration chips such as GPUs and TPUs. However, the memory capacity of a single GPU only grows at a linear rate and cannot meet the needs of large-scale model training. The gap between the memory requirements for DNN model training and the available GPU memory limits researchers' exploration of larger-scale and more powerful DNN models.

[0003] In order to train larger models with limited memory resources, researchers usually use smaller training batches, but this may not fully utilize the parallel computing capabilities of the GPU, which in turn affects the throughput of model training. In addition, larger training batches usually help the model to obtain better generalization capabilities during training. At the algorithmic level, model compression algorithms use the redundancy of model parameters and network structures to reduce the memory requirements of the compressed model. However, model compression algorithms usually affect the accuracy and convergence of the model and require the adjustment of a large number of hyperparameters. At the system level, distributed training alleviates the problem of insufficient GPU memory by increasing the number of computing devices. However, distributed training is costly and requires solving frequent communication and synchronization problems between multiple GPUs. In addition, memory allocation optimization uses tensor information to find the optimal memory allocation solution through heuristic methods.

[0004] Memory swapping and recomputation are techniques for memory optimization for a single GPU at the system level. Memory swapping unloads temporarily unneeded activation values ​​to CPU memory and reloads them to GPU memory when needed. Recomputation releases temporarily unneeded activation values ​​directly from GPU memory and recomputes them when needed. There are currently some works exploring methods that combine memory swapping and recomputation, but most of the work is based on static computation graphs. However, users need to predefine the complete network structure under static computation graphs, which is not convenient for debugging. Compared with static computation graphs, dynamic computation graphs can obtain the calculation results of each layer instantly, which is convenient for debugging. However, dynamic computation graphs cannot obtain complete computation graph information in advance. How to use the information obtained at runtime to make decisions and formulate memory release strategies for appropriate tensors at the right time is challenging. Summary of the invention

[0005] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a GPU memory optimization method and system for DNN training under limited resources are provided. The present invention aims to achieve GPU memory optimization for DNN training under limited resources, so that the same GPU can support larger DNN models and larger training batches, while reducing the impact on training efficiency and reducing runtime overhead.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: A GPU memory optimization method for DNN training under limited resources, comprising: S1, traverse and select a current operator op from the operator sequence O of the input DNN model. If the traversal has not been completed, jump to step S2; otherwise, end and exit; S2, execute the current operator op and record the runtime information of the tensor executing the current operator op; S3, if the memory usage of the GPU is greater than or equal to a preset threshold, select tensors in part of the GPU memory, and directly evict or swap the selected tensors to the CPU memory to release the GPU memory; jump to step S1.

[0007] Optionally, the runtime information of the current operator op on the tensor recorded in step S2 includes: the computation time of recalculating the tensor t , the time to exchange tensor t between GPU memory and CPU memory , the memory size that can be obtained in the memory pool by releasing the tensor t , and the time that tensor t has not been used recently 、The memory size occupied by the underlying storage of tensor t and the computation time of the operator that produces the tensor t , and there are: , , In the above formula, is the memory size occupied by the underlying storage of tensor t, is the bandwidth between GPU memory and CPU memory, is the computation time of the operator that produces the tensor t; is the eviction neighborhood tensor of tensor t; To generate a tensor The computation time of the operator, where the tensor is the eviction neighborhood tensor of tensor t is used to generate the tensor t or the tensor generated by the tensor t.

[0008] Optionally, selecting a portion of the tensors in the GPU memory in step S3 includes: calculating the selection cost of the tensors in the GPU memory using a heuristic valuation function shown in the following formula: , In the above formula, is the selection cost of tensor t, is the computation time of recalculating the tensor t, is the time to exchange tensor t between GPU memory and CPU memory, To release the memory size that tensor t can obtain in the memory pool, is the time that tensor t has not been used recently; then one or more tensors with the smallest cost are selected as the selected tensors.

[0009] Optionally, in step S3, directly evicting or swapping the selected tensor to the CPU memory to release the GPU memory includes respectively calculating the cost of directly evicting or swapping the selected tensor to the CPU memory, and selecting a method with the minimum cost to release the GPU memory.

[0010] Optionally, respectively calculating the cost of directly evicting or exchanging the selected tensors to the CPU memory and selecting a method with the lowest cost to release the GPU memory includes: calculating the release cost of the selected tensor using a heuristic valuation function shown in the following formula: , In the above formula, is the release cost of tensor t, is the computation time of recalculating the tensor t, The time when tensor t is swapped between GPU memory and CPU memory; if the release cost of tensor t is greater than 1, tensor t is swapped to CPU memory to release GPU memory; otherwise, tensor t is directly evicted to release GPU memory.

[0011] Optionally, before executing the current operator op in step S2, the tensor required for the operation of the current operator op is checked: S2.1, traverse and select a current tensor t from the tensor set inputs required for the current operator op operation. If the traversal has not been completed, jump to step S2.2; otherwise, execute the current operator op and jump to step S3; S2.2, determine whether the current tensor t is in the GPU memory. If the previous tensor t is in the GPU memory, jump to step S2.1; otherwise, determine the reason why the current tensor t is not in the GPU memory. If it is not in the GPU memory due to swapping to the CPU memory, swap the current tensor t from the CPU memory to the GPU memory; if it is not in the GPU memory due to being directly evicted, recalculate the current tensor t to make it enter the GPU memory; jump to step S2.1.

[0012] In addition, the present invention also provides a GPU memory optimization system for DNN training under limited resources, which is used to apply the GPU memory optimization method for DNN training under limited resources described above, comprising: The operator executor is used to traverse and select the current operator op from the operator sequence O of the input DNN model; Tensor monitor, which is used to record the runtime information of tensors used by the operator executor to execute the current operator op and the GPU memory usage; A tensor selector is used to select tensors in part of the GPU memory when the GPU memory usage is greater than or equal to a preset threshold; A policy maker for evicting or swapping selected tensors directly to CPU memory to free up GPU memory; The operator executor, tensor monitor, tensor selector and strategy maker are connected in sequence.

[0013] In addition, the present invention also provides a GPU memory optimization system for DNN training under limited resources, including a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the GPU memory optimization method for DNN training under limited resources.

[0014] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the GPU memory optimization method for DNN training under limited resources through a processor.

[0015] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the GPU memory optimization method for DNN training under limited resources through a processor.

[0016] Compared with the prior art, the present invention mainly has the following advantages: The GPU memory optimization method for DNN training under limited resources of the present invention uses the spatial locality and temporal locality information of tensors recorded by the runtime system in a similar way to the cache hit algorithm to selectively evict tensors in the GPU memory from the GPU memory or swap them out to the CPU memory to reduce the peak memory during the DNN model training phase. The present invention does not need to obtain complete computational graph information in advance, and runs completely transparently to the user, and can be applied in scenarios of DNN model training based on dynamic computational graphs. Compared with native deep learning frameworks, the present invention can support larger training batches, so that the same GPU can support larger DNN models and larger training batches, while reducing the impact on training efficiency and runtime overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of the basic flow of the method of the embodiment of the present invention.

[0018] Figure 2 Schematic diagram of runtime information of a tensor in an embodiment of the present invention.

[0019] Figure 3 This is an example of dynamic tensor exchange and recalculation in an embodiment of the present invention.

[0020] Figure 4 Schematic diagram of the system structure in an embodiment of the present invention.

[0021] Figure 5 Schematic diagram of two strategies for releasing GPU memory formulated by a strategy maker in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to achieve GPU memory optimization for DNN training under limited resources, the present invention proposes a GPU memory optimization method (Dynamic Optimization of GPU Memory, referred to as DOME) for DNN training under limited resources. Figure 1 As shown, the GPU memory optimization method for DNN training under limited resources in this embodiment includes: S1, traverse and select a current operator op from the operator sequence O of the input DNN model. If the traversal has not been completed, jump to step S2; otherwise, end and exit; S2, execute the current operator op and record the runtime information of the tensor executing the current operator op; S3, if the memory usage of the GPU is greater than or equal to a preset threshold, select tensors in part of the GPU memory, and directly evict or swap the selected tensors to the CPU memory to release the GPU memory; jump to step S1.

[0023] DNN model training includes two stages: forward propagation and back propagation. The feature maps accumulated on the forward propagation path are the main consumer of GPU memory. Memory swapping and recalculation utilize the first-in-last-out lifecycle characteristics of feature maps. By releasing the GPU memory occupied by feature maps, the instantaneous memory demand of deep learning training is lower than the GPU memory capacity, thereby ensuring that the DNN model can be trained. The training of the DNN model can be regarded as a series of operations on tensors by operators. In this embodiment, the tensor monitor is responsible for tracking and recording the information of tensors generated by operator calculations. The runtime information of the current operator op on the tensor recorded in step S2 of this embodiment includes: the calculation time of the tensor t recalculated , the time to exchange tensor t between GPU memory and CPU memory , the memory size that can be obtained in the memory pool by releasing the tensor t , the time that tensor t has not been used recently 、The memory size occupied by the underlying storage of tensor t and the computation time of the operator that produces the tensor t , and there are: , , In the above formula, is the memory size occupied by the underlying storage of tensor t, is the bandwidth between GPU memory and CPU memory, is the computation time of the operator that produces the tensor t; is the eviction neighborhood tensor of tensor t; To generate a tensor The computation time of the operator, where the tensor is the eviction neighborhood tensor of tensor t is used to generate the tensor t or the tensor generated by the tensor t. Figure 2 : is a schematic diagram of the runtime information of tensors in this embodiment, where solid circles represent tensors in GPU memory, and dotted circles represent tensors not in GPU memory. Figure 2 As shown in (a), the tensor of this embodiment , , In GPU memory, tensors and tensors , are allocated to different memory segments, tensors , The memory size of the underlying storage of the tensor , As shown in the figure, release the tensor , The amount of memory available in the memory pool and As shown in the figure. Figure 2 In (b), the tensor , In GPU memory, For tensors Time not recently used, To generate a tensor The tensor selector selects the tensor to be released at the moment of the rightmost dashed line in the figure.

[0024] In order to reduce the instantaneous memory demand of the GPU for model training, it is necessary to selectively evict or swap some tensors in the GPU memory directly to the CPU memory. The method of this embodiment (DOME) draws on the ideas of cache replacement algorithm and dynamic tensor recalculation. When the memory demand of model training is greater than the threshold of GPU memory capacity, the tensor selector uses a heuristic valuation function to calculate the cost of exchanging or recalculating candidate tensors based on the runtime information obtained by the tensor monitor to select the tensor that is most suitable for release at the moment. Specifically, selecting tensors in part of the GPU memory in step S3 of this embodiment includes: using the heuristic valuation function shown in the following formula to calculate the selection cost of the tensor in the GPU memory: , In the above formula, is the selection cost of tensor t, is the computation time of recalculating the tensor t, is the time to exchange tensor t between GPU memory and CPU memory, To release the memory size that tensor t can obtain in the memory pool, is the time that tensor t has not been used recently; then one or more tensors with the lowest cost are selected as the selected tensors. In this embodiment, the above-mentioned tensor selection heuristic valuation function with higher training efficiency is designed for the tensor selector. Using this function, the tensor selector will give priority to tensors with the lowest time cost required to regain, the largest memory pool space that can be actually obtained after release, and will not be needed again in a short time after release. When searching for tensors that can be released, the tensor selector will select the tensor with the lowest release cost calculated by the heuristic valuation function, and consider it to be the best choice at the moment.

[0025] In step S3 of this embodiment, directly expelling or swapping the selected tensor to the CPU memory to release the GPU memory includes respectively calculating the cost of directly expelling or swapping the selected tensor to the CPU memory, and selecting the method with the lowest cost to release the GPU memory. Figure 3 As shown, it is assumed that the GPU memory can accommodate up to 3 tensor nodes. In state 1, the tensor , , In GPU memory, when you need to calculate tensors When the input tensor It is no longer in GPU memory. You need to restore the tensor first. , but at this time the GPU memory has reached the set threshold, and it is necessary to choose to release the tensors that are still resident in the GPU memory. In state 2, according to the algorithm we designed, the tensor The most suitable candidate tensor is selected by the tensor selector and released from GPU memory by swapping out or evicting. Regenerate by recalculation or swapping in. At this point, the GPU memory has reached the set threshold. According to the algorithm we designed, the tensor is selected and released from GPU memory. In state 4, the tensor Successfully received by the input tensor and Calculation generation.

[0026] Memory swapping does not occupy additional computing resources of the GPU, but the overhead of data transmission between the GPU and the CPU is too high. Recalculation can release memory instantly, but it will bring additional computing overhead and the risk of training failure. In this embodiment, a more intuitive strategy formulation heuristic valuation function is designed for the characteristics of dynamic calculation graphs that are interpreted and executed at the same time and cannot obtain complete calculation graph information in advance. Specifically, this embodiment calculates the cost of directly expelling or exchanging the selected tensors to the CPU memory, and selects the method with the lowest cost to release the GPU memory, including: using the heuristic valuation function shown in the following formula to calculate the release cost of the selected tensor: , In the above formula, is the release cost of tensor t, is the computation time of recalculating the tensor t, The time to exchange tensor t between GPU memory and CPU memory; if the release cost of tensor t is greater than 1, tensor t is chosen to be exchanged to CPU memory to release GPU memory; otherwise, tensor t is chosen to be directly evicted to release GPU memory. The policy maker uses the runtime information obtained by the tensor monitor to formulate a memory release policy for the tensor that is most suitable for release at the moment selected by the tensor selector. Specifically, when the overhead of memory exchange is relatively smaller, the policy maker chooses to swap the tensor from GPU memory to CPU memory; when the overhead of recalculation is relatively smaller, the policy maker chooses to directly evict the tensor from the GPU memory. This more efficient decision-making method can minimize runtime overhead.

[0027] In this embodiment, before executing the current operator op in step S2, the tensor required for the operation of the current operator op is checked: S2.1, traverse and select a current tensor t from the tensor set inputs required for the current operator op operation. If the traversal has not been completed, jump to step S2.2; otherwise, execute the current operator op and jump to step S3; S2.2, determine whether the current tensor t is in the GPU memory. If the previous tensor t is in the GPU memory, jump to step S2.1; otherwise, determine the reason why the current tensor t is not in the GPU memory. If it is not in the GPU memory due to swapping to the CPU memory, swap the current tensor t from the CPU memory to the GPU memory; if it is not in the GPU memory due to being directly evicted, recalculate the current tensor t to make it enter the GPU memory; jump to step S2.1.

[0028] like Figure 4 As shown, this embodiment also provides a GPU memory optimization system for DNN training under limited resources for applying the GPU memory optimization method for DNN training under limited resources described above, including: The operator executor is used to traverse and select the current operator op from the operator sequence O of the input DNN model; The tensor monitor is used to record the runtime information of the tensors of the current operator op executed by the operator executor and the memory usage of the GPU; for example Figure 4 In the example, the tensor and tensors For operator Input, tensor For operator Output: A tensor selector is used to select tensors in part of the GPU memory when the GPU memory usage is greater than or equal to a preset threshold; A policy maker for evicting or swapping selected tensors directly to CPU memory to free up GPU memory; The operator executor, tensor monitor, tensor selector and policy maker are connected in sequence. The main functions of each module are described as follows: During the training phase of the model, the operator executor interprets and executes the operator operation according to the user code, which is supported by the deep learning framework. The tensor monitor is responsible for continuously monitoring the GPU memory usage and recording the runtime information generated by the model training. When the memory requirement of the model training exceeds the threshold of the GPU memory capacity, the tensor selector selects the most suitable tensor to be released from the GPU memory according to the runtime information recorded by the tensor monitor. The policy maker selects the way to release the tensor memory from the perspective of minimizing the runtime overhead. Its working process is as follows: First, for each round of iteration of the DNN model training, the operator executor executes the operator operation in the calculation graph, and the tensor monitor records the input tensor of the operator, the operation time of the operator, and the size of the output tensor, and updates the usage time of the output tensor. Then, if the tensor involved in the operation is not in the GPU memory, the operator executor will check the history of the tensor. If the tensor is evicted, it is restored by recalculation. If the tensor is swapped out to the CPU memory, it is restored by swapping. At this time, the tensor monitor updates the usage time of the tensor. After that, when the operation of each operator is completed, the tensor monitor checks whether the memory usage of the current GPU exceeds the memory threshold. If it exceeds, the tensor selection and release operation will begin. Specifically, the tensor selector will add the tensors that are still resident in the GPU to the candidate set, and calculate the release cost of each candidate tensor based on the information recorded by the tensor monitor. Figure 5 (a) shows the tensor in GPU memory Swapping to CPU memory to free up GPU memory, Figure 5 (b) shows the tensor Direct eviction to free up GPU memory, in this case you need to use tensors The tensor recorded by the previous checkpoint Use the recalculation chain to recalculate to get the tensor ,The policy maker module selects a release strategy for the candidate tensor with the lowest release cost by comparing the memory swap and recomputation costs of the tensors. Finally, the tensor t is released and deleted from the candidate set. The above process continues until the memory usage of the current GPU is lower than the memory threshold.

[0029] In summary, the GPU memory optimization method for DNN training under limited resources in this embodiment uses a similar idea to the cache hit algorithm, and utilizes the spatial locality and temporal locality information of tensors recorded by the runtime system to selectively evict tensors in GPU memory from GPU memory or swap them out to CPU memory to reduce the peak memory during the DNN model training phase. This embodiment does not need to obtain complete computational graph information in advance, and runs completely transparently to the user, and can be applied to scenarios of DNN model training based on dynamic computational graphs. Compared with native deep learning frameworks, this embodiment can support larger training batches, so that the same GPU can support larger DNN models and larger training batches, while reducing the impact on training efficiency and reducing runtime overhead.

[0030] In addition, this embodiment also provides a GPU memory optimization system for DNN training under limited resources, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the GPU memory optimization method for DNN training under limited resources.

[0031] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program or instructions, and the computer program or instructions are programmed or configured to execute the GPU memory optimization method for DNN training under limited resources through a processor.

[0032] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the GPU memory optimization method for DNN training under limited resources through a processor.

[0033] Those skilled in the art should understand that the technical solutions provided by the embodiments of the present application may be in the form of methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0034] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A GPU memory optimization method for DNN training under limited resources, characterized in that: include: S1, traverse and select a current operator op from the operator sequence O of the input DNN model. If the traversal has not been completed, jump to step S2; otherwise, end and exit; S2, execute the current operator op and record the runtime information of the tensor executing the current operator op; S3, if the GPU memory usage is greater than or equal to a preset threshold, select some tensors in the GPU memory and directly evict or swap the selected tensors to the CPU memory to free up the GPU memory; Jump to step S1.

2. The GPU memory optimization method for DNN training under limited resources according to claim 1, characterized in that: The runtime information of the current operator op on the tensor recorded in step S2 includes: the computation time of recalculating the tensor t , the time to exchange tensor t between GPU memory and CPU memory , the memory size that can be obtained in the memory pool by releasing the tensor t , the time that tensor t has not been used recently 、The memory size occupied by the underlying storage of tensor t and the computation time of the operator that produces the tensor t , and there are: ， ， In the above formula, is the memory size occupied by the underlying storage of tensor t, is the bandwidth between GPU memory and CPU memory, is the computation time of the operator that produces the tensor t; is the eviction neighborhood tensor of tensor t; To generate a tensor The computation time of the operator, where the tensor is the eviction neighborhood tensor of tensor t is used to generate the tensor t or the tensor generated by the tensor t.

3. The GPU memory optimization method for DNN training under limited resources according to claim 2 is characterized in that: Selecting some tensors in the GPU memory in step S3 includes: calculating the selection cost of the tensors in the GPU memory using the heuristic valuation function shown in the following formula: ， In the above formula, is the selection cost of tensor t, is the computation time of recalculating the tensor t, is the time to exchange tensor t between GPU memory and CPU memory, To release the memory size that tensor t can obtain in the memory pool, is the time that tensor t has not been used recently; then one or more tensors with the smallest cost are selected as the selected tensors.

4. The GPU memory optimization method for DNN training under limited resources according to claim 3 is characterized in that: Directly evicting or swapping the selected tensor to the CPU memory to release the GPU memory in step S3 includes respectively calculating the cost of directly evicting or swapping the selected tensor to the CPU memory, and selecting a method with the minimum cost to release the GPU memory.

5. The GPU memory optimization method for DNN training under limited resources according to claim 4 is characterized in that: The respectively calculating the cost of directly expelling or exchanging the selected tensors to the CPU memory and selecting the method with the minimum cost to release the GPU memory includes: using the heuristic valuation function shown in the following formula to calculate the release cost of the selected tensor: ， In the above formula, is the release cost of tensor t, is the computation time of recalculating the tensor t, The time when tensor t is swapped between GPU memory and CPU memory; if the release cost of tensor t is greater than 1, tensor t is swapped to CPU memory to release GPU memory; otherwise, tensor t is directly evicted to release GPU memory.

6. The GPU memory optimization method for DNN training under limited resources according to claim 1, characterized in that: Before executing the current operator op in step S2, the tensors required for the current operator op are checked: S2.1, traverse and select a current tensor t from the tensor set inputs required for the current operator op operation. If the traversal has not been completed, jump to step S2.2; Otherwise, execute the current operator op and jump to step S3; S2.2, determine whether the current tensor t is in the GPU memory. If the previous tensor t is in the GPU memory, jump to step S2.1; Otherwise, determine the reason why the current tensor t is not in the GPU memory. If it is not in the GPU memory due to swapping to the CPU memory, swap the current tensor t from the CPU memory to the GPU memory; if it is not in the GPU memory due to being directly evicted, recalculate the current tensor t to put it in the GPU memory; Jump to step S2.

1.

7. A GPU memory optimization system for DNN training under limited resources, which is used to apply the GPU memory optimization method for DNN training under limited resources as described in any one of claims 1 to 6, characterized in that: include: The operator executor is used to traverse and select the current operator op from the operator sequence O of the input DNN model; Tensor monitor, which is used to record the runtime information of tensors used by the operator executor to execute the current operator op and the GPU memory usage; A tensor selector is used to select tensors in part of the GPU memory when the GPU memory usage is greater than or equal to a preset threshold; A policy maker for evicting or swapping selected tensors directly to CPU memory to free up GPU memory; The operator executor, tensor monitor, tensor selector and strategy maker are connected in sequence.

8. A GPU memory optimization system for DNN training under limited resources, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the GPU memory optimization method for DNN training under limited resources as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute the GPU memory optimization method for DNN training under limited resources described in any one of claims 1 to 6 through a processor.

10. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute the GPU memory optimization method for DNN training under limited resources described in any one of claims 1 to 6 through a processor.

Images