A method to reduce GPU memory usage during deep neural network training

Abstract

The present invention discloses a method for reducing GPU memory usage during deep neural network training. The method is applicable to a deep neural network (DNN) framework based on a CPU-GPU heterogeneous computing platform, provides three levels of memory optimization options: basic-normal-extreme, realizes zero-overhead memory allocation at runtime through static computational graph analysis, selects appropriate operator recalculation strategies through heuristic search, and analyzes and limits the additional memory overhead of the Cudnn library convolution algorithm. Users only need to control the optimization level of one parameter to achieve memory usage optimization during neural network training.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning and computer science, and more particularly to a method for reducing GPU memory usage during deep neural network training. Background Art

[0002] Deep learning has been widely studied and applied in academia and industry, mainly due to the growth of hardware computing power, especially GPU, which provides powerful parallel floating-point matrix computing capabilities. But even now, GPU out of memory (OOM) is still one of the main problems encountered by most researchers when training neural networks. The main reason is that the number of parameters in the neural network model is very large and is still growing rapidly. No matter what the subject is, it is an unavoidable step to allow network training to run normally. It is not difficult to see that when memory resources are fixed, making improvements at the software level, improving memory usage efficiency, and reducing memory usage during neural network training has considerable research value.

[0003] In the past, the methods to reduce memory usage mainly include:

[0004] 1. Model quantization: Use lower precision data types to represent model parameters;

[0005] 2. Memory sharing: variables with non-overlapping life cycles can share the same memory;

[0006] 3. Dynamic memory pool: Maintain a user-mode memory pool to reduce memory application and release at the CUDA level;

[0007] 4. Computation for memory: some reusable temporary objects in the calculation process are released when not in use in exchange for more memory. The released variables will be recalculated and loaded in the next round of iteration;

[0008] 5. Memory swap: swap temporarily unused variables in the GPU to the CPU, and then swap them back when needed.

[0009] The above methods may reduce the accuracy of the model when executed, or generate fixed strategies based on some prior knowledge. At the same time, they require developers to have professional knowledge in the field of deep learning, and have a high threshold for use. Summary of the invention

[0010] The purpose of the present invention is to provide a method for reducing GPU memory usage during deep neural network training in view of the shortcomings of the prior art. Specifically, it relates to a method for achieving multiple different levels of GPU memory optimization using only one environment variable when the user does not have much theoretical knowledge of deep learning.

[0011] The object of the present invention is to reduce the GPU memory usage during deep neural network training, comprising the following steps: A method for reducing the GPU memory usage during deep neural network training, comprising the following steps:

[0012] The actual memory application plan is calculated based on the checkpoint set;

[0013] It includes the following sub-steps:

[0014] (a) Initialize the reference dependency of the corresponding Tensor according to the number of outputs of each node in the static computation graph:

[0015] (b) Examine each Tensor in topological order;

[0016] (c) Check whether the current free memory block list is not empty. If there is no free memory block, apply for a new memory block for the current Tensor. If there is a free memory block, check whether the ratio of its size to the size of the current Tensor falls within a reasonable range. If there is a free memory block and the ratio of its size to the size of the current Tensor falls within a reasonable range, allocate the free memory block to the current Tensor to avoid new memory application, otherwise apply for new memory space for the Tensor.

[0017] (d) For all inputs of the current Tensor, reduce its reference count, indicating that the current Tensor has completed the calculation and no longer depends on the input Tensor; if there is an object in the input Tensor list whose reference count is reduced to 0, the memory of the Tensor is released and the corresponding memory block is added to the free memory block list;

[0018] (e) Repeat steps (c) to (d) until all Tensors are examined and the final memory application plan is output.

[0019] Furthermore, the reasonable interval in step (c) is a searchable value, which is 1, 2, 4, 8, or 16.

[0020] Furthermore, the checkpoint set is an empty set or is obtained by the following steps:

[0021] (1) Start a trial run during the initial training to collect equipment environment information and the operation status of the neural network layer by layer;

[0022] (2) Checkpoint candidate selection: Based on the information collected in step (1), select appropriate nodes (layers) as candidate points for recalculating checkpoints;

[0023] (3) Select several appropriate nodes from the candidate points given in step (2) as the checkpoints set.

[0024] Furthermore, when the trial run is started during the initial training in step (1), the operator type, memory usage, calculation time, auxiliary space size, and execution time and throughput of the copy operation between the CPU and GPU of each node in the calculation graph are collected.

[0025] Furthermore, when evaluating whether a node is suitable as a recomputation checkpoint in step (2), its survival time, computation time, and feature map size are considered; when evaluating whether a node is suitable for being swapped to the CPU, whether the time interval between two operations that depend on the node is long enough to cover the copy overhead between the CPU and GPU is considered.

[0026] Furthermore, selecting the checkpoints set in step (3) specifically includes the following sub-steps:

[0027] (3.1) Set the initialization threshold to 0 and start n rounds of iterations;

[0028] (3.2) In each iteration, each Tensor in the neural network is examined in topological order; the input of the neural network is used as the first checkpoint, which is also the first Block; at the same time, the parameter variables of the neural network are not added to the Block;

[0029] (3.3) Check whether the current tensor is a candidate point. If not, add it to the current Block directly. If the current tensor is a candidate point, check whether the current Block exceeds the threshold. If it does not exceed the threshold, add the current tensor to the current Block. Otherwise, create a new Block and mark the current tensor as a checkpoint.

[0030] (3.4) Repeat step (3.3) until all Tensors are traversed and the current checkpoint set is obtained. The threshold is updated and memory is requested to obtain the actual memory usage.

[0031] (3.5) Repeat steps (3.2) to (3.4) with the threshold updated in step (3.4) until the search for the predetermined number of iterations n is completed, and output the checkpoint solution that minimizes the actual memory usage.

[0032] Furthermore, the updating threshold in step (3.4) is specifically:

[0033] For each T i At this moment, its memory usage is MemAllocTi They are all maintained at a similar level, so the formula for updating the threshold is as follows:

[0034]

[0035] Furthermore, it also includes limiting the Cudnn auxiliary space according to the additional space occupied by the GPU computing library during actual runtime.

[0036] Furthermore, the process of limiting the Cudnn auxiliary space is specifically as follows:

[0037] According to the extra space occupied by the GPU computing library during actual runtime, the Cudnn auxiliary space is limited using the cudnnGetConvolutionForwardAlgorithm_v7 function and the size of the output feature map of each convolutional layer; the extra space is limited to 20 times the size of the output feature map.

[0038] Furthermore, the additional space limit is preferably 0.05 times the size of the output feature map.

[0039] The beneficial effects of the present invention are: the method proposed by the present invention relies on runtime data for automatic analysis instead of prior knowledge, and therefore has better versatility and robustness; the present invention provides a fine-grained heuristic memory application search scheme, which is more efficient than traditional genetic algorithms or simulated annealing methods; the present invention provides a multi-level memory optimization method, which implements a simple and effective GPU memory saving scheme, which can be combined with mainstream deep learning frameworks to improve GPU memory utilization and reduce training costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flow chart for generating a memory management scheme for the present invention;

[0041] Figure 2 Checkpoint diagram for neural network retraining;

[0042] Figure 3 A flowchart of the heuristic search for the best memory solution of the present invention;

[0043] Figure 4 This is a schematic diagram of the memory allocation process of the present invention;

[0044] Figure 5 The flowchart of the memory application scheme of the present invention is calculated. DETAILED DESCRIPTION

[0045] The present invention will be described in further detail below in conjunction with the accompanying drawings and specific embodiments, and the purpose and effects of the present invention will become more apparent.

[0046] Here are some of the terms used in this article: CPU memory refers to the high-speed memory connected to the CPU north bridge chip through a bus; GPU memory refers to the memory that is separate from the CPU memory and is dedicated to the GPU; checkpoint refers to the Tensor retained in the memory in the recalculation strategy, from which the remaining Tensors are recalculated during backpropagation.

[0047] The present invention proposes a method for reducing GPU memory usage during deep neural network training. The method is applicable to a deep neural network (DNN) framework based on a CPU-GPU heterogeneous computing platform. It realizes zero-overhead memory allocation at runtime through static computational graph analysis, selects appropriate operator recalculation strategies through heuristic search, and analyzes and limits the additional memory overhead of the Cudnn library convolution algorithm, providing three levels of memory optimization options: basic, normal, and extreme. Users only need to control the optimization level of one parameter to achieve memory usage optimization during neural network training. Figure 1 A flow chart of the method created by the present invention is given. The method proposed by the present invention can be well controlled by only using one parameter, and combined with runtime data analysis and heuristic search, to complete the generation of a low-occupancy GPU memory application plan.

[0048] Deep learning is a feature learning method that transforms raw data into an abstract representation through a nonlinear model. This conversion process is not artificial but learned from the data. The deep learning model is usually represented by a deep neural network, which is a nonlinear mathematical model modeled after biological characteristics.

[0049] GPU (Graphics Processing Unit) was originally used mainly for graphics rendering, but due to its high bandwidth, high parallelism, and more registers and cache structures, it is very suitable for matrix operations. In recent years, it has also been widely used in computationally intensive deep neural network calculations.

[0050] There are now complete development tools for using GPUs to train deep neural networks, which greatly reduces the threshold for researchers to use them; however, due to the rapid growth of neural network size, the problem of insufficient GPU memory has increasingly troubled researchers. Therefore, the present invention aims to propose a method for improving GPU memory usage to alleviate the current situation of insufficient GPU memory.

[0051] like Figure 1 As shown, the present invention proposes a method for reducing GPU memory usage during deep neural network training, specifically: obtaining an actual memory application plan based on a checkpoint set calculation.

[0052] To better explain the operation logic and its purpose, refer to Figure 4 The schematic diagram of the memory allocation process shown in the figure shows that the horizontal axis is the logical time and the vertical axis is the cumulative memory usage. It can be seen that there is no mutual dependence between the three variables Tensor 1, Tensor 2, and Tensor 3, so each of them applies for a piece of memory; at time t1, Tensor 1 completes the calculation, and Tensor 4, which depends on its result, starts to calculate. At this time, the free memory blocks are Chunk 1 and Chunk 2. The algorithm of the present invention will arrange them into Chunk 2 which is closest in size. The same is true for Tensor 5 at time t2. It is worth noting that at time t3, the size of Tensor 6 is larger than the free memory blocks Chunk 2 and Chunk 3. At this time, the algorithm of the present invention will expand Chunk 3 because its size is closer to Tensor 6. It should be emphasized that this expansion is not performed at runtime, but through Figure 5 After the algorithm is planned, Chunk 3 will be applied based on the size of Tensor 6 at the beginning. That is, Figure 5 The memory application plan is a simulation of all Tensor memory allocations. After the final plan is generated, actual memory application will be performed to ensure that no unnecessary memory application, release, and expansion operations occur during runtime.

[0053] First, several candidate points are randomly selected as checkpoints, and all other nodes are marked as discardable. After the calculation is completed, the discardable nodes will not be saved in the memory, but will be directly discarded. These nodes are recalculated again by the checkpoints residing in the memory during the back propagation process. In this embodiment of the present invention, the part of the nodes recalculated by checkpointi is called Blocki, so the goal of the heuristic memory application scheme search is:

[0054] argmin(max(Mem Blocki ))for i∈[0,n]

[0055] Where n is the number of checkpoints. That is, among multiple checkpoint selection schemes, find the scheme that returns the minimum memory usage.

[0056] It includes the following sub-steps:

[0057] (a) According to the number of outputs of each node in the static computation graph, initialize the reference dependency of the corresponding Tensor: one output corresponds to one reference; maintain two data structures, the applied memory block and the free memory block, both initialized to empty;

[0058] (b) Each Tensor is examined in topological order, where the parameters (input, weight, etc.) of the deep learning network model do not participate in memory allocation;

[0059] (c) Check whether the current free memory block list is not empty. If there is no free memory block, apply for a new memory block for the current Tensor. If there is a free memory block, check whether the ratio of its size to the size of the current Tensor falls within a reasonable range. If there is a free memory block and the ratio of its size to the size of the current Tensor falls within a reasonable range, allocate the free memory block to the current Tensor to avoid new memory application, otherwise apply for new memory space for the Tensor.

[0060] The reasonable interval is a searchable value; when the value is 1, 2, 4, 8, and 16, a complete memory application scheme inference is performed, and then the scheme that minimizes the total amount of memory applied is selected. If the current Tensor is larger than the current free memory block, then expansion will occur at this time.

[0061] (d) For all inputs of the current Tensor, reduce their reference counts, indicating that the current Tensor has completed the calculation and no longer depends on the input Tensor. If there is an object in the input Tensor list whose reference count is reduced to 0, the memory of the Tensor is released and the corresponding memory block is added to the free memory block list;

[0062] (e) Repeat steps (c) to (d) until all Tensors are examined and the final memory application plan is output.

[0063] The checkpoint set is an empty set or is obtained by the following steps:

[0064] Step (1): Start a trial run during the initial training to collect device environment information and the operation status of the neural network layer by layer to provide necessary information for subsequent algorithm decisions; during the trial run, it is necessary to collect the operator type, memory usage, calculation time, auxiliary space size of each node in the calculation graph, as well as the execution time and throughput of the copy operation between the CPU and GPU.

[0065] In an embodiment of the present invention, a profile tool (including Nvidia Nsight System, Cudaevent API, etc.) is first used to perform a forward-reverse run on the target network, and profile information is collected during this process. The profile information includes the operator (Op) type, life cycle, dependency duration, feature map size information corresponding to the tensor Tensor, and bandwidth information for data transmission between CPU and GPU.

[0066] The collected profile information is mainly for the following three tasks: first, to collect the running information of Tensor to determine which Tensor can be used as candidate checkpoints; second, to collect the bandwidth and rate information of CPU-GPU data transmission to help Tensor overlap between memory exchange and calculation; third, to collect the information of additional memory space occupied by different Cudnn convolution algorithms to prepare for the extreme optimization mode (i.e., limiting the auxiliary space of Cudnn).

[0067] Step (2): Using the profile information collected in step (1), select candidate checkpoints from several tensors.

[0068] The step (2) is equivalent to the rough screening process of selecting checkpoints. When evaluating whether a node is suitable as a candidate point for recalculating the checkpoint, its survival time, calculation time, and feature map size are mainly considered; when evaluating whether a node is suitable for being exchanged to the CPU, the main consideration is whether the time interval between two operations that depend on the node is long enough to cover the copy overhead between the CPU and the GPU.

[0069] In the embodiment of the present invention, first, all input variables and parameter variables, including weights, bias, etc., and temporary variables with a short life cycle are filtered out. Secondly, the selected Tensor mainly considers three factors: first, the calculation time t of the Tensor is considered. com With the Tensor feature map size s fea The larger the ratio, the greater the cost of recalculation, and the more it should be considered as a checkpoint; second, consider the time interval between the two dependencies of the Tensor, which is usually the interval between the forward calculation and the reverse calculation of the Tensor. The longer the interval, the more suitable it is to be swapped to the CPU memory; third, consider the feature map size s of the Tensor itself fea The larger the value, the more likely the Tensor is to be the memory bottleneck in the overall training process and the more worthy it is to be swapped to the CPU memory. Considering that memory swapping is enabled only for extreme optimization, the weight ratio of the normalized scores of the three data mentioned above is 10:1:1, so as to obtain the score ranking of all Tensors, and select the appropriate number of Tensors with the highest scores as candidate checkpoints.

[0070] Step (3): Select several appropriate nodes as checkpoints from the candidate points given in step (2). Based on the static computation graph analysis, the memory application plan and overall memory usage of each node are obtained through heuristic memory search. To better define the operation logic and purpose of step (3), refer to Figure 2 The neural network retraining Checkpoint diagram is shown in Figure 1. In the retraining logic, the complete neural network is divided into several blocks. The first layer in the block is the checkpoint, followed by the tensor that can be discarded. Taking Block 3 as an example, the Conv layer is used as the checkpoint, and the subsequent BN layer, ReLU layer, and MaxPool layer can recycle the memory after the operation is completed; during back propagation, the subsequent 3 layers of data are recalculated from the Conv layer to calculate the gradient. According to the definition, for a certain time T i , the Tensors resident in the memory include all Tensors of the current Block and all CheckpointTensors before the Block. The memory usage at this time is:

[0071]

[0072] Among them, MemAlloc Ti is the memory usage at the i-th moment, MemAlloc Blocki is the memory usage of the corresponding Block at the i-th moment, MemAlloc ckptj is the memory size occupied by the j-th checkpoint, where the value of j ranges from 0 to i-1, that is, all checkpoints before the current Block; i is the subscript number, and n is the total number of all Blocks, which is also the total number of all checkpoints.

[0073] The present invention proposes a heuristic search for the best memory solution, such as Figure 3 As shown in the figure, the purpose of the heuristic search for the best memory solution is to find a combination of Checkpoints that minimizes the maximum value of the total amount of Tensors resident in memory at all times (minimizes the peak value of memory usage), that is:

[0074]

[0075] Among them, checkpoints is a specific set of checkpoints, and formula (2) indicates that this checkpoint set should minimize the corresponding peak memory usage.

[0076] At the extreme level of optimization, if memory swapping is enabled, all checkpoints except the current block are swapped to the CPU memory, and the target formula becomes:

[0077] checkpoints=argmin checkpoints (max(MemAlloc Blocki )),i∈[1,n] (3)

[0078] It should be noted that after enabling memory swapping, according to the above formula, you only need to set all candidate points as checkpoints to obtain the minimum block size, that is, the minimum peak memory usage. However, the premise for this formula to be valid is that all checkpoints other than the current block can be swapped to the CPU and swapped back to the GPU memory when appropriate. This requires ensuring that the computation time of a block is greater than the swap time of adjacent checkpoints, including the time for swapping out and swapping in, plus the time overhead for scheduling and synchronization. Under this constraint, a heuristic search is performed for the best memory solution.

[0079] The step (3) is equivalent to the fine screening process of selecting checkpoints. The search for the best checkpoint set specifically includes the following sub-steps:

[0080] (3.1) Initialize the threshold to 0 and start n rounds of iterations, where n is a custom constant. Usually, n is 5 to 7 to make the threshold converge;

[0081] (3.2) In each iteration, each Tensor in the neural network is examined in topological order. The input (data layer) of the neural network is used as the first checkpoint and also the first Block. At the same time, the parameter variables (weights, bias, etc.) of the neural network are not added to the Block.

[0082] (3.3) Check whether the current tensor is a candidate point. If not, add it to the current Block directly. If the current tensor is a candidate point, check whether the current Block exceeds the threshold. If it does not exceed the threshold, add the current tensor to the current Block. Otherwise, create a new Block and mark the current tensor as a checkpoint.

[0083] (3.4) Repeat step (3.3) until all Tensors are inspected and the current checkpoint set is obtained. The threshold is updated and memory is requested to obtain the actual memory usage.

[0084] The threshold update is specifically as follows: in the embodiment of the present invention, for each Ti At this moment, its memory usage is MemAlloc Ti They are all maintained at a similar level, so the threshold is updated as follows:

[0085]

[0086] Among them, Threshold is the threshold value.

[0087] (3.5) Using the threshold updated in step (3.4), repeat steps (3.2) to (3.4) until the search for the predetermined number of iterations n is completed, and output the checkpoint solution that minimizes the actual memory usage.

[0088] like Figure 1 As shown in FIG. 1 , after several iterations, including the search for the theoretically optimal checkpoint solution and the actual optimal memory allocation solution, the final memory allocation result is obtained.

[0089] Furthermore, the method of the present invention utilizes the cudnnGetConvolutionForwardAlgorithm_v7() function to restrict the Cudnn auxiliary space.

[0090] Cudnn is a deep neural network operator acceleration library designed specifically for GPUs launched by Nvidia, which supports acceleration of operations such as convolution, pooling, and normalization. Cudnn supports the implementation of 7 convolution algorithms, including FFT, GEMM, WINOGRAD, etc. Some of these algorithms require additional GPU memory as a cache for intermediate results.

[0091] The additional space limit of the Cudnn convolution algorithm is determined according to the size of the output feature map of each convolution layer: for basic level optimization, there is no limit; for normal level optimization, the additional space limit is 20 times the size of the output feature map; for extreme level optimization, the additional space limit is 0.05 times the size of the output feature map. For a feature map of size 24.5MB, the Cudnn auxiliary space required for calculation ranges from 0, 0.64MB, 443MB to 2145MB. In order to avoid the situation where GPU memory is insufficient due to Cudnn auxiliary space, step 105 limits the auxiliary space of Cudnn. The choice of limiting threshold comes from Figure 1 The test run information of step 101.

[0092] Cudnn provides the cudnnGetConvolutionForwardAlgorithm_v7() function, which can give the size of the Cudnn auxiliary space required by different algorithms for the convolution operation of the specified input. The present invention uses this as a basis to filter out algorithms that exceed the limit.

[0093] like Figure 1 As shown, the present invention provides three different levels of optimization schemes. In the basic level optimization, only the Tensor life cycle analysis and memory application optimization described in the calculation memory application scheme are enabled to ensure the fastest running speed; in the normal and extreme level optimizations, all optimizations are enabled. Among them, the extreme level optimization enables memory swapping and stricter Cudnn auxiliary space restrictions to achieve maximum memory savings.

[0094] Further, Figure 3 The search direction of the heuristic search for the best memory solution is formula (2)(3), which is a theoretical memory usage value. The actual memory planning needs to be determined by Figure 5 The memory application scheme shown is given. The checkpoint scheme finally output in step (3.5) is the basic mode, that is, Figure 5 The memory application plan with the lowest actual memory application value.

[0095] For all levels of optimization, the memory pool enables analysis of variable lifecycles and their dependencies. For normal and extreme levels of optimization, node discard optimization is enabled to recycle and recalculate non-checkpoint nodes. For extreme level optimization, checkpoints automatic swap optimization is enabled to swap temporarily unused checkpoint nodes into CPU memory.

[0096] The above are only specific implementation methods of the present invention, which cannot be used to limit the scope of the present invention. Equivalent changes made by ordinary technicians in this technical field based on this creation, as well as changes known to technicians in this field, should still fall within the scope of the present invention.

Claims

1. A method for reducing GPU memory usage during deep neural network training, characterized in that: The following steps are included: The actual memory application plan is calculated based on the checkpoint set; It includes the following sub-steps: (a) Initialize the reference dependency of the corresponding Tensor according to the number of outputs of each node in the static computation graph: (b) Examine each Tensor in topological order; (c) Check whether the current free memory block list is not empty. If there is no free memory block, apply for a new memory block for the current Tensor; If there is a free memory block, check whether the ratio of its size to the size of the current Tensor falls within a reasonable range; if there is a free memory block and the ratio of its size to the size of the current Tensor falls within a reasonable range, allocate the free memory block to the current Tensor to avoid new memory requests, otherwise apply for new memory space for the Tensor; (d) For all inputs of the current Tensor, reduce its reference count, indicating that the current Tensor has completed the calculation and no longer depends on the input Tensor; if there is an object in the input Tensor list whose reference count is reduced to 0, the memory of the Tensor is released and the corresponding memory block is added to the free memory block list; (e) Repeat steps (c) to (d) until all Tensors are inspected and the final memory application plan is output; The checkpoint set is an empty set or is obtained by the following steps; (1) Start a trial run during the initial training to collect equipment environment information and the operation status of the neural network layer by layer; (2) Checkpoint candidate selection: Based on the information collected in step (1), select appropriate nodes or layers as candidate points for recalculating checkpoints; (3) Select several appropriate nodes from the candidate points given in step (2) as the checkpoints set; The step (3) of selecting the checkpoints set specifically includes the following sub-steps: (3.1) Set the initialization threshold to 0 and start n rounds of iterations; (3.2) In each iteration, each Tensor in the neural network is examined in topological order; the input of the neural network is used as the first checkpoint, which is also the first Block; at the same time, the parameter variables of the neural network are not added to the Block; (3.3) Check whether the current tensor is a candidate point. If not, add it directly to the current Block; If the current Tensor is a candidate point, check whether the current Block exceeds the threshold: if it does not exceed the threshold, add the current Tensor to the current Block, otherwise create a new Block and mark the current Tensor as a checkpoint; (3.4) Repeat step (3.3) until all Tensors are traversed and the current checkpoint set is obtained. The threshold is updated and memory is requested to obtain the actual memory usage. (3.5) Repeat steps (3.2) to (3.4) with the threshold updated in step (3.4) until the search for the predetermined number of iterations n is completed, and output the checkpoint solution that minimizes the actual memory usage.

2. The method according to claim 1, characterized in that: The reasonable interval in step (c) is a value that can be searched, and takes 1, 2, 4, 8, and 16.

3. The method according to claim 1, characterized in that When the trial run is started during the initial training in step (1), the operator type, memory usage, calculation time, auxiliary space size, and execution time and throughput of the copy operation between the CPU and GPU of each node in the calculation graph are collected.

4. The method according to claim 1, characterized in that: When evaluating whether a node is suitable as a recalculation checkpoint in step (2), its survival time, computation time, and feature map size are considered; when evaluating whether a node is suitable for being swapped to the CPU, whether the time interval between two operations that depend on the node is long enough to cover the copy overhead between the CPU and the GPU is considered.

5. The method according to claim 1, characterized in that The updating threshold in step (3.4) is specifically: For each At this moment, its memory usage are maintained at a similar level, so the threshold The updated formula is as follows:

6. The method according to claim 1, characterized in that It also includes limiting the Cudnn auxiliary space based on the additional space occupied by the GPU computing library during actual runtime.

7. The method according to claim 6, characterized in that The process of limiting the Cudnn auxiliary space is specifically as follows: According to the extra space occupied by the GPU computing library during actual runtime, the Cudnn auxiliary space is limited using the cudnnGetConvolutionForwardAlgorithm_v7() function and the size of the output feature map of each convolutional layer; the extra space is limited to 20 times the size of the output feature map.

8. The method according to claim 7, characterized in that The additional space is limited to 0.05 times the output feature map size.

Images