Video memory allocation method and device for model training and computing equipment

Abstract

The invention discloses a video memory allocation method and device for model training, computing equipment and a storage medium, and belongs to the technical field of artificial intelligence. According to the method, in the video memory allocation process, when the minimum data exchange capacity is obtained, the batch size capable of enabling the ideal training throughput rate to be maximum is referred, and then a target tensor exchange scheme is obtained from tensor exchange schemes determined based on the minimum data exchange capacity; therefore, the video memory space used for storing the tensor needing to be exchanged is allocated and shared based on the target tensor exchange scheme, and the purpose of accelerating model training can be achieved on the premise that the model precision and convergence are not affected.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computing device and storage medium for allocating video memory for model training. Background Art

[0002] Deep neural network (DNN) models are often used to handle complex tasks such as image recognition, object detection, and natural language processing. As the complexity of tasks continues to increase, researchers have proposed deeper models to extract deep features of data.

[0003] During the model training process, GPU is required to provide computing power support. Therefore, a video memory allocation method is urgently needed to achieve fast model training. Summary of the invention

[0004] The embodiment of the present application provides a method, device, computing device and storage medium for allocating video memory for model training, which can realize fast training of the model. The technical solution is as follows.

[0005] In a first aspect, a method for allocating video memory for model training is provided, the method comprising: obtaining a minimum data exchange amount of a DNN model under a target batch size, the target batch size referring to the number of samples input at each iteration of the DNN model, the target batch size being used to maximize an ideal training throughput of the DNN model, the minimum data exchange amount referring to a total amount of data that needs to be exchanged at least during model training based on the target batch size; obtaining a target tensor exchange scheme based on multiple tensor exchange schemes, the multiple tensor exchange schemes being obtained based on the DNN model and the minimum data exchange amount under the target batch size, and being used to indicate tensors that need to be exchanged in the DNN model, the target tensor exchange scheme indicating a tensor exchange scheme that minimizes the training time for one round of the model; and allocating shared video memory space to the DNN model based on the target tensor exchange scheme, the shared video memory space being used to store tensors that need to be exchanged.

[0006] Among them, obtaining the minimum data exchange amount based on the target batch size means that, in one iteration, training is performed based on samples of the target batch size, the video memory space occupied by the samples in this batch for training is obtained without video memory limitation, and the minimum data exchange amount is obtained based on the video memory space and the video memory space of the GPU.

[0007] The embodiment of the present application provides a video memory allocation method for model training, which can achieve the purpose of accelerating model training. In the video memory allocation process, when obtaining the minimum data exchange amount, the batch size that can maximize the ideal training throughput is referred to, and then the target tensor exchange scheme is obtained from the tensor exchange scheme determined based on the minimum data exchange amount, so as to allocate shared video memory space for storing tensors that need to be exchanged based on the target tensor exchange scheme. Through the above process, it is determined which tensors need to be exchanged during the training process, and shared video memory space is allocated for these tensors, so that the video memory consumption in part of the model training process is saved by tensor exchange between video memory and internal memory, and part of the video memory pressure is shared by the memory, so that the video memory originally required for model training is reduced, so that the model can be trained with a larger batch size in each batch, and the increase in batch size can improve the training throughput, that is, the time required to complete one epoch of training is shorter than that of conventional training. The technical solution does not change the tensor and model structure, so it can achieve training acceleration without affecting the accuracy and convergence of the model.

[0008] In some embodiments, obtaining the minimum data exchange amount of the DNN model under the target batch size includes: determining the target batch size of the DNN model based on the DNN model; and determining the minimum data exchange amount of the DNN model based on the target batch size.

[0009] In the above process, the ideal training throughput of the DNN model under batch size b is calculated, and the target batch size b is selected by comparing the ideal training throughput when b is different. 1 , and then calculate the minimum amount of data exchange.

[0010] This application determines the target batch size to ensure the possibility of improving the training throughput. The increase in batch size can bring training performance benefits. Furthermore, the calculation of the ideal training throughput is based on the measured computing time of the DNN model on the GPU and the hardware bandwidth of the PCIe bus. Taking into account the change in the amount of communication data for tensor exchange between video memory and memory, the determined target batch size is more practical. It should be noted that the determination of the target batch size in this application is applicable to any model training based on tensor exchange between video memory and memory.

[0011] In some embodiments, based on the DNN model, determining a target batch size of the DNN model includes:

[0012] Based on the batch size b, the forward propagation calculation time and the backward propagation calculation time of the DNN model are obtained, wherein the forward propagation calculation time refers to the sum of the forward calculation time of each layer in the DNN model, and the backward propagation calculation time refers to the sum of the backward calculation time of each layer in the DNN model, wherein the batch size b is an integer greater than or equal to 0;

[0013] Determine an ideal training throughput rate for the batch size b based on the forward propagation computation time and the backpropagation computation time;

[0014] If the ideal training throughput under the batch size b is less than or equal to the maximum training throughput, the target batch size is determined based on the batch size b, and the target batch size is the batch size b minus 1;

[0015] If the ideal training throughput under the batch size b is greater than the maximum training throughput, the maximum training throughput is updated based on the ideal training throughput under the batch size b, and the batch size b is incremented. Based on the incremented batch size b+1, the step of determining the ideal training throughput is performed.

[0016] In some embodiments, obtaining the forward propagation calculation time and the backward propagation calculation time of the DNN model based on the batch size b includes: training the DNN model based on the video memory space allocated for the DNN model and samples of the batch size b, and obtaining the forward propagation calculation time and the backward propagation calculation time of the DNN model during the training process.

[0017] Among them, the video memory space allocated to the DNN model refers to applying for video memory space as the video memory required for calculation of each layer according to the size of the video memory space occupied by the layer that occupies the largest video memory during model training. During the calculation process, the above-mentioned video memory is used to store the tensors corresponding to each layer after the calculation of a layer is completed, and after the calculation of the next layer of this layer is completed, the tensors stored in the video memory are replaced with the tensors corresponding to the next layer. Through the time-sharing storage of tensors in the video memory, video memory space can be greatly saved, and insufficient video memory space can be avoided. In some embodiments, corresponding video memory space can also be allocated to each layer of the DNN model to store the tensors of each layer.

[0018] In some embodiments, based on multiple tensor exchange schemes, obtaining a target tensor exchange scheme includes:

[0019] Based on the number of layers of the DNN model and the minimum data exchange amount under the target batch size, obtaining a first number of tensor exchange schemes;

[0020] Based on a genetic algorithm, the first number of tensor exchange schemes are iteratively processed to obtain a target tensor exchange scheme.

[0021] In some embodiments, the acquiring the first number of tensor exchange schemes based on the number of layers of the DNN model and the minimum data exchange amount under the target batch size includes:

[0022] Based on the number of layers of the DNN model, a plurality of tensor exchange schemes are randomly generated, and based on the constraint of the minimum data exchange amount under the target batch size, the first number of tensor exchange schemes that meet the constraint are obtained from the plurality of tensor exchange schemes.

[0023] In some embodiments, the iterative processing of the first number of tensor exchange schemes based on a genetic algorithm to obtain a target tensor exchange scheme includes:

[0024] In each round of evolution, the first number of tensor exchange schemes in the population are extracted, and the training time of each tensor exchange scheme in the population under the target batch size is predicted;

[0025] Determine the fitness of each tensor exchange scheme in the population based on the training time of each tensor exchange scheme in the population under the target batch size, wherein the fitness indicates the possibility that the corresponding tensor exchange scheme is preserved during the evolution process;

[0026] If the total number of evolutionary rounds has not been reached, each tensor exchange scheme in the population is evolved based on the mutation probability, and the prediction step is continued based on the evolved population;

[0027] Among them, evolving each tensor exchange scheme in the population means performing a crossover operation on each pair of adjacent tensor exchange schemes in the population, and flipping the tensor exchange schemes after the crossover operation according to the mutation probability. It should be noted that the tensor exchange schemes obtained based on the above evolution all meet the constraint conditions;

[0028] If the total number of evolutionary rounds is reached, based on the fitness of each tensor exchange scheme in the current population, the tensor exchange scheme whose fitness meets the fitness condition is selected as the target tensor exchange scheme.

[0029] Among them, fitness is an indicator used to reflect the quality of tensor exchange schemes. Tensor exchange schemes with large fitness are more likely to be preserved during the evolution process.

[0030] When searching for tensor exchange schemes based on genetic algorithms, the prediction of training time under a tensor exchange scheme is achieved using simulated training, without the need for actual training on the GPU, which greatly shortens the search time. In addition, simulated training calculates the training time based on the measured values ​​of each layer in the DNN model, and takes into account the measured bandwidth of communication on the PCIe bus for different amounts of data, thereby ensuring the accuracy of the prediction. In addition, since simulated training is based on the actual training calculation process, a simulated training program is used on the CPU to predict the training time of the DNN model under a given tensor exchange scheme on the GPU, the actual time overhead of predicting the training time in this way is much less than the time overhead of actual training on the GPU.

[0031] In some embodiments, allocating shared video memory space to the DNN model based on the target tensor exchange scheme includes:

[0032] Based on the tensor that occupies the largest video memory space in the target tensor exchange scheme, shared video memory space is allocated to the DNN model, memory space is allocated to each layer in the DNN model, and based on the target tensor exchange scheme of the DNN model, video memory space is allocated to layers that do not require tensor exchange. The video memory space is used to store tensors that do not need to be exchanged.

[0033] The video memory allocation method in the present application applies for shared video memory space according to the maximum tensor size in the DNN model, and stipulates that tensors that need to be exchanged can only be stored using shared video memory space, and the remaining tensors use fixed video memory space to store. Therefore, when implementing the technical solution of the present application, the size of video memory fragments will not exceed the size of the shared video memory space, that is, the video memory fragments are controlled within the size of a tensor, thereby maximizing the video memory utilization.

[0034] In a second aspect, a model training method is provided, the method comprising:

[0035] The DNN model is trained based on the training samples; during the training process of the DNN model, for the tensor of the i-th layer, if the tensor of the i-th layer is a tensor that needs to be exchanged, the tensor of the i-th layer is stored in the shared video memory space, and when the swap-out time corresponding to the i-th layer tensor is reached, the i-th layer tensor is swapped out from the shared video memory space to the corresponding memory space, and when the swap-in time corresponding to the i-th layer tensor is reached, the i-th layer tensor in the memory space is swapped back into the shared video memory space, where i is an integer greater than 0.

[0036] In some embodiments, the above method also includes: allocating shared video memory space to the DNN model based on a target tensor exchange scheme of the DNN model, wherein the target tensor exchange scheme is obtained based on the DNN model and the minimum data exchange amount of the DNN model at the target batch size.

[0037] Wherein, swapping out the tensor from the shared video memory space to the corresponding memory space includes: copying the tensor in the shared video memory space to the corresponding memory space, and unloading the tensor from the shared video memory space. Swapping the tensor in the memory space back into the shared video memory space includes: copying the tensor in the memory space to the shared video memory space, and optionally, for the swapped-in tensor, after the copying is completed, the tensor can be unloaded from the memory space, and the tensor in the memory space can also be overwritten during the next batch of training.

[0038] Through this model training method, tensor exchange between video memory and internal memory is realized to save part of the video memory consumption in the model training process. By sharing part of the video memory pressure with the internal memory, the amount of video memory originally required for model training is reduced, so that the DNN model can be trained with a larger batch size. The increase in batch size can improve the training throughput, that is, the time required to complete one epoch of training is shorter than that of conventional training.

[0039] In a third aspect, a video memory allocation device for model training is provided, which is used to execute the video memory allocation method for model training. Specifically, the video memory allocation device for model training includes a functional module for executing the video memory allocation method for model training provided by the first aspect or any optional method of the first aspect.

[0040] In a fourth aspect, a model training device is provided for executing the above-mentioned model training method. Specifically, the model training device includes a functional module for executing the model training method provided in the above-mentioned second aspect or any optional manner of the above-mentioned second aspect.

[0041] In a fifth aspect, a computing device or a computing device cluster is provided, which includes a processor for executing program code so that the computing device or the computing device cluster executes to implement operations performed by the above-mentioned graphics memory allocation method for model training or the model training method.

[0042] In a sixth aspect, a computer-readable storage medium is provided, in which at least one program code is stored. The program code is read by a processor to enable a computing device to perform operations performed by the above-mentioned video memory allocation method for model training or the model training method.

[0043] In the seventh aspect, a computer program product or a computer program is provided, which includes a program code, and the program code is stored in a computer-readable storage medium. A processor of a computing device reads the program code from the computer-readable storage medium, and the processor executes the program code, so that the computing device executes the methods provided in various optional implementations of the first and second aspects above.

[0044] Based on the implementations provided in the above aspects, the present application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0046] Figure 2 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0047] Figure 3 It is a flowchart of a method for allocating video memory for model training provided in an embodiment of the present application;

[0048] Figure 4 It is a flow chart of a method for determining a target batch size and a minimum data exchange amount provided by an embodiment of the present application;

[0049] Figure 5 This is a flow chart of a method for obtaining the forward propagation calculation time and the back propagation calculation time of a DNN model provided in an embodiment of the present application;

[0050] Figure 6 It is a flow chart of a method for searching for a tensor exchange scheme based on a genetic algorithm provided in an embodiment of the present application;

[0051] Figure 7 It is a flow chart of a method for iteratively processing a tensor exchange scheme provided in an embodiment of the present application;

[0052] Figure 8 This is a flow chart of a method for obtaining forward propagation calculation time for model training provided in an embodiment of the present application;

[0053] Fig. 9 It is a flow chart of a method for obtaining the back propagation calculation time of model training provided in an embodiment of the present application;

[0054] Fig.10 It is a flow chart of a method for video memory allocation based on a target tensor exchange scheme provided in an embodiment of the present application;

[0055] Fig.11 It is a flow chart of a model training method provided in an embodiment of the present application;

[0056] Fig.12 It is a flow chart of a method for allocating shared video memory space provided by an embodiment of the present application;

[0057] Fig.13 It is a structural schematic diagram of a video memory allocation device for model training provided in an embodiment of the present application;

[0058] Fig.14 It is a structural schematic diagram of a device for model training provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0060] Figure 1 1 is a schematic diagram of a computing device provided in an embodiment of the present application. It should be understood that the computing device described below can implement any function in any of the following methods. Generally, the electronic device 100 includes: a processor 101 and a memory 102.

[0061] The processor 101 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 101 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 101 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 101 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 101 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0062] The memory 102 may include one or more computer-readable storage media, which may be non-transitory. The memory 102 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 102 is used to store at least one program, which is used to be executed by the processor 101 to implement the video memory allocation method or model training method for model training provided in the method embodiment of the present application.

[0063] In some embodiments, the electronic device 100 may further include: a peripheral device interface 103 and at least one peripheral device. The processor 101, the memory 102 and the peripheral device interface 103 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 103 via a bus, a signal line or a circuit board.

[0064] In some embodiments, the computing device may be a separate physical server, or may be implemented as Figure 2 The computing device cluster shown is a server cluster or distributed file system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Taking the computing device as a cloud server as an example, the computing device can also be called a cloud platform (i.e., the abbreviation of cloud computing platform), which refers to services based on hardware resources and software resources, providing computing, network and storage capabilities. Through the network "cloud", the huge data computing processing is processed and analyzed at the remote end and then returned to the user, which has the characteristics of large-scale, distributed, virtualized, high availability, scalability, on-demand service and security. The cloud platform can realize the rapid issuance and release of configurable computing resources with a small management cost or a low interaction complexity between users and service providers.

[0065] The video memory allocation method for model training provided in the embodiment of the present application can achieve the purpose of accelerating model training. In the video memory allocation process, when obtaining the minimum data exchange amount, the batch size that can maximize the ideal training throughput is referred to, and then the target tensor exchange scheme is obtained from the tensor exchange scheme determined based on the minimum data exchange amount, so as to allocate the shared video memory space for storing the tensors that need to be exchanged based on the target tensor exchange scheme. Accordingly, the video memory allocation process provided by the present application is as follows: Figure 3 The three parts are shown.

[0066] 301. Obtain the minimum data exchange amount of the DNN model under the target batch size. In this process, based on the DNN model to be trained, the target batch size of the DNN model is determined, and the target batch size is used to maximize the ideal training throughput of the DNN model. Then, based on the target batch size, the minimum data exchange amount of the DNN model is determined. The minimum data exchange amount refers to the total amount of data that needs to be exchanged at least in the process of model training based on the target batch size, and the total amount of data is not less than the sum of the data amounts of the tensors that need to be exchanged.

[0067] 302. The process of searching for tensor exchange schemes, that is, obtaining a target tensor exchange scheme based on multiple tensor exchange schemes. The multiple tensor exchange schemes are obtained based on the minimum data exchange amount under the DNN model and the target batch size, and are used to indicate the tensors that need to be exchanged in the DNN model. The target tensor exchange scheme indicates the tensor exchange scheme that minimizes the training time of a round of the model. In this process, the tensor exchange scheme is searched based on the iteration of the genetic algorithm. In the iterative process, the training time of the model corresponding to each tensor exchange scheme is predicted based on the simulation training method.

[0068] 303. Video memory allocation process, that is, allocating shared video memory space for the DNN model based on the target tensor exchange scheme. In this process, video memory allocation is performed based on the determined tensor exchange scheme.

[0069] Through the above process, it is determined which tensors need to be exchanged during the training process, and shared video memory space is allocated for these tensors, so that the video memory consumption in part of the model training process can be saved by exchanging tensors between video memory and internal memory, and part of the video memory pressure can be shared by memory, so that the amount of video memory originally required for model training is reduced, so that the DNN model can be trained with a larger batch size. The increase in batch size can improve the training throughput, that is, the time required to complete an epoch of training is shorter than the time for conventional training. This technical solution does not change the tensor and model structure, so it can achieve training acceleration without affecting the accuracy and convergence of the model. Moreover, under the same hardware conditions, the improvement in training throughput of this application will increase with the increase in the number of network layers.

[0070] Below, based on Figure 4 The process shown in the figure introduces the process of the first part above. Figure 4 It is a flowchart of the process of determining the target batch size and the minimum data exchange amount provided in an embodiment of the present application.

[0071] 401. Based on the batch size, obtain the forward propagation calculation time and the backward propagation calculation time of the DNN model, wherein the forward propagation calculation time of the DNN model refers to the sum of the forward calculation time of each layer in the DNN model, and the backward propagation calculation time of the DNN model refers to the sum of the backward calculation time of each layer in the DNN model.

[0072] It should be noted that before executing step 401, the PCIe bandwidth B and the target batch size upper limit b are initialized. max , Maximum training throughput Itp max .

[0073] In the embodiment of the present application, B represents the PCIe bandwidth; Itp represents the ideal training throughput of the DNN model under different batch size configurations, that is, the training throughput of the DNN model obtained by testing a batch of samples without video memory restrictions; Itp max represents the maximum training throughput rate. Before executing step 401, the value of the maximum training throughput rate is initialized to zero.

[0074] In step 401, the forward propagation calculation time and the back propagation calculation time are obtained by training a batch of samples. The training process is implemented by the video memory space allocated to the model. That is, the above step 401 includes: based on the video memory space allocated to the model and the samples of the batch size, the DNN model is trained to obtain the forward propagation calculation time and the back propagation calculation time of the DNN model during the training process. The following is an introduction to the process of obtaining the forward propagation calculation time and the back propagation calculation time. Figure 5 , Figure 5 This is a flow chart of a method for obtaining the forward propagation calculation time and the back propagation calculation time of the DNN model provided in an embodiment of the present application.

[0075] 4011. At the beginning, obtain b and net for this calculation, where b is used to represent the batch size, that is, the number of samples input in each round of training, and net is used to represent the model code.

[0076] During initialization, b=0, and during the first calculation process, b=b+1, where b is an integer greater than or equal to 0.

[0077] 4012. Apply for video memory space M according to the video memory space occupied by the layer with the largest video memory in the net. shared As the video memory for each layer calculation.

[0078] The video memory space occupied by each layer refers to the video memory space that the tensor corresponding to the layer may occupy. For different layers of the DNN model, the video memory space occupied by each layer can be determined by comparing the size of the tensor corresponding to each layer, and the video memory space occupied by each layer can be compared to determine the layer that occupies the largest video memory and the video memory space occupied by the layer.

[0079] In order to prevent the memory space required during the calculation from exceeding the GPU memory limit, the corresponding memory size is applied according to the layer that occupies the largest memory in the net to store the tensors corresponding to each layer.

[0080] It should be noted that during the calculation process, the above-mentioned video memory is used to store the tensors corresponding to each layer after the calculation of the layer is completed, and after the calculation of the next layer of the layer is completed, the tensors stored in the video memory are replaced by the tensors corresponding to the next layer.

[0081] This step 4012 is explained by taking video memory as an example to implement training. By using video memory to store tensors in time-sharing mode, video memory space can be greatly saved and insufficient video memory space can be avoided. In some embodiments, corresponding video memory space can also be allocated to each layer of the DNN model to store tensors of each layer.

[0082] 4013. Check whether all layers in net have been traversed. If not, get the next layer number i and set M shared Assign to the i-th layer, execute step 4014, and if the traversal is completed, execute step 4015.

[0083] 4014. Use the samples of the batch size to run the forward calculation and reverse calculation of the i-th layer in the GPU, save the forward calculation time and reverse calculation time of the i-th layer in the forward propagation calculation timetable forwardlist(i) and the backward propagation calculation timetable backwardlist(i), respectively, and execute step 4013.

[0084] 4015. Accumulate the time recorded in the forward propagation calculation schedule forwardlist and the backward propagation calculation schedule backwardlist respectively to obtain the forward propagation calculation time ft(b) and the backward propagation calculation time bt(b) of the DNN model under the batch size.

[0085] The above steps 4011 to 4015 are an implementation method for obtaining the above forward propagation calculation time and back propagation calculation time. In this process, a certain number of samples are trained based on the GPU to obtain the time consumed for training under the batch size without the need for a complete model training process. In the process of selecting the target batch size, multiple trainings based on different batch sizes may be involved. It should be noted that the forward propagation calculation time and back propagation calculation time calculated above are both ideal training times, that is, the forward propagation calculation time and back propagation calculation time for input samples of a batch size without video memory limitations.

[0086] 402. Determine an ideal training throughput rate for the batch size based on the batch size, the PCIe bandwidth, and the forward propagation calculation time and the back propagation calculation time of the DNN model for the batch size.

[0087] The above step 402 is implemented by the following formulas (1) and (2):

[0088]

[0089]

[0090] In formula (1), Itp(b) is used to represent the ideal training throughput under batch size b; b is used to represent the batch size (batchsize); S net (b) is used to indicate the size of the video memory space occupied by using samples of this batch size for model training without video memory restrictions; S gpu It is used to indicate the size of GPU memory space; B is used to indicate the size of PCIe bandwidth; ft(b) is used to indicate the forward propagation calculation time of the DNN model under this batch size; bt(b) is used to indicate the backward propagation calculation time of the DNN model under this batch size. In formula (2), L is used to indicate the number of DNN model layers; x l A tensor used to represent the lth layer; dx l Used to represent the error gradient of the tensor of layer l; w l The weight used to represent the tensor of the lth layer; dw l Used to represent the weight gradient of the tensor of the lth layer; the size() function is used to indicate the size.

[0091] 403. If the ideal training throughput under the batch size is less than or equal to the maximum training throughput, based on the batch size, determine the target batch size of the DNN model, and the target batch size is the batch size minus 1.

[0092] In the embodiment of the present application, the change of Itp follows the following rule: when b is small, the training throughput Itp is small, and as b increases, the throughput Itp shows a trend of first increasing and then decreasing. Therefore, in the embodiment of the present application, if Itp(b)>Itp max , it indicates that Itp may not have reached its peak value, and it is necessary to continue to increase b to increase Itp; if Itp(b)≤Itp max , it indicates that Itp has reached its peak value. max The corresponding batchsize, that is, the previous b is the target batch size.

[0093] 404. If the ideal training throughput under the batch size is greater than the maximum training throughput, the maximum training throughput is updated based on the ideal training throughput under the batch size, and the batch size is incremented, and step 401 is executed based on the incremented batch size.

[0094] In step 404, after Itp(b) is calculated according to formula (1), if Itp(b)>Itp max , then let Itp max=Itp(b), b=b+1, and repeat 401; if Itp(b)≤Itp max , then use b 1 Denotes the target batch size, let b 1 =b-1, determine the target batch size.

[0095] In some embodiments, updating the maximum training throughput based on the ideal training throughput at the batch size refers to: assigning the ideal training throughput at the batch size to the maximum training throughput.

[0096] The above target batch size selection process is to predict the ideal training throughput of the DNN model under different batch size configurations, and then select the batch size that maximizes the ideal training throughput based on the calculation of the ideal training throughput as the target batch size.

[0097] 405. Obtain the minimum data exchange amount under the target batch size.

[0098] The minimum data exchange amount under the target batch size is used to constrain the tensor exchange scheme so that the obtained tensor exchange scheme can improve the training speed.

[0099] The acquisition process of step 405 is implemented by the following formula (3):

[0100] s min =S net (b 1 )-S gpu (3)

[0101] In the above determination process, a part of the training steps are first performed based on the GPU, so that the GPU works for a period of time, and its utilization rate is about 50%, and after the forward calculation time and the reverse calculation time are collected, the GPU utilization rate will drop to 0. Of course, in some embodiments, the above calculation process can also be performed without the GPU, thereby further reducing the GPU occupancy rate.

[0102] In an embodiment of the present application, before searching for a tensor exchange scheme, the target batch size is first determined to ensure the possibility that the present application can improve the training throughput. The increase in the target batch size can bring training performance benefits. Furthermore, the calculation of the ideal training throughput is based on the measured computing time of the DNN model on the GPU and the hardware bandwidth of the PCIe bus. Taking into account the change in the amount of communication data for tensor exchange between video memory and memory, the determined target batch size is more practical. It should be noted that the determination of the target batch size in the present application is applicable to any model training based on tensor exchange between video memory and memory.

[0103] Below, based on Figure 6The process shown in the figure introduces the process of the second part mentioned above. Figure 6 is a flow chart of the search process of the tensor exchange solution based on the genetic algorithm provided in the embodiment of the present application. Figure 6 , the process includes the following steps.

[0104] 601. Based on the number of layers of the DNN model and the minimum data exchange amount under the target batch size, obtain a first number of tensor exchange schemes, each tensor exchange scheme is used to indicate a layer in the DNN model for performing tensor exchange.

[0105] In some embodiments, based on the number of layers of the DNN model, multiple tensor exchange schemes are randomly generated, and based on the constraint of the minimum data exchange amount under the target batch size, a first number of tensor exchange schemes that meet the constraint are obtained from the multiple tensor exchange schemes. By using the above constraint to screen the tensor exchange schemes, the obtained tensor exchange schemes can improve the training speed.

[0106] The above constraints are the following formulas (4) and (5):

[0107] object∶mint(s) (4)

[0108]

[0109] Wherein, formula (4) is used to represent the objective function, t(s) is used to represent the forward propagation calculation time or the back propagation calculation time obtained by simulating training of a batch of samples under the target batch size based on the tensor exchange scheme. Formula (5) is used to represent the constraint condition, S off (s) is used to represent the amount of data exchange obtained based on the tensor exchange scheme; L is used to represent the number of DNN model layers; s is used to represent the tensor exchange scheme, expressed in vector form, and the lth element in the vector is represented by s l ;s l The value of is used to indicate whether the tensor of the lth layer needs to be exchanged. If s l is zero, it means that the tensor of layer l does not need to be exchanged. l If it is not zero, it means that the tensor of the lth layer needs to be exchanged; m l Used to represent the size of the tensor at layer l; S min It is used to represent the minimum amount of data exchange, that is, the total amount of data that needs to be exchanged at least in the process of model training based on samples of the target batch size. The tensor exchange solution search is transformed into solving the constrained optimization model shown in formula (4) and formula (5).

[0110] 602. Based on a genetic algorithm, iteratively process the first number of tensor exchange schemes to obtain a target tensor exchange scheme.

[0111] See also Figure 7 , the iterative processing process includes the following steps 6021 to 6025.

[0112] 6021. When initializing a population, the number of individuals in the population is set to a first number G, the mutation probability and the total number of evolutionary rounds are set, and the tensor exchange scheme of the first number is encoded to obtain each individual in the population, wherein the mutation probability is used to indicate the probability of an individual mutation, and the total number of evolutionary rounds is used to indicate the algorithm stop condition.

[0113] Among them, encoding the first number of tensor exchange schemes to obtain each individual in the population includes: when encoding, assigning 1 to the element corresponding to the layer performing tensor exchange in the encoding vector, and assigning 0 to the remaining elements, wherein 1 is used to indicate that the layer needs to perform tensor exchange, and 0 is used to indicate that the layer does not need to perform tensor exchange.

[0114] In the present embodiment, each individual in the population can use s j (j=1,2,…,G) to represent it.

[0115] It should be noted that the embodiment of the present application is described by taking the example of first obtaining a tensor exchange scheme and then performing iteration based on a genetic algorithm. In some embodiments, during the configuration of the genetic algorithm, a tensor exchange scheme is generated based on constraints and then encoded. The embodiment of the present application does not limit this.

[0116] 6022. In each round of evolution, the first number of tensor exchange schemes in the population are extracted, and the training time of each tensor exchange scheme in the population under the target batch size is predicted.

[0117] Among them, in the first round of evolution, the extracted tensor exchange scheme is the original tensor exchange scheme in the population, and in the subsequent evolution process, each time the population is evolved, the new tensor exchange scheme obtained by evolution will be put back into the population to obtain the optimized population. When extracting, the probability of the tensor exchange scheme being extracted will be calculated based on the fitness calculated in the previous round of evolution, and the extraction will be performed based on the probability, and the prediction process will be performed based on the extracted tensor exchange scheme.

[0118] In some embodiments, the probability refers to the probability of the tensor exchange scheme being selected in the roulette algorithm to increase the randomness of the extraction.

[0119] See also Figure 8 , the forward propagation calculation time acquisition process includes the following steps 801 to 806.

[0120] 801. Input PCIe bandwidth B, computation graph CG, forward propagation computation schedule forwardlist, and encoding vector s of tensor exchange scheme.

[0121] The PCIe bandwidth B is determined based on the bandwidth function B(m) and the size of the tensor exchanged in the tensor exchange scheme to be calculated, and the computation graph CG is used to represent the training process of the DNN model, which is obtained from the model code. The forward propagation computation schedule forwardlist is obtained based on the above steps 4011 to 4015.

[0122] Among them, the bandwidth function B(m) takes the tensor size as the independent variable and is obtained based on the test of the transmission time (or transmission rate) of tensors of different sizes (that is, the amount of transmitted data) on the PCIe bus. Its form is a fitted polynomial or a comparison table of data size and transmission rate.

[0123] 802. Initialize the forward propagation calculation time, the calculation time after the tensor starts to be unloaded, and the unloading time of the tensor.

[0124] Among them, unloading tensors means unloading tensors from GPU to CPU, the computation time after tensors start to be unloaded means the computation time from the start of unloading to the current time point, expressed as temp, the unloading time of tensors means the time consumed by unloading, expressed as swaptime, and the forward propagation computation time is expressed as t(s). The above three variables are initialized to 0.

[0125] 803. Check whether the forward propagation part of CG is traversed. If the traversal is completed, execute step 806. If the traversal is not completed, obtain the number i of the next layer from the forward propagation part of CG, add the forward calculation time of the i-th layer to the forward propagation calculation time, and execute step 804.

[0126] Among them, the forward calculation time of the i-th layer is accumulated into the forward propagation calculation time, that is, t(s)=t(s)+forwardlist(i).

[0127] 804. Based on the encoding vector of the tensor exchange scheme, check whether the tensor of the i-th layer needs to be exchanged. If not, add the time of the i-th layer in the forward propagation calculation time forwardlist to the calculation time after the last tensor that needs to be exchanged starts to be unloaded, and execute step 803; if necessary, check whether the calculation time after the last tensor that needs to be exchanged starts to be unloaded is less than the unloading time of the tensor. If so, add the waiting time for unloading to the forward propagation calculation time, and execute step 805. If not, execute step 805.

[0128] If the calculation time after the last tensor that needs to be exchanged starts to be unloaded is less than the unloading time of the tensor, it means that the last tensor that needs to be exchanged has not been unloaded. The difference between the unloading time of the tensor and the calculation time after the last tensor that needs to be exchanged starts to be unloaded is determined as the waiting time for unloading, and the waiting time for unloading is added to the forward propagation calculation time, and then step 805 is executed. If the calculation time after the last tensor that needs to be exchanged starts to be unloaded is not less than the unloading time of the tensor, it indicates that the unloading is completed, and step 805 can be executed directly.

[0129] 805. Calculate the unloading time of the tensor that needs to be exchanged in the i-th layer, put the number of the i-th layer into the stack, reset the calculation time after the tensor starts to be unloaded, and execute step 803, wherein the stack is used to record the number of the layer where the tensor that needs to be exchanged is located.

[0130] The calculation of the unloading time of the tensor is implemented by the following formula (6).

[0131]

[0132] In formula (6), swaptime is used to represent the unloading time of the tensor; m l Used to represent the size of the tensor of the lth layer; B(m l ) is used to represent the bandwidth function value calculated based on the size of the tensor of the lth layer using the bandwidth function.

[0133] 806. Output forward propagation calculation time t(s).

[0134] The above steps 801 to 806 are the process of predicting the forward propagation calculation time of different tensor exchange schemes by using simulation training. It should be noted that the steps of predicting the reverse propagation calculation time of different tensor exchange schemes are the same as the above process, see Fig. 9 , the steps to predict the backpropagation computation time of different tensor exchange schemes include:

[0135] 901. Input PCIe bandwidth B, computation graph CG, backwardlist, a backpropagation computation schedule, and encoding vector s of the tensor exchange scheme.

[0136] The PCIe bandwidth B is determined based on the bandwidth function B(m) and the size of the tensor exchanged in the tensor exchange scheme to be calculated, and the computation graph CG is used to represent the training process of the DNN model, which is obtained from the model code. The backward propagation computation schedule backwardlist is obtained based on the above steps 4011 to 4015.

[0137] Among them, the bandwidth function B(m) takes the tensor size as the independent variable and is obtained based on the test of the transmission time (or transmission rate) of tensors of different sizes (that is, the amount of transmitted data) on the PCIe bus. Its form is a fitted polynomial or a comparison table of data size and transmission rate.

[0138] 902. Initialize the back propagation calculation time, the calculation time after the tensor prefetching starts, and the tensor prefetching time.

[0139] Among them, prefetching tensors means prefetching tensors from the CPU to the GPU, the computing time after the tensor prefetching starts means the computing time from the start of prefetching to the current time point, expressed as temp, the tensor prefetching time means the time consumed by prefetching, expressed as swaptime, and the back propagation computing time is expressed as t(s). The above three variables are initialized to 0.

[0140] 903. Check whether the traversal of the reverse propagation part of CG is completed. If the traversal is completed, execute step 906. If the traversal is not completed, obtain the number i of the next layer and the tensor required for the i-th layer from the reverse propagation part of CG, add the reverse calculation time of the i-th layer to the reverse propagation calculation time, and execute step 904.

[0141] The tensor required for the i-th layer refers to the tensor required for the i-th layer calculation. For example, if the i-th layer calculation requires the tensor calculated by the i-1-th layer, such as the feature map, then the tensor calculated by the i-1-th layer is the tensor required for the i-th layer.

[0142] Among them, the reverse calculation time of the i-th layer is accumulated into the backward propagation calculation time, that is, t(s)=t(s)+backwardlist(i).

[0143] 904. Check whether the tensors required for the i-th layer are in the GPU. If so, add the time of the i-th layer in the backward propagation calculation time backwardlist to the calculation time after the prefetching of the tensors required for the i-th layer begins, and execute step 903. If not, check whether the calculation time after the prefetching of the tensors required for the i-th layer begins is less than the prefetching time of the tensors required for the i-th layer. If so, add the waiting time for prefetching to the forward propagation calculation time, and execute step 905. If not, execute step 905.

[0144] If the computing time after the prefetching of the tensors required for the i-th layer starts is less than the prefetching time of the tensors required for the i-th layer, it means that the prefetching of the tensors required for the i-th layer has not been completed, and the difference between the prefetching time of the tensors required for the i-th layer and the computing time after the prefetching of the tensors required for the i-th layer starts is determined as the waiting time for prefetching, and the waiting time for prefetching is added to the back propagation computing time, and then step 905 is executed. If the computing time after the prefetching of the tensors required for the i-th layer starts is not less than the prefetching time of the tensors required for the i-th layer, it means that the prefetching has been completed, and step 905 can be executed directly.

[0145] 905. Take out the number of the corresponding layer of the tensor required for the i-th layer from the stack, calculate the pre-fetch time of the tensor required for the i-th layer, reset the calculation time after the pre-fetch of the tensor required for the i-th layer starts, and execute step 903, wherein the stack is used to record the number of the layer where the tensor to be exchanged is located.

[0146] The calculation of the tensor pre-fetch time is implemented by the above formula (6).

[0147] 906. Output the back-propagation calculation time t(s).

[0148] The above process uses simulation training to predict the training time of different tensor exchange schemes. The training time includes forward propagation calculation time and back propagation calculation time. There is no need to perform real training on the GPU. Compared with the actual operation of DNN models based on different tensor exchange schemes, the search time is greatly shortened. Furthermore, the prediction process takes into account the impact of the transmission data size on the PCIe bandwidth, and uses different bandwidth values ​​according to the size of the exchanged tensor to more accurately predict the unloading time and pre-fetching time of the tensor, so that the predicted training time of the tensor exchange scheme can be more accurate based on the unloading time and expected time.

[0149] 6023. Based on the training time of each tensor exchange scheme in the population under the target batch size, the fitness of each tensor exchange scheme in the population is determined, wherein the fitness is an indicator used to reflect the pros and cons of the tensor exchange scheme, and the tensor exchange scheme with a large fitness is more likely to be preserved during the evolution process.

[0150] The fitness is obtained by the following formula (7):

[0151]

[0152]

[0153] Among them, p i Based on the fitness determination of each tensor exchange scheme, it is obtained through formula (8), which represents the possibility that the corresponding tensor exchange scheme is preserved during the evolution process.

[0154] 6024. If the total number of evolutionary rounds has not been reached, each tensor exchange scheme in the population is evolved based on the mutation probability, and step 6022 is continued based on the evolved population.

[0155] Among them, based on the mutation probability, evolving each tensor exchange scheme in the population includes: performing a crossover operation on each pair of adjacent tensor exchange schemes in the population, and flipping the tensor exchange schemes after the crossover operation according to the mutation probability.

[0156] Among them, performing a crossover operation on each pair of adjacent tensor exchange schemes in the population includes: for any pair of adjacent tensor exchange schemes, randomly determining a crossover point from the L-1 elements of the tensor exchange scheme, using the crossover point as a dividing point, and exchanging the elements after the crossover point in the two tensor exchange schemes to obtain a tensor exchange scheme after the crossover operation. Among them, the crossover point can be the same or different for different adjacent tensor exchange schemes, and the embodiments of the present application do not limit this.

[0157] For example, every two adjacent individuals in the population 2i and 2i+1 A crossover operation is performed as shown in the following example. An example of the crossover operation is given. In the following example, an integer is randomly selected from 1 to L-1 as the crossover point c. Accordingly, the tensor exchange scheme before and after the crossover operation is shown below.

[0158]

[0159] The step of flipping the tensor exchange scheme after the crossover operation according to the mutation probability includes: flipping each element in the tensor exchange scheme after the crossover operation according to the mutation probability.

[0160] It should be noted that the above-mentioned tensor exchange schemes based on evolution all meet the constraints.

[0161] 6025. If the total number of evolutionary rounds is reached, based on the fitness of each tensor exchange scheme in the population obtained in the last round of evolution, a tensor exchange scheme whose fitness meets the fitness condition is selected as the target tensor exchange scheme.

[0162] Among them, the tensor exchange scheme whose fitness meets the fitness condition is the tensor exchange scheme with the largest fitness in the population, that is, after a certain number of iterations of the genetic algorithm, a tensor exchange scheme with relatively high fitness can be searched.

[0163] When searching for tensor exchange schemes based on genetic algorithms, the prediction of training time under a tensor exchange scheme is achieved using simulated training, without the need for actual training on the GPU, which greatly shortens the search time. In addition, the simulated training calculates the training time based on the measured values ​​of each layer in the DNN model, and takes into account the measured bandwidth of communication on the PCIe bus for different amounts of data, thereby ensuring the accuracy of the prediction.

[0164] Below, based on Fig.10 The process shown in the figure introduces the process of the third part mentioned above. Fig.10 is a flowchart of a video memory allocation process based on a target tensor exchange solution provided in an embodiment of the present application. Fig.10 , the process includes the following steps.

[0165] 1001. Based on the tensor that occupies the largest video memory space in the target tensor exchange scheme of the DNN model, a shared video memory space is allocated to the DNN model, wherein the shared video memory space is used to store the tensors that need to be exchanged.

[0166] 1002. Allocate memory space for each layer in the DNN model.

[0167] 1003. Based on the target tensor exchange scheme of the DNN model, a video memory space is allocated for the layers that do not need to perform tensor exchange, and the video memory space is used to store tensors that do not need to be exchanged.

[0168] The above steps 1002 and 1003 can be implemented by the following process: Set the batch size of the DNN model to b 1 ; Check whether all layers in net have been traversed. If not, get the next layer number i and continue to the next step; if completed, start training. Allocate memory space for the i-th layer tensor; Check the value of s(i). If it is 0, allocate video memory space for the i-th layer tensor. If it is 0, do not allocate video memory for it.

[0169] Through the above process, according to the target batch size obtained in the first part and the target tensor exchange scheme obtained in the second part, memory allocation and video memory allocation are performed for all tensors in the DNN model, and fixed and exclusive video memory space is allocated to tensors that do not need to participate in the exchange. Tensors that need to participate in the exchange use a shared video memory space uniformly during the training process, and the remaining tensors that need to participate in the exchange use the shared video memory space in a time-sharing manner during the training process.

[0170] In the current model training process based on tensor exchange between video memory and internal memory, since tensors of different sizes are constantly entering and exiting the video memory, video memory fragmentation will be generated, and the unnecessary video memory waste caused by the fragmented space will also take time to process the fragments. Video memory fragments are generated by tensors that need to be exchanged. However, the video memory allocation method for model training in this application applies for shared video memory space according to the maximum tensor size in the DNN model, and stipulates that tensors that need to be exchanged can only be stored in shared video memory space, and the remaining tensors use fixed video memory space. Therefore, when implementing the technical solution of this application, the size of video memory fragments will not exceed the size of the tensor pool, that is, the video memory fragments are controlled within the size of a tensor, so that the video memory utilization rate can be maximized.

[0171] In the embodiment of the present application, when obtaining the minimum data exchange amount, the target batch size that can maximize the ideal training throughput is referenced, and based on the minimum data exchange amount and the target batch size, constraints are established to obtain multiple tensor exchange schemes, and then a genetic algorithm is used to obtain the target tensor exchange scheme from the multiple tensor exchange schemes, so as to allocate shared video memory space for storing tensors that need to be exchanged based on the target tensor exchange scheme. Through the above process, it is determined which tensors need to be exchanged during the training process, and shared video memory space is allocated for these tensors, so that the video memory consumption in the training process of some models can be saved by exchanging tensors between video memory and internal memory, and the video memory pressure can be shared by memory, so that the video memory originally required for model training can be reduced, so that the DNN model can be trained with a larger batch size, and the increase in batch size can improve the training throughput, that is, the time required to complete the training of an epoch is shorter than that of conventional training. This technical solution does not change the tensor and model structure, so it can achieve training acceleration without affecting the accuracy and convergence of the model.

[0172] Through the processing of the above three parts, the target tensor exchange scheme applied in the training process and the video memory space allocated for implementing the strategy can be determined for the DNN model. In the training process, for the tensor of the i-th layer, if the target tensor exchange scheme indicates that the tensor of the i-th layer is a tensor that needs to be exchanged, the i-th layer tensor is stored in the shared video memory space. When the swap-out time corresponding to the i-th layer tensor is reached, the i-th layer tensor is swapped out from the shared video memory space to the corresponding memory space. When the swap-in time corresponding to the i-th layer tensor is reached, the i-th layer tensor in the memory space is swapped back into the shared video memory space. If the target tensor exchange scheme indicates that the tensor of the i-th layer is a tensor that does not need to be exchanged, the i-th layer tensor is stored in the corresponding video memory space. The swap-out time and swap-in time corresponding to the tensors of each layer are determined based on the model code of the DNN model.

[0173] Among them, swapping out the tensor from the shared video memory space to the corresponding memory space includes: copying the tensor in the shared video memory space to the corresponding memory space, and unloading the tensor from the shared video memory space. Swapping the tensor in the memory space back into the shared video memory space includes: copying the tensor in the memory space to the shared video memory space. Optionally, for the swapped-in tensor, the tensor can be unloaded from the memory space after the copy is completed, and the tensor in the memory space can also be overwritten during the next batch of training. This embodiment of the present application does not limit this.

[0174] Below, as Fig.11 The specific example shown is used to illustrate the tensor exchange process in the above training process.

[0175] like Fig.11 As shown in the figure, x l (c) represents the tensor of the lth layer in the DNN model. The tensor that needs to be exchanged between the video memory and the main memory is x 0 、x 2 、x 5 , use the shared video memory space TP to store the above tensors, and the tensor that does not need to be exchanged is x 1 、x 3 、x 4 , each has an exclusive video memory space. c is a counter attached to each tensor, and its initial value indicates the remaining number of times the tensor is used in the forward propagation. 0 and x 5 They need to be used by layer 1 and loss module respectively, so the initial value of c is 1, x 2 It is used by layers 3 and 4, so the initial value of c is 2. Each time c is used, it is reduced by 1. After c is cleared and the tensor is unloaded, the tensor can be deleted from TP and TP is given to the next tensor for use. In the reverse propagation, each time x is used by c, it is increased by 1. When c is restored to its initial value, the current tensor can be deleted from TP and the next tensor can be pre-fetched. The embodiment of the present application uses a stack swapstack to record the number of the unloaded tensor. Once the TP is idle in the reverse propagation, the tensor referred to by the number at the top of the stack is pre-fetched and the number is thrown out of the stack.

[0176] This example includes the following steps:

[0177] During the forward propagation, at t 1 Period, the initial tensor x 0 Stored in TP, used in the first layer of the DNN model 0 Calculate and get x 1 , x 1 Belong to the tensor that does not need to be exchanged, x 1 Stored in the corresponding fixed video memory space, after the calculation is completed, x0 The c is cleared to zero, indicating that the tensor will no longer be used in the forward propagation process, and x 0 Copy to the corresponding memory space and unload x from TP 0 , and x 0 The corresponding layer number is stored in the stack swapstack.

[0178] In t 2 Period, in the second layer of the DNN model, using x in fixed video memory space 1 Calculate and get x 2 .x 2 It is a tensor that needs to be exchanged and needs to be used twice in the forward propagation process. Therefore, x 2 Stored in TP, x 1 It should be noted that x 1 It is a tensor that does not need to be exchanged, so it is still stored in the corresponding fixed video memory space.

[0179] In t 3 Period, using x in the third layer of the DNN model 2 Calculate and get x 3 .x 3 Belong to the tensor that does not need to be exchanged, x 3 Stored in the corresponding fixed video memory space. It should be noted that x 2 There is one usage left, no need to uninstall, it is still stored in TP.

[0180] In t 4 Period, using x in the fourth layer of the DNN model 2 and x 3 Calculate and get x 4 .x 4 Belong to the tensor that does not need to be exchanged, x 4 Stored in the corresponding fixed video memory space, x 2 The number of times the tensor is used is cleared to zero, indicating that it is no longer used in the forward propagation process, and x in TP is 2 Copy to the corresponding memory space and unload x from TP 2 , and x 2 The corresponding layer number is stored in the stack swapstack.

[0181] In t 5 Period, using x in the fifth layer of the DNN model 4 Calculate and get x 5 .x 5 It is a tensor that needs to be exchanged and needs to be used once in the forward propagation process, so x 5 Stored in TP, x4 It should be noted that, but x 4 It is a tensor that does not need to be exchanged, so it is still stored in the corresponding fixed video memory space.

[0182] In t 6 Time period and t 7 period, that is, the last layer of forward propagation and the first layer of backward propagation both need to use x in the loss module 5 Therefore, there is no need to uninstall x 5 At t 6 Time period, x 5 Clear c to zero and set x 5 The corresponding layer number is stored in the stack, and at t 7 Time period, x 5 c is incremented by 1, the initial value is restored, and x is unloaded from TP 5 , x 5 The number of the corresponding layer is thrown from the stack.

[0183] In t 8 Period, using x in the fifth layer of the DNN model 4 At this time, TP is idle, and the x corresponding to number 2 at the top of the stack is pre-fetched. 2 and stored in TP, x 2 At this time, c is 0, and x 2 The number of the corresponding layer is thrown from the stack.

[0184] In t 9 Period, using x in the fourth layer of the DNN model 3 and x 2 Perform calculations. 3 and x 2 The c of all becomes 1. It should be noted that x 3 c has been restored to its initial value, x 3 is a tensor stored in fixed video memory space, so it does not need to be unloaded. 2 c has not been restored to its original value and does not need to be uninstalled.

[0185] In t 10 Period, using x in the third layer of the DNN model 2 Calculate, x 2 c is incremented by 1, the initial value is restored, and x is unloaded from TP 2 .

[0186] In t 11 Period, using x in the second layer of the DNN model 1 Calculate, x 1 c plus one. It should be noted that at this time x 1 c returns to its original value, but x1 It is a tensor stored in a fixed video memory space, so it does not need to be unloaded. At this time, TP is idle, and the x corresponding to the number 0 at the top of the stack is pre-fetched. 0 and stored in TP, x 0 At this time, c is 0, and x 0 The number of the corresponding layer is thrown from the stack.

[0187] In t 12 Period, using x in the first layer of the DNN model 0 Calculate, x 0 c is incremented by 1, the initial value is restored, and x is unloaded from TP 0 .

[0188] Fig.12 This is a schematic diagram of the tensors stored by TP at different times during the DNN model training process. Fig.11 It can be seen that at t 1 Time period, the value stored in TP is x 0 , at t 2 to 4 Time period, the value stored in TP is x 2 , at t 5 to 7 Time period, the value stored in TP is x 5 , at t 8 to 10 Time period, the value stored in TP is x 2 , at t 11 to 12 Time period, the value stored in TP is x 0 .

[0189] The above describes the method of the embodiment of the present application, and the following describes the device of the embodiment of the present application. It should be understood that the device described below has any function of the computing device in the above method. Figures 1 to 12 The video memory allocation method for model training according to the embodiment of the present application is described in detail. Based on the same inventive concept, the following will be combined with Fig.13 The following describes a video memory allocation device for model training according to an embodiment of the present application. It should be understood that the technical features described in the method embodiment are also applicable to the following device embodiment.

[0190] See also Fig.13 , an embodiment of the present application provides a video memory allocation device for model training, the device comprising:

[0191] The data exchange amount acquisition module 1301 is used to obtain the minimum data exchange amount of the DNN model under the target batch size, where the target batch size refers to the number of samples input at each iteration of the DNN model, and the target batch size is used to maximize the ideal training throughput of the DNN model. The minimum data exchange amount refers to the total amount of data that needs to be exchanged at least during the model training based on the target batch size;

[0192] An exchange scheme acquisition module 1302 is used to acquire a target tensor exchange scheme based on multiple tensor exchange schemes, wherein the multiple tensor exchange schemes are obtained based on the DNN model and the minimum data exchange amount under the target batch size, and are used to indicate tensors that need to be exchanged in the DNN model, and the target tensor exchange scheme indicates a tensor exchange scheme that minimizes the training time of one round of the model;

[0193] The allocation module 1303 is used to allocate shared video memory space to the DNN model based on the target tensor exchange scheme, and the shared video memory space is used to store tensors that need to be exchanged.

[0194] In some embodiments, the data exchange amount acquisition module includes:

[0195] A batch size determination unit, configured to determine the target batch size of the DNN model based on the DNN model;

[0196] The exchange amount determination unit is used to determine the minimum data exchange amount of the DNN model based on the target batch size.

[0197] In some embodiments, the batch size determination unit includes:

[0198] A time acquisition subunit, used to acquire the forward propagation calculation time and the backward propagation calculation time of the DNN model based on the batch size b, wherein the forward propagation calculation time refers to the sum of the forward calculation time of each layer in the DNN model, and the backward propagation calculation time refers to the sum of the backward calculation time of each layer in the DNN model, wherein b is an integer greater than or equal to 0;

[0199] A throughput acquisition subunit, for determining an ideal training throughput under the batch size b based on the forward propagation calculation time and the back propagation calculation time;

[0200] The quantity determination subunit is used to determine the target batch size based on the batch size b if the ideal training throughput under the batch size b is less than or equal to the maximum training throughput, and the target batch size is the batch size b minus 1; if the ideal training throughput under the batch size b is greater than the maximum training throughput, then the maximum training throughput is updated based on the ideal training throughput under the batch size b, and the batch size b is incremented, and based on the incremented batch size b+1, the step of determining the ideal training throughput is performed.

[0201] In some embodiments, the time acquisition subunit is used to:

[0202] Based on the video memory space allocated for the DNN model and the samples of the batch size b, the DNN model is trained to obtain the forward propagation calculation time and the back propagation calculation time of the DNN model during the training process.

[0203] In some embodiments, the exchange scheme acquisition module includes:

[0204] A first scheme acquisition unit, configured to acquire a first number of tensor exchange schemes based on the number of layers of the DNN model and the minimum data exchange amount under the target batch size;

[0205] The second solution acquisition unit iteratively processes the first number of tensor exchange solutions based on a genetic algorithm to obtain a target tensor exchange solution.

[0206] In some embodiments, the first solution acquisition unit is used to:

[0207] Based on the number of layers of the model, a plurality of tensor exchange schemes are randomly generated, and based on the constraint of the minimum data exchange amount under the target batch size, the first number of tensor exchange schemes that meet the constraint are obtained from the plurality of tensor exchange schemes.

[0208] In some embodiments, the second solution acquisition unit is used to:

[0209] In each round of evolution, the first number of tensor exchange schemes in the population are extracted, and the training time of each tensor exchange scheme in the population under the target batch size is predicted;

[0210] Determine the fitness of each tensor exchange scheme in the population based on the training time of each tensor exchange scheme in the population under the target batch size, wherein the fitness indicates the possibility that the corresponding tensor exchange scheme is preserved during the evolution process;

[0211] If the total number of evolutionary rounds has not been reached, each tensor exchange scheme in the population is evolved based on the mutation probability, and the prediction step is continued based on the evolved population;

[0212] If the total number of evolutionary rounds is reached, based on the fitness of each tensor exchange scheme in the population obtained in the last round of evolution, the tensor exchange scheme whose fitness meets the fitness condition is selected as the target tensor exchange scheme.

[0213] In some embodiments, the allocation module is used to allocate shared video memory space for the model based on the tensor that occupies the largest video memory space in the target tensor exchange scheme.

[0214] It should be understood that the video memory allocation device for model training corresponds to the computing device in the above-mentioned method embodiment, and the modules in the device and the above-mentioned other operations and / or functions are respectively various steps and methods implemented by the computing device in the method embodiment to realize the specific details. Please refer to the above-mentioned method embodiment. For the sake of brevity, they will not be repeated here.

[0215] See also Fig.14 , an embodiment of the present application provides a model training device, the device comprising:

[0216] The training module 1401 is used for, during the training process of the DNN model, for the tensor of the i-th layer, if the target tensor exchange scheme indicates that the tensor of the i-th layer is a tensor that needs to be exchanged, then the tensor of the i-th layer is stored in the shared video memory space, and when the swap-out time corresponding to the i-th layer tensor is reached, the i-th layer tensor is swapped out from the shared video memory space to the corresponding memory space, and when the swap-in time corresponding to the i-th layer tensor is reached, the i-th layer tensor in the memory space is swapped back into the shared video memory space, wherein i is an integer greater than 0.

[0217] It should be understood that the device for model training corresponds to the computing device in the above-mentioned method embodiment, and the modules in the device and the above-mentioned other operations and / or functions are respectively various steps and methods implemented by the computing device in the method embodiment. The specific details can be found in the above-mentioned method embodiment. For the sake of brevity, they will not be repeated here.

[0218] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including a program code, and the program code can be executed by a processor in a terminal to complete the video memory allocation method or model training method for model training in the above embodiment. For example, the computer-readable storage medium is a non-temporary computer-readable storage medium, such as a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device.

[0219] An embodiment of the present application also provides a computer program product or a computer program, which includes a program code. The computer instructions are stored in a computer-readable storage medium. The processor in the terminal reads the program code from the computer-readable storage medium, and the processor executes the program code, so that the terminal executes the above-mentioned video memory allocation method for model training or model training method.

[0220] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the device is running, the processor can execute the computer execution instructions stored in the memory so that the chip executes the video memory allocation method or model training method for model training in the above-mentioned method embodiments.

[0221] Among them, the device, equipment, computer-readable storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0222] Through the description of the above implementation methods, those skilled in the art can understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functional allocation can be completed by different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the video memory allocation method for model training or the model training method embodiment provided in the above embodiment belongs to the same concept, and its specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0223] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0224] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0225] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0226] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks or optical disks.

[0227] In the description of this application, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, "at least one" means one or more, and "plurality" means two or more. The words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not limit them to be different.

[0228] In this application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the words "exemplary" or "for example" is intended to present the related concepts in a concrete way.

[0229] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the samples involved in this application were obtained with full authorization.

[0230] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0231] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A video memory allocation method for model training, characterized in that: The method comprises: Obtaining a minimum data exchange amount of the DNN model under a target batch size, wherein the target batch size refers to the number of samples input at each iteration of the DNN model, and the target batch size is used to maximize the ideal training throughput of the DNN model, and the minimum data exchange amount refers to the total amount of data that needs to be exchanged at least during model training based on the target batch size; Based on multiple tensor exchange schemes, a target tensor exchange scheme is obtained, wherein the multiple tensor exchange schemes are obtained based on the DNN model and the minimum data exchange amount under the target batch size, and are used to indicate tensors that need to be exchanged in the DNN model, and the target tensor exchange scheme indicates a tensor exchange scheme that minimizes a round of model training time; Based on the target tensor exchange scheme, a shared video memory space is allocated to the DNN model, and the shared video memory space is used to store tensors that need to be exchanged.

2. The method according to claim 1, characterized in that The method of obtaining the minimum data exchange amount of the DNN model under the target batch size includes: Based on the DNN model, determining the target batch size of the DNN model; Based on the target batch size, the minimum data exchange amount of the DNN model is determined.

3. The method according to claim 2, characterized in that The determining, based on the DNN model, the target batch size of the DNN model comprises: Based on the batch size b, the forward propagation calculation time and the backward propagation calculation time of the DNN model are obtained, wherein the forward propagation calculation time refers to the sum of the forward calculation time of each layer in the DNN model, and the backward propagation calculation time refers to the sum of the backward calculation time of each layer in the DNN model, wherein the batch size b is an integer greater than or equal to 0; Determining an ideal training throughput rate under the batch size b based on the forward propagation calculation time and the back propagation calculation time; If the ideal training throughput under the batch size b is less than or equal to the maximum training throughput, the target batch size is determined based on the batch size b, and the target batch size is the batch size b minus 1; If the ideal training throughput under the batch size b is greater than the maximum training throughput, the maximum training throughput is updated based on the ideal training throughput under the batch size b, and the batch size b is incremented. Based on the incremented batch size b+1, the step of determining the ideal training throughput is performed.

4. The method according to claim 3, characterized in that The obtaining of the forward propagation calculation time and the backward propagation calculation time of the DNN model based on the batch size b includes: Based on the video memory space allocated for the DNN model and the samples of the batch size b, the DNN model is trained to obtain the forward propagation calculation time and the back propagation calculation time of the DNN model during the training process.

5. The method according to claim 1, characterized in that The acquiring of a target tensor exchange scheme based on multiple tensor exchange schemes includes: Based on the number of layers of the DNN model and the minimum data exchange amount under the target batch size, obtaining a first number of tensor exchange schemes; Based on a genetic algorithm, the first number of tensor exchange schemes are iteratively processed to obtain the target tensor exchange scheme.

6. The method according to claim 5, characterized in that The acquiring of a first number of tensor exchange schemes based on the number of layers of the DNN model and the minimum data exchange amount under the target batch size includes: Based on the number of layers of the DNN model, a plurality of tensor exchange schemes are randomly generated, so as to obtain the first number of tensor exchange schemes that meet the constraint condition of the minimum data exchange amount under the target batch size from the plurality of tensor exchange schemes.

7. The method according to claim 5, characterized in that The iterative processing of the first number of tensor exchange schemes based on the genetic algorithm to obtain the target tensor exchange scheme includes: In each round of evolution, extracting a first number of tensor exchange schemes in the population, and predicting the training time of each tensor exchange scheme in the population under the target batch size; Determining the fitness of each tensor exchange scheme in the population based on the training time of each tensor exchange scheme in the population under the target batch size, wherein the fitness indicates the likelihood that the corresponding tensor exchange scheme is preserved during the evolution process; If the total number of evolutionary rounds has not been reached, each tensor exchange scheme in the population is evolved based on the mutation probability, and the prediction step is continued based on the evolved population; If the total number of evolutionary rounds is reached, based on the fitness of each tensor exchange scheme in the current population, a tensor exchange scheme whose fitness meets the fitness condition is selected as the target tensor exchange scheme.

8. The method according to any one of claims 1 to 7, characterized in that: The allocating shared video memory space for the DNN model based on the target tensor exchange scheme includes: Based on the tensor that occupies the largest video memory space in the target tensor exchange scheme, shared video memory space is allocated to the DNN model.

9. A model training method, characterized in that: The method comprises: Train the model based on the training samples; During the training process of the DNN model, for the tensor of the i-th layer, if the tensor of the i-th layer is a tensor that needs to be exchanged, the tensor of the i-th layer is stored in the shared video memory space, and when the swap-out time corresponding to the i-th layer tensor is reached, the i-th layer tensor is swapped out from the shared video memory space to the corresponding memory space, and when the swap-in time corresponding to the i-th layer tensor is reached, the i-th layer tensor in the memory space is swapped back into the shared video memory space, where i is an integer greater than 0.

10. A video memory allocation device for model training, characterized in that: The device comprises: A data exchange amount acquisition module is used to obtain the minimum data exchange amount of the DNN model under the target batch size, wherein the target batch size refers to the number of samples input at each iteration of the DNN model, and the target batch size is used to maximize the ideal training throughput of the DNN model. The minimum data exchange amount refers to the total amount of data that needs to be exchanged at least during the model training based on the target batch size; An exchange scheme acquisition module, used to acquire a target tensor exchange scheme based on multiple tensor exchange schemes, wherein the multiple tensor exchange schemes are obtained based on the DNN model and the minimum data exchange amount under the target batch size, and are used to indicate tensors that need to be exchanged in the DNN model, and the target tensor exchange scheme indicates a tensor exchange scheme that minimizes the training time of one round of the model; An allocation module is used to allocate shared video memory space to the DNN model based on the target tensor exchange scheme, where the shared video memory space is used to store tensors that need to be exchanged.

11. A model training device, characterized in that: The device comprises: The training module is used to train the DNN model based on training samples; During the training process of the DNN model, for the tensor of the i-th layer, if the tensor of the i-th layer is a tensor that needs to be exchanged, the tensor of the i-th layer is stored in the shared video memory space, and when the swap-out time corresponding to the i-th layer tensor is reached, the i-th layer tensor is swapped out from the shared video memory space to the corresponding memory space, and when the swap-in time corresponding to the i-th layer tensor is reached, the i-th layer tensor in the memory space is swapped back into the shared video memory space, where i is an integer greater than 0.

12. A computing device, characterized in that: The computing device comprises a processor, and the processor is configured to execute program code so that the computing device performs the method according to any one of claims 1 to 9.

13. A computing device cluster, characterized in that: The method comprises a plurality of computing devices, including a processor, wherein the processor is configured to execute program code, so that the computing device cluster executes the method according to any one of claims 1 to 9.

14. A computer-readable storage medium, characterized in that: At least one program code is stored in the storage medium, and the at least one program code is read by a processor to enable a computing device to execute the method according to any one of claims 1 to 9.

Images