A deep neural network model training method and device
By dividing the deep neural network model into sub-models and optimizing GPU allocation, the limitations of GPU memory space and data transmission bandwidth are solved, thereby improving the performance of model training.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TSINGHUA UNIVERSITY
- Filing Date
- 2023-03-23
- Publication Date
- 2026-08-04
AI Technical Summary
During model training, the limited GPU memory and restricted data transfer bandwidth between the CPU and GPU negatively impact model training performance.
The deep neural network model is divided into multiple sub-models, and the GPU for each sub-model is determined based on the objective of mixing linear programming information and interleaved mapping information. Training is then performed separately using the GPUs of each sub-model to alleviate communication competition.
This reduces the memory requirements of a single GPU, alleviates communication competition between sub-models, and improves model training performance.
Smart Images
Figure CN116467072B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of model training technology, and in particular relates to a method and apparatus for training deep neural network models. Background Technology
[0002] As the size of the model continues to increase, the storage requirements for model training also increase, leading to the following problems: First, the video memory of the general-purpose graphics processing unit (GPU) device used in the model is limited, making it difficult to meet the training requirements of the model.
[0003] Secondly, the data transfer bandwidth between the Central Processing Unit (CPU) and the GPU, and between multiple GPUs, is limited by the bandwidth of the Peripheral Component Interconnect Express (PCIe) standard, resulting in severe communication contention and thus affecting the performance of model training. Summary of the Invention
[0004] This application provides a method and apparatus for training a deep neural network model, which can improve the training performance of the model.
[0005] On one hand, embodiments of this application provide a method for training a deep neural network model, the method comprising: Based on the target hybrid linear programming information, the target deep neural network model is divided into multiple sub-models; each sub-model contains one computational layer or multiple consecutive computational layers in the target deep neural network model; the target hybrid linear programming information is information that satisfies the limitations of the video memory space of a general-purpose graphics processing unit (GPU) and the limitations of the execution order of the computational layers in the target deep neural network model, and information that minimizes the single-round training time of the target deep neural network model; the single-round training time is the time required to train all computational layers in the target deep neural network model once in the execution order; Based on the target interleaving mapping information, the GPU used to train each of the sub-models is determined; the target interleaving mapping information is the interleaving mapping information with the least competition among the multiple sub-models; Based on the real training dataset, the sub-models are trained separately using the GPUs of each sub-model to obtain the trained models.
[0006] In one possible implementation, before dividing the target deep neural network model into multiple sub-models based on the target hybrid linear programming information, the method further includes: The target deep neural network model is divided according to any one of the hybrid linear programming information to obtain multiple pre-allocated sub-models; the hybrid linear programming information satisfies the GPU's memory space constraints and the execution order constraints. The plurality of pre-assigned sub-models in the operating environment are trained according to the simulated training dataset; the operating environment is constructed according to preset hardware information; Obtain the single-round training time of the multiple pre-assigned sub-models; The hybrid linear programming information with the shortest single-round training time is taken as the target hybrid linear programming information.
[0007] In one possible implementation, before determining the GPU for training each sub-model based on the target interleaving mapping information, the method further includes: Based on the GPU topology corresponding to any type of interleaved mapping information, the degree of competition between sub-models is determined; the GPU topology includes whether the GPUs mapped by any two sub-models share the same root complex; when the GPUs mapped by two sub-models do not share the same root complex, the degree of competition between the two sub-models is minimized. The mapping scheme with the least competition is used as the target interleaved mapping information.
[0008] In one possible implementation, the degree of competition between sub-models satisfies the following relationship: in, C(i,j) For sub-model i and submodels j The level of competition, S(i,j) It represents the sub-model i and submodels j Do the mapped GPUs share the same root complex values? |i - j| In the sub-model i and submodels j The time difference between executions in the execution order.
[0009] In one possible implementation, the step of training the sub-models separately using their respective GPUs on a real training dataset to obtain the trained models includes: According to the execution order, the first sub-model and the corresponding temporary data of the first sub-model are obtained from the central processing unit (CPU); The first sub-model and its corresponding temporary data are uploaded to the video memory space of the first GPU; the first GPU is the GPU used to compute the first sub-model; the first sub-model is one of the multiple sub-models that the first GPU is responsible for processing. The training parameters obtained by training the first sub-model using the first GPU are returned to the memory of the CPU; The parameters of the first sub-model are updated on the CPU according to the training parameters; The process continues until all sub-models have completed their parameter updates, resulting in the trained model.
[0010] In one possible implementation, the method further includes: After the first GPU trains the first sub-model, the memory space occupied by the first sub-model and its corresponding temporary data in the GPU's memory space is released.
[0011] In one possible implementation, the method further includes: If there is free memory in the first GPU, the second sub-model waiting to be computed and the temporary data of the second sub-model are stored in the memory of the first GPU; the first GPU is also the GPU used to compute the second sub-model.
[0012] In one possible implementation, the step of training the sub-models separately using their respective GPUs on a real training dataset to obtain the trained models includes: Based on the batch order and execution order of the real training dataset, the first sub-model and the corresponding temporary data are obtained from the central processing unit (CPU). The first sub-model and its corresponding temporary data are uploaded to the video memory of the first GPU in the order of the batch; the first GPU is the GPU used to compute the first sub-model; the first sub-model is one of the multiple sub-models that are processed on the first GPU. The training parameters obtained by training the first sub-model by the first GPU in the batch order are returned to the memory of the first CPU; Update the parameters of the first sub-model corresponding to the batch order on the first CPU according to the training parameters corresponding to the batch order; This process continues until all sub-models in different batch orders have completed parameter updates, resulting in trained models in different batch orders.
[0013] In one possible implementation, the limitation information of the GPU's video memory space is as follows: None of the following can exceed the maximum capacity of the GPU's video memory: The memory overhead of the sub-model in the GPU's memory space during the training phase; The amount of temporary data in the GPU's video memory space; The amount of data that will be uploaded from the CPU to the GPU's memory space, including the sub-model and the temporary data of the sub-model. The amount of video memory used by the system.
[0014] In one possible implementation, the execution order constraint information includes: After the sub-model to be computed and its temporary data are all transferred from the CPU's memory to the GPU's video memory, the execution order of the sub-model is either forward propagation or backward propagation.
[0015] In one possible implementation, the method further includes: When sampling multiple computational layers with similar structures and computations in the sub-model, the layer memory overhead and layer time overhead of any one of the multiple computational layers are sampled during the forward or backward propagation process. The average memory overhead and computation time of each computation layer are calculated separately to obtain the average layer memory overhead and average layer time overhead of each computation layer. The memory and time overhead of the sub-model are calculated based on the average memory and time overhead of each computational layer in the sub-model; the single-round training time includes the time overhead of training the sub-model.
[0016] On the other hand, embodiments of this application provide a deep neural network model training apparatus, including: A partitioning module is used to divide a target deep neural network model into multiple sub-models based on target mixed linear programming information. Each sub-model contains one computational layer or multiple consecutive computational layers in the target deep neural network model. The target mixed linear programming information is information that satisfies the limitations of the video memory space of a general-purpose graphics processing unit (GPU) and the limitations of the execution order of the computational layers in the target deep neural network model, and that minimizes the single-round training time of the target deep neural network model. The single-round training time is the time required to train all computational layers in the target deep neural network model once in the execution order. The determination module is used to determine the GPU used to train each of the sub-models based on the target interleaving mapping information; the target interleaving mapping information is the interleaving mapping information with the least competition among the multiple sub-models; The training module is used to train the sub-models separately using the GPUs of each sub-model based on the real training dataset, so as to obtain the trained model.
[0017] In another aspect, embodiments of this application provide a model training device, the device comprising: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the deep neural network model training method as described in any of the above.
[0018] In another aspect, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the deep neural network model training method as described in any of the above claims.
[0019] In another aspect, embodiments of this application provide a computer program product, wherein the instructions in the computer program product, when executed by the processor of an electronic device, cause the electronic device to perform the deep neural network model training method as described in any of the above claims.
[0020] The deep neural network model training method and apparatus of this application divides the target deep neural network model into multiple sub-models based on the target hybrid linear programming information. By dividing the target deep neural network model, the memory capacity requirement of a single GPU for training is reduced. Based on the target interleaved mapping information with the lowest degree of competition among the sub-models, the GPU used to train each sub-model is determined, thereby alleviating the communication competition between the sub-models. Then, based on the real training dataset, multiple sub-models are trained to obtain a training model with better training performance. Attached Figure Description
[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 This is a flowchart illustrating the deep neural network model training method provided in the embodiments of this application; Figure 2 This is a flowchart of the model training provided in the embodiments of this application; Figure 3 This is a schematic diagram showing the GPU memory space usage provided in an embodiment of this application; Figure 4 This is a schematic diagram illustrating the execution order provided in the embodiments of this application; Figure 5 This is a schematic diagram of a similar layer provided in an embodiment of this application; Figure 6 This is a schematic diagram of the GPU topology array provided in an embodiment of this application; Figure 7 This is a schematic diagram of the CPU-to-GPU data transfer queue provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of the deep neural network model training device provided in the embodiments of this application; Figure 9 This is a schematic diagram of the structure of the model training device provided in the embodiments of this application. Detailed Implementation
[0023] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.
[0024] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.
[0025] Related technologies, such as the background technology, provide a method and apparatus for training a deep neural network model, which trains the model based on target hybrid linear programming information. Target deep neural network model Divided into multiple sub-models, through... Target Deep neural network model The partitioning reduces the memory requirements of a single GPU used for training. Based on the target interleaving mapping information with the least competition among sub-models, the GPU used to train each sub-model is determined, alleviating communication competition between sub-models. Then, based on the real training dataset, multiple sub-models are trained to obtain a training model with better training performance.
[0026] To facilitate understanding of this application, the following terms will be explained first.
[0027] Forward propagation: The training data is input into the input layer of a deep neural network (DNN), passes through the hidden layers, and finally reaches the output layer to output the result.
[0028] Backpropagation: Since there is an error between the input and output of the DNN, the error between the estimated value and the actual value is calculated, and this error is propagated back from the output layer to the hidden layer until it reaches the input layer.
[0029] Heterogeneous storage: Employing a novel horizontal cascading model, it takes over storage devices of various brands, models, and interfaces online, breaking down capacity barriers between storage devices and enabling centralized scheduling of cross-device storage resources. Storage virtualization technology, combined with multiple advanced features, ensures efficient utilization and reliable management of storage resources.
[0030] Pipeline parallelism: Based on model parallelism, the next batch begins before the previous batch finishes, making full use of computing resources. The model is split into layers, and different layers are placed on different GPUs. During training, data flows across the GPUs like a stream.
[0031] To address the problems of existing technologies, embodiments of this application provide a method, apparatus, device, and computer storage medium for training deep neural network models. The following is in conjunction with… Figure 1 The deep neural network model training method provided in the embodiments of this application will be introduced.
[0032] Figure 1 This is a flowchart illustrating the deep neural network model training method provided in this application embodiment, as shown below. Figure 1 As shown, the deep neural network model training method provided in this application embodiment includes the following steps: S101 to S103.
[0033] S101, based on the target hybrid linear programming information, divide the target deep neural network model into multiple sub-models; Each sub-model contains one or multiple consecutive computational layers from the target deep neural network model. The target hybrid linear programming information satisfies the limitations of the GPU's memory space and the execution order of the computational layers in the target deep neural network model, while minimizing the single-round training time of the target deep neural network model. The single-round training time is the time required to train all computational layers in the target deep neural network model once, in the execution order.
[0034] S102, Based on the target interleaving mapping information, determine the GPU to be used to train each sub-model; As a possible implementation of S102, each sub-model is mapped to a GPU based on the target interleaving mapping information, and each GPU is responsible for training multiple sub-models. The target interleaving mapping information is the interleaving mapping information with the least competition among multiple sub-models.
[0035] S103: Based on the real training dataset, train each sub-model separately using the GPU of each sub-model to obtain the trained model.
[0036] As a possible implementation of S103, the target deep neural network model is trained using a pipelined parallel approach that supports heterogeneous storage, based on a real training dataset.
[0037] In one example, the pipelined parallel training process supporting heterogeneous storage is as follows: During model initialization, model data and optimizer parameters are stored in CPU memory. Before the GPU trains a sub-model, the sub-model data and necessary temporary data are uploaded to GPU memory. After the GPU completes training the sub-model, the trained parameters are written back to CPU memory, and the occupied GPU memory space is released. The sub-model updates its parameters on the CPU based on the trained parameters. After the parameter update is complete, the next round of training begins.
[0038] The model parameters and optimizer parameters are stored in CPU memory. During training, before computation begins, each sub-model transfers its necessary model data and temporary data to the corresponding GPU memory. The CPU memory stores the full-precision model. During training, users can choose to use different precision levels, such as full-precision or half-precision, during transfer and computation. Before forward propagation, the system transfers the sub-model's model data to GPU memory. After forward propagation, the system transfers the activation values generated by the sub-model to CPU memory for backpropagation. Before backpropagation, the system transfers the sub-model's model data and activation values generated during forward propagation to GPU memory. After backpropagation, the system transfers the model gradients calculated by the sub-model to CPU memory. Once the gradients of a sub-model have been transferred to CPU memory, the system calculates the model parameters to be used in the next round based on these gradients.
[0039] Figure 2 This is a flowchart of the model training provided in the embodiments of this application, such as... Figure 2As shown, the N-layer deep learning model divides the target into multiple sub-models by mixing linear programming information. The first sub-model includes layers 1 and 2, the second sub-model includes layers 3 and 4, the third sub-model includes layers 5 and 6, the first and third sub-models are mapped to GPU #1, the second sub-model is mapped to GPU #2, and so on. Figure 2 This is merely an example and does not limit the structure of the training sub-models in this application.
[0040] The above describes a deep neural network model training method provided in this application embodiment. Based on the target hybrid linear programming information, the target deep neural network model is divided into multiple sub-models. By dividing the target deep neural network model, the memory capacity requirement of a single GPU used for training is reduced. Based on the target interleaved mapping information with the lowest degree of competition among sub-models, the GPU used to train each sub-model is determined, alleviating communication competition between sub-models. Then, based on the real training dataset, multiple sub-models are trained to obtain a training model with better training performance.
[0041] In one possible implementation, before dividing the target deep neural network model into multiple sub-models based on the target hybrid linear programming information, the method further includes: The target deep neural network model is divided according to any kind of hybrid linear programming information to obtain multiple pre-assigned sub-models; As one possible implementation, any hybrid linear programming information satisfies the GPU's memory space constraints and execution order constraints. The target deep neural network model is decomposed into several sub-models using any hybrid linear programming information. Each sub-model contains one layer or multiple consecutive layers of the target deep neural network model, and the sub-models do not overlap with each other.
[0042] Train multiple pre-assigned sub-models in the runtime environment based on the simulated training dataset; As one possible implementation, the runtime environment is built based on pre-defined hardware information. This hardware information is either input by the user or obtained from the operating system.
[0043] In one example, the hardware information includes: the GPU memory capacity corresponding to the hardware device, the GPU's computing power, the inter-GPU communication bandwidth, the communication bandwidth between the CPU and the GPU, and the GPU's topology.
[0044] Obtain the single-round training time for multiple pre-assigned sub-models; The mixed linear programming information with the shortest single-round training time is taken as the target mixed linear programming information.
[0045] Therefore, by comparing the single-round training time of various hybrid linear programming information, the target hybrid linear programming information with the shortest single-round training time is determined.
[0046] In one possible implementation, before determining the GPU to train each sub-model based on the target interleaving mapping information, the method further includes: Determine the degree of competition between sub-models based on the GPU topology corresponding to any type of interleaved mapping information; As one possible implementation, the system searches for interleaved mapping information based on the GPU's topological grid.
[0047] In one example, the GPU topology includes whether any two sub-models are mapped to GPUs that share the same root complex. The level of competition between two sub-models is minimized when the GPUs mapped to them do not share the same root complex. Each sub-model corresponds to one GPU, and each GPU can correspond to one or more sub-models.
[0048] The mapping scheme with the least competition is used as the target interleaved mapping information.
[0049] Therefore, by comparing the degree of competition among various interleaved mapping information, the target interleaved mapping information with the lowest degree of competition is determined.
[0050] In one possible implementation, the degree of competition between sub-models satisfies the following relationship: in, C(i,j) The degree of competition between sub-model i and sub-model j. S(i,j) It represents the sub-model i and submodels j Do the mapped GPUs share the same root complex value? When two GPUs share the same root complex, S(i,j) It is 1 if it is not 1, otherwise it is 0. |i - j| This indicates that in submodel i and submodel... j The time difference between executions in the execution order. A larger execution time difference means less communication contention, and vice versa.
[0051] Therefore, according to the sub-model i and submodels j Do the mapped GPUs share the same root complex values and sub-models? i and submodels j Determine the sub-model based on the time difference in the execution sequence. i and submodels j The level of competition.
[0052] The topology of a GPU is represented by an array of GPU topologies. GPUs located in the same array share a root complex and will share the communication bandwidth if they communicate simultaneously.
[0053] Figure 6 This is a schematic diagram of the GPU topology array provided in the embodiments of this application, such as... Figure 6 As shown, the topology arrays are [1,2] and [3,4]. GPU#1 and GPU#2 share the root complex GPU#1, and GPU#3 and GPU#4 share the root complex GPU#3.
[0054] The system prioritizes using the GPU topology input by the user. If the user does not input a GPU topology, the system attempts to obtain hardware information to derive the GPU topology. If the system cannot obtain hardware information, it attempts to communicate between different GPUs and collect bandwidth data to infer possible GPU topologies.
[0055] In one possible implementation, the sub-models are trained separately using GPUs on each sub-model based on a real training dataset to obtain a trained model, including: According to the execution order, the first sub-model and the corresponding temporary data of the first sub-model are obtained from the central processing unit (CPU); The first sub-model and its corresponding temporary data are uploaded to the video memory space of the first GPU; the first GPU is the GPU in the GPU used to compute the first sub-model; the first sub-model is one of the multiple sub-models that the first GPU is responsible for processing. The training parameters obtained by training the first sub-model on the first GPU are returned to the CPU's memory; Update the parameters of the first sub-model on the CPU based on the training parameters; The process continues until all sub-models have completed their parameter updates, resulting in the trained model.
[0056] Therefore, in the order of execution, the sub-models are trained separately using the GPUs of each sub-model to obtain the trained model.
[0057] In one possible implementation, the method also includes: After the first sub-model is trained on the first GPU, the memory space occupied by the first sub-model and its corresponding temporary data in the GPU's memory space is released.
[0058] Therefore, by training the first sub-model on the first GPU, the occupied GPU memory space is released. In one possible implementation, the method also includes: When there is free memory in the first GPU, the second sub-model waiting to be computed and its temporary data are pre-acquired and stored in the memory of the first GPU. The computation of the first sub-model masks the transmission of data related to the second sub-model. The first GPU is also the GPU used to compute the second sub-model.
[0059] While the GPU is training the sub-model, the idle GPU memory space is used to prefetch the data for the next sub-model to be trained. Each GPU corresponds to a CPU-to-GPU data transfer queue, which stores the addresses of the data to be transferred in CPU memory and is arranged according to the pipeline execution order.
[0060] Figure 7 This is a schematic diagram of the CPU-to-GPU data transfer queue provided in an embodiment of this application, as shown below. Figure 7 As shown, queue #1 contains the sub-model data to be trained by GPU #1, queue #2 contains the sub-model data to be trained by GPU #2, queue #3 contains the sub-model data to be trained by GPU #3, and queue #4 contains the sub-model data to be trained by GPU #4. The sub-model data to be trained is stored in CPU memory.
[0061] The system estimates the maximum GPU memory required for the current sub-model computation based on previous sampling information. When there is free GPU memory, the system retrieves data from the queue that can be stored in the available memory space and transfers it from CPU memory to GPU memory. The data transfer process and the computation of the current sub-model use different streams, and computation and transfer are executed in parallel.
[0062] Therefore, by storing the second sub-model waiting to be computed and its temporary data in the memory space of the first GPU in advance, the training time of the model is reduced and the training efficiency is improved.
[0063] In one possible implementation, the sub-models are trained separately using GPUs on each sub-model based on a real training dataset to obtain a trained model, including: Based on the batch order and execution order of the real training dataset, obtain the first sub-model and the corresponding temporary data from the central processing unit (CPU); The first sub-model and its corresponding temporary data are uploaded to the video memory space of the first GPU in batch order; the first GPU is the GPU used to compute the first sub-model; the first sub-model is one of the multiple sub-models that the first GPU is responsible for processing. The training parameters obtained by training the first sub-model on the first GPU in batch order are returned to the memory of the first CPU. Update the parameters of the first sub-model corresponding to the batch order on the first CPU according to the training parameters corresponding to the batch order; This process continues until all sub-models in different batch orders have completed parameter updates, resulting in trained models in different batch orders.
[0064] One possible implementation is to divide the training data into multiple batches for each round, with training of each batch of data executed sequentially on each sub-model. Different sub-models can train different batches of data simultaneously in the order of execution. On a single GPU, a single sub-model must complete the forward propagation of all batches of data in a single round before proceeding to the forward propagation of the next sub-model; similarly, a single sub-model must complete the backward propagation of all batches of data in a single round before proceeding to the backward propagation of the next sub-model.
[0065] Before the forward or backward propagation computation of a sub-model begins, the system checks whether all data for that sub-model has been prefetched into the GPU memory before computation. If the sub-model data is incomplete, the computation will be paused, and the system will actively transfer data from the CPU memory to the corresponding GPU memory. Only after all data has been transferred to the GPU memory can the sub-model begin forward or backward propagation computation.
[0066] Therefore, training efficiency is improved by using different GPUs to process different batches of data in parallel.
[0067] In one possible implementation, the GPU's memory space limitations are as follows: None of the following can exceed the maximum capacity of the GPU's video memory: The memory overhead of the sub-model in the GPU's memory space during the training phase; The amount of temporary data in the GPU's video memory space; The amount of data that will be uploaded from the CPU to the GPU's memory space, including the sub-model and the temporary data of the sub-model. The amount of video memory used by the system.
[0068] Figure 3 This is a schematic diagram illustrating the GPU memory space usage provided in an embodiment of this application, as shown below. Figure 3 As shown, within a given GPU, the system estimates the maximum GPU memory overhead for the currently trained sub-model during training based on its model parameters and computation mode. The size of the free GPU memory space is the total GPU memory space capacity minus the maximum GPU memory overhead and the system's running memory overhead. This free GPU memory space is used to store the data for the next sub-model to be trained on this GPU, along with related temporary data.
[0069] The total amount of the next sub-model to be trained and temporary data prefetched on this GPU cannot exceed the size of the free GPU memory. During the prefetching process, sub-model data is prefetched first, followed by the temporary data required for training that sub-model. Temporary data corresponding to batches computed earlier is prefetched first. When the free GPU memory cannot fully prefetch the data and related temporary data of the next sub-model to be computed on this GPU, the system only prefetches the amount of model data and related temporary data of the next sub-model that can be stored in the free GPU memory. The remaining data of the next sub-model and related temporary data not yet transferred to the GPU memory will be transferred to the GPU memory after the current sub-model computation is completed and the memory space is released.
[0070] Therefore, by limiting the mixing mapping information using the GPU's video memory space, the time required to determine the target mixing mapping information is shortened.
[0071] In one possible implementation, the execution order constraint information includes: After the sub-models to be computed and their temporary data are all transferred from the CPU's memory to the GPU's video memory, the execution order of the sub-models is either forward propagation or backward propagation.
[0072] In forward propagation, the current sub-model begins forward propagation only after the previous sub-model finishes computation and transfers its activations to the GPU memory mapped to that sub-model. In backpropagation, the current sub-model begins backpropagation only after the next sub-model finishes computation and transfers its errors to the GPU memory mapped to that sub-model. Training on different batches of data is executed sequentially on a single sub-model, and training on different batches of data can be executed concurrently on different sub-models. Only one sub-model is trained on a GPU at a time. During the forward propagation phase, on a single GPU, a sub-model can only begin forward propagation on the next sub-model after processing all batches of data in one round of forward propagation. Similarly, on a single GPU, a sub-model can only begin backpropagation on the next sub-model after processing all batches of data in one round of backpropagation.
[0073] Figure 4 This is a schematic diagram illustrating the execution order provided in the embodiments of this application, such as... Figure 4 As shown, F i,j B represents the forward propagation computation of sub-model i on the j-th batch of data; i,j C represents the backpropagation computation of submodel i on the j-th batch of data; iThis indicates that sub-model i is transferred from CPU memory to GPU memory.
[0074] For a given sub-model, after the activation values generated by the forward propagation of the previous sub-model are transferred to the GPU memory of that sub-model, the sub-model needs to use these activation values as input to perform the corresponding forward propagation calculation. Except for the last sub-model, after the calculation is complete, the generated activation values need to be transferred to the GPU of the next sub-model. After the error generated by the backpropagation of the next sub-model is transferred to the GPU memory of that sub-model, the sub-model needs to use this error as input to perform the corresponding backpropagation calculation. Except for the first sub-model, the calculation error needs to be transferred to the GPU of the previous sub-model.
[0075] Therefore, restricting the order of model training reduces the error rate of model training.
[0076] In one possible implementation, the method also includes: When sampling multiple computational layers with similar structures and computations in a sub-model, sample the layer memory overhead and layer time overhead of any one of the multiple computational layers during the forward or backward propagation process. The memory overhead and computation time of each computation layer are averaged to obtain the average layer memory overhead and average layer time overhead for each computation layer. The memory and time costs of the sub-model are calculated based on the average memory and time costs of each computational layer in the sub-model; the training time per round includes the time cost of training the sub-model.
[0077] As one possible implementation, hybrid linear programming requires sampling at each layer of the model to collect the changes in memory overhead and computation time at each layer during forward and backward propagation.
[0078] The system generates a set of simulated data based on the input training hyperparameters to simulate the input of real training. Before real training, the system feeds the simulated data into the model and collects information on the changes in memory overhead and computation time for each layer under these training hyperparameters. After each layer is trained multiple times in a simulated manner, the average value of the sampled data is taken.
[0079] The system uses a single GPU to sample the model during training. Initially, the model is stored in CPU memory. Then, each layer of the model is sequentially transferred to the GPU for simulation training. After sampling of a layer is completed, that layer is released from the GPU memory, and the next layer is transferred to the GPU memory for sampling. The forward propagation activations and backpropagation errors transferred between layers are stored in the GPU memory until they are no longer needed.
[0080] Figure 5 This is a schematic diagram of a similar layer provided in an embodiment of this application, such as... Figure 5 As shown, during sampling, simulated training data is used to train the deep learning model, and the computation time and memory overhead of forward propagation and backward propagation are collected. For layers with similar structure and computation, only representative layers are sampled, while for other layers, the representative layers are reused to collect the computation time and memory overhead data of forward propagation and backward propagation.
[0081] Before sampling begins, the system compares the dimensions and computational tasks of each layer of the model. Layers with the same dimensions and computational tasks are grouped into similar layers. During the actual sampling process, only one representative layer in a similar group needs to be sampled. Other layers in the similar group can skip the sampling calculation and directly reuse the sampled data from the representative layer.
[0082] Therefore, by reducing the sampling of computational layers, the efficiency of model training can be improved.
[0083] The above describes the specific implementation of model training in this application. Accordingly, this application also provides a deep neural network model training device.
[0084] Figure 8 This is a schematic diagram of the structure of the deep neural network model training device provided in the embodiments of this application, as shown below. Figure 8 As shown, the deep neural network model training device 800 includes: The partitioning module 801 is used to divide the target deep neural network model into multiple sub-models based on the target mixed linear programming information. Each sub-model contains one computational layer or multiple consecutive computational layers from the target deep neural network model. The target mixed linear programming information is information that satisfies the limitations of the GPU's video memory space and the limitations of the execution order of the computational layers in the target deep neural network model, and minimizes the single-round training time of the target deep neural network model. The single-round training time is the time required to train all computational layers in the target deep neural network model once in the execution order. The determination module 802 is used to determine the GPU used to train each sub-model based on the target interleaving mapping information; the target interleaving mapping information is the interleaving mapping information with the least competition among multiple sub-models; Training module 803 is used to train the sub-models separately using the GPUs of each sub-model based on the real training dataset, so as to obtain the training model.
[0085] The above describes a deep neural network model training device provided in this application embodiment. Based on the target hybrid linear programming information, the target deep neural network model is divided into multiple sub-models. By dividing the target deep neural network model, the memory capacity requirement of a single GPU used for training is reduced. Based on the target interleaved mapping information with the lowest degree of competition among sub-models, the GPU used to train each sub-model is determined, alleviating communication competition between sub-models. Then, based on the real training dataset, multiple sub-models are trained to obtain a training model with better training performance.
[0086] In one possible implementation, before dividing the target deep neural network model into multiple sub-models based on the target hybrid linear programming information, the deep neural network model training device 800 may further include: The target hybrid linear programming information determination module is used to divide the target deep neural network model according to any hybrid linear programming information to obtain multiple pre-allocated sub-models; any hybrid linear programming information satisfies the GPU's memory space constraints and execution order constraints. The simulation training dataset is used to train multiple pre-assigned sub-models in the runtime environment; the runtime environment is constructed based on preset hardware information. Obtain the single-round training time for multiple pre-assigned sub-models; The mixed linear programming information with the shortest single-round training time is taken as the target mixed linear programming information.
[0087] Therefore, by comparing the single-round training time of various hybrid linear programming information, the target hybrid linear programming information with the shortest single-round training time is determined.
[0088] In one possible implementation, before determining the GPU for training each sub-model based on the target interleaving mapping information, the deep neural network model training device 800 may further include: The target interleaving mapping information determination module is used to determine the degree of competition between sub-models based on the GPU topology corresponding to any interleaving mapping information; the GPU topology includes whether the GPUs mapped by any two sub-models share the same root complex; when the GPUs mapped by two sub-models do not share the same root complex, the degree of competition between the two sub-models is minimized. The mapping scheme with the least competition is used as the target interleaved mapping information.
[0089] Therefore, by comparing the degree of competition among various interleaved mapping information, the target interleaved mapping information with the lowest degree of competition is determined.
[0090] In one possible implementation, the degree of competition between sub-models satisfies the following relationship: in, C(i,j) For sub-model i and submodels j The level of competition, S(i,j) It represents the sub-model i and submodels j Do the mapped GPUs share the same root complex values? |i - j| In the sub-model i and submodels j The time difference between executions in the execution order.
[0091] Therefore, according to the sub-model i and submodels j Do the mapped GPUs share the same root complex values and sub-models? i and submodels j Determine the sub-model based on the time difference in the execution sequence. i and submodels j The level of competition.
[0092] In one possible implementation, the sub-models are trained separately using GPUs on each sub-model based on a real training dataset to obtain a trained model, including: According to the execution order, the first sub-model and the corresponding temporary data of the first sub-model are obtained from the central processing unit (CPU); The first sub-model and its corresponding temporary data are uploaded to the video memory space of the first GPU; the first GPU is the GPU in the GPU used to compute the first sub-model; the first sub-model is one of the multiple sub-models that the first GPU is responsible for processing. The training parameters obtained by training the first sub-model on the first GPU are returned to the CPU's memory; Update the parameters of the first sub-model on the CPU based on the training parameters; The process continues until all sub-models have completed their parameter updates, resulting in the trained model.
[0093] Therefore, in the order of execution, the sub-models are trained separately using the GPUs of each sub-model to obtain the trained model.
[0094] In one possible implementation, the deep neural network model training device 800 may further include: The release module is used to release the GPU memory space occupied by the first sub-model and its corresponding temporary data after the first GPU has trained the first sub-model.
[0095] Therefore, by training the first sub-model on the first GPU, the occupied GPU memory space is released. In one possible implementation, the deep neural network model training device 800 may further include: The storage module is used to pre-acquire and store the second sub-model waiting to be computed and its temporary data in the memory space of the first GPU when there is free memory space in the first GPU, thereby masking the transmission of data related to the second sub-model through the computation of the first sub-model; the first GPU is also the GPU in the GPU used to compute the second sub-model.
[0096] Therefore, by storing the second sub-model waiting to be computed and its temporary data in the memory space of the first GPU in advance, the training time of the model is reduced and the training efficiency is improved.
[0097] In one possible implementation, the training module 803 can also be used for: Based on the batch order and execution order of the real training dataset, obtain the first sub-model and the corresponding temporary data from the central processing unit (CPU); The first sub-model and its corresponding temporary data are uploaded to the video memory space of the first GPU in batch order; the first GPU is the GPU used to compute the first sub-model; the first sub-model is one of the multiple sub-models that the first GPU is responsible for processing. The training parameters obtained by training the first sub-model on the first GPU in batch order are returned to the memory of the first CPU. Update the parameters of the first sub-model corresponding to the batch order on the first CPU according to the training parameters corresponding to the batch order; This process continues until all sub-models in different batch orders have completed parameter updates, resulting in trained models in different batch orders.
[0098] Therefore, training efficiency is improved by using different GPUs to process different batches of data in parallel.
[0099] In one possible implementation, the GPU's memory space limitations are as follows: None of the following can exceed the maximum capacity of the GPU's video memory: The memory overhead of the sub-model in the GPU's memory space during the training phase; The amount of temporary data in the GPU's video memory space; The amount of data that will be uploaded from the CPU to the GPU's memory space, including the sub-model and the temporary data of the sub-model. The amount of video memory used by the system.
[0100] Therefore, by limiting the mixing mapping information using the GPU's video memory space, the time required to determine the target mixing mapping information is shortened.
[0101] In one possible implementation, the execution order constraint information includes: After the sub-models to be computed and their temporary data are all transferred from the CPU's memory to the GPU's video memory, the execution order of the sub-models is either forward propagation or backward propagation.
[0102] Therefore, restricting the order of model training reduces the error rate of model training.
[0103] In one possible implementation, the deep neural network model training device 800 may further include: The sampling module is used to sample the memory overhead and time overhead of any one of the multiple computational layers that are similar in structure and computation in the sub-model during the forward or backward propagation process. The memory overhead and computation time of each computation layer are averaged to obtain the average layer memory overhead and average layer time overhead for each computation layer. The memory and time costs of the sub-model are calculated based on the average memory and time costs of each computational layer in the sub-model; the training time per round includes the time cost of training the sub-model.
[0104] Therefore, by reducing the sampling of computational layers, the efficiency of model training can be improved.
[0105] The above describes the specific implementation of model training in this application. Accordingly, this application also provides a model training device.
[0106] Figure 9 This is a schematic diagram of the structure of the model training device provided in the embodiments of this application, as shown below. Figure 9 As shown, the model training device may include a processor 901 and a memory 902 storing computer program instructions.
[0107] Specifically, the processor 901 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.
[0108] Memory 902 may include mass storage for data or instructions. For example, and not limitingly, memory 902 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 902 may include removable or non-removable (or fixed) media. Where appropriate, memory 902 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 902 is non-volatile solid-state memory.
[0109] Memory 902 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this application.
[0110] The processor 901 reads and executes computer program instructions stored in the memory 902 to implement any of the deep neural network model training methods in the above embodiments.
[0111] In one example, the model training device may also include a communication interface 903 and a bus 910. For example, Figure 9 As shown, the processor 901, memory 902, and communication interface 903 are connected through bus 910 and complete communication with each other.
[0112] The communication interface 903 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.
[0113] Bus 910 includes hardware, software, or both, that couples components of a model training device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 910 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.
[0114] This model training device can execute the deep neural network model training method in the embodiments of this application, thereby achieving a combination of Figure 1 and Figure 8 The method and apparatus for training deep neural network models are described.
[0115] In addition, in conjunction with the deep neural network model training method in the above embodiments, this application also provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the deep neural network model training method as described above.
[0116] In conjunction with the deep neural network model training method in the above embodiments, this application also provides a computer program product, wherein when the instructions in the computer program product are executed by the processor of an electronic device, the electronic device executes the deep neural network model training method as described above.
[0117] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.
[0118] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.
[0119] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.
[0120] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.
[0121] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.
Claims
1. A method for training a deep neural network model, the method comprising: include: Based on the target hybrid linear programming information, the target deep neural network model is divided into multiple sub-models; each sub-model contains one computational layer or multiple consecutive computational layers in the target deep neural network model; the target hybrid linear programming information is information that satisfies the limitations of the video memory space of a general-purpose graphics processing unit (GPU) and the limitations of the execution order of the computational layers in the target deep neural network model, and information that minimizes the single-round training time of the target deep neural network model; the single-round training time is the time required to train all computational layers in the target deep neural network model once in the execution order; Based on the target interleaving mapping information, the GPU used to train each of the sub-models is determined; the target interleaving mapping information is the interleaving mapping information with the least competition among the multiple sub-models, and the competition between the two sub-models is minimized when the GPUs mapped by the two sub-models do not share the same root complex. Based on the real training dataset, the sub-models are trained separately using the GPUs of each sub-model to obtain the trained models; The step of training the sub-models using GPUs on each sub-model based on a real training dataset to obtain a training model includes: obtaining a first sub-model and its corresponding temporary data from the CPU in execution order; uploading the first sub-model and its corresponding temporary data to the video memory of a first GPU; the first GPU being the GPU used to compute the first sub-model; the first sub-model being one of multiple sub-models processed on the first GPU; returning the training parameters obtained by training the first sub-model on the first GPU to the memory of the CPU; updating the parameters of the first sub-model on the CPU according to the training parameters; and so on until all sub-models have completed parameter updates to obtain the training model. 2.The method of Claim 1, wherein, Before dividing the target deep neural network model into multiple sub-models based on the target hybrid linear programming information, the method further includes: The target deep neural network model is divided according to any one of the hybrid linear programming information to obtain multiple pre-allocated sub-models; the hybrid linear programming information satisfies the GPU's memory space constraints and the execution order constraints. The multiple pre-assigned sub-models in the runtime environment are trained based on the simulated training dataset; the runtime environment is constructed based on preset hardware information. Obtain the single-round training time of the multiple pre-assigned sub-models; The hybrid linear programming information with the shortest single-round training time is taken as the target hybrid linear programming information. 3.The method of Claim 1, wherein, Before determining the GPU for training each sub-model based on the target interleaving mapping information, the method further includes: Based on the GPU topology corresponding to any type of interleaved mapping information, the degree of competition between sub-models is determined; the GPU topology includes whether the GPUs mapped by any two sub-models share the same root complex; The mapping scheme with the least competition is used as the target interleaved mapping information. 4.The method of Claim 3, wherein, The degree of competition between sub-models satisfies the following relationship: in, C(i,j) For sub-model i and submodels j The level of competition, S(i,j) It represents the sub-model i and submodels j Do the mapped GPUs share the same root complex values? |i - j| In the sub-model i and submodels j The time difference between executions in the execution order. 5.The method of Claim 1, wherein The method further includes: After the first GPU trains the first sub-model, the memory space occupied by the first sub-model and its corresponding temporary data in the GPU's memory space is released. 6.The method of Claim 1, wherein, The method further includes: If there is free memory in the first GPU, the second sub-model waiting to be computed and the temporary data of the second sub-model are stored in the memory of the first GPU; the first GPU is also the GPU used to compute the second sub-model. 7.The method of Claim 1, wherein The step of training the sub-models using GPUs on each sub-model based on a real training dataset to obtain the trained model includes: Based on the batch order and execution order of the real training dataset, the first sub-model and the corresponding temporary data are obtained from the central processing unit (CPU). The first sub-model and its corresponding temporary data are uploaded to the video memory of the first GPU in the order of the batch; the first GPU is the GPU used to compute the first sub-model; the first sub-model is one of the multiple sub-models that are processed on the first GPU. The training parameters obtained by training the first sub-model on the first GPU in the batch order are returned to the memory of the CPU; Update the parameters of the first sub-model corresponding to the batch order on the CPU according to the training parameters corresponding to the batch order; This process continues until all sub-models in different batch orders have completed parameter updates, resulting in trained models in different batch orders. 8.The method of any one of claims 1-7, wherein The limitations on the GPU's video memory space are as follows: None of the following can exceed the maximum capacity of the GPU's video memory: The memory overhead of the sub-model in the GPU's memory space during the training phase; The amount of temporary data in the GPU's video memory space; The amount of data that will be uploaded from the CPU to the GPU's memory space, including the sub-model and the temporary data of the sub-model. The amount of video memory used by the system. 9.The method of any one of claims 1-7, wherein The execution order restriction information includes: After the sub-model to be computed and its temporary data are all transferred from the CPU's memory to the GPU's video memory, the execution order of the sub-model is either forward propagation or backward propagation. 10.The method of Claim 1, wherein The method further includes: When sampling multiple computational layers with similar structures and computations in the sub-model, the layer memory overhead and layer time overhead of any one of the multiple computational layers are sampled during the forward or backward propagation process. The average memory overhead and computation time of each computation layer are calculated separately to obtain the average layer memory overhead and average layer time overhead of each computation layer. The memory and time overhead of the sub-model are calculated based on the average memory and time overhead of each computational layer in the sub-model; the single-round training time includes the time overhead of training the sub-model.
11. A deep neural network model training apparatus, characterized by comprising: include: A partitioning module is used to divide a target deep neural network model into multiple sub-models based on target mixed linear programming information. Each sub-model contains one computational layer or multiple consecutive computational layers in the target deep neural network model. The target mixed linear programming information is information that satisfies the limitations of the video memory space of a general-purpose graphics processing unit (GPU) and the limitations of the execution order of the computational layers in the target deep neural network model, and that minimizes the single-round training time of the target deep neural network model. The single-round training time is the time required to train all computational layers in the target deep neural network model once in the execution order. The determination module is used to determine the GPU used to train each of the sub-models based on the target interleaving mapping information; the target interleaving mapping information is the interleaving mapping information with the least competition among the multiple sub-models, and the competition between the two sub-models is minimized when the GPUs mapped by the two sub-models do not share the same root complex. The training module is used to train the sub-models separately using the GPUs of each sub-model based on the real training dataset, so as to obtain the training model; Specifically, the training module is used to: obtain a first sub-model and its corresponding temporary data from the central processing unit (CPU) according to the execution order; upload the first sub-model and its corresponding temporary data to the video memory space of a first GPU; the first GPU is the GPU used to compute the first sub-model; the first sub-model is one of multiple sub-models processed on the first GPU; return the training parameters obtained by the first GPU in training the first sub-model to the memory of the CPU; update the parameters of the first sub-model on the CPU according to the training parameters; until all sub-models have completed parameter updates, a trained model is obtained.