Method, device, electronic device and storage medium for training large model in wide area network

Abstract

The present application relates to the field of artificial intelligence technology, and specifically provides a method, device, electronic device and storage medium for training a large model in a wide area network. A method for training a large model in a wide area network, comprising performing the following steps for each target GPU in each GPU: performing forward and backward propagation processing on batch data, obtaining the first gradient value of the locally generated target gradient slice and the second gradient value of each first GPU in other wide area network nodes that is assigned the same gradient slice; performing parameter update according to the first gradient value and each second gradient value corresponding to the target gradient slice, and obtaining the parameter value of the first parameter slice corresponding to the corresponding optimizer slice; receiving the parameter value of the respective second parameter slice sent by each first GPU; receiving the parameter value of the respective third parameter slice sent by each second GPU in the same wide area network node. In this way, both the reduction of the video memory usage of each GPU and the improvement of the model training performance can be taken into account.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a method, device, electronic device and storage medium for training a large model in a wide area network. Background Art

[0002] In the model training scenario, since model training usually requires a huge amount of data support, in order to improve training efficiency, the amount of batch data processed simultaneously during training usually needs to be increased, which will significantly increase the burden on video memory. Furthermore, model training requires frequent calculation of gradients and optimizer states during the process, which will multiply the video memory required for training. When distributed model training is performed based on a wide area network to reduce existing memory usage, since nodes usually communicate frequently, there is a large communication delay and low training efficiency.

[0003] Therefore, in the process of training large models over wide area networks, how to balance reducing the memory usage of each graphics processing unit (GPU) and improving model training performance is a problem that needs to be solved. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method, device, electronic device and storage medium for training a large model over a wide area network, so as to take into account both the reduction of the video memory occupancy of each GPU and the improvement of the model training performance when training a large model over a wide area network.

[0005] On the one hand, an embodiment of the present application provides a method for training a large model in a wide area network, which is applied to a distributed model training system, wherein the distributed model training system includes multiple wide area network nodes; each wide area network node includes multiple graphics processors; all model parameters of the target model to be trained are divided into multiple parameter slices; each graphics processor stores all model parameters of the target model, gradient slices corresponding to some parameter slices, and optimizer slices corresponding to one parameter slice; each gradient slice is a set of gradients corresponding to each model parameter in the corresponding parameter slice; each optimizer slice is a set of optimizer states corresponding to each model parameter in the corresponding parameter slice; the gradient slices corresponding to each parameter slice are respectively assigned to each graphics processor in the same wide area network node; the optimizer slices corresponding to different parameter slices are respectively assigned to different graphics processors; including:

[0006] For each target GPU in each GPU, perform the following steps:

[0007] Perform forward and backward propagation on the batch data to be processed to obtain the first gradient value of the corresponding target gradient slice;

[0008] receiving second gradient values ​​of respective corresponding target gradient slices sent by at least one first graphics processor in other wide area network nodes; the first graphics processor is a graphics processor to which the same gradient slice as the target graphics processor is allocated;

[0009] Perform parameter update according to the first gradient value and at least one second gradient value corresponding to the target gradient slice, and obtain the parameter value of the first parameter slice corresponding to the corresponding optimizer slice;

[0010] receiving parameter values ​​of second parameter slices respectively corresponding to respective optimizer slices sent by at least one first graphics processor;

[0011] Receive parameter values ​​of third parameter slices corresponding to respective optimizer slices sent by at least one second graphics processor; the second graphics processor is a graphics processor located in the same wide area network node as the target graphics processor.

[0012] In one embodiment, before performing forward and backward propagation processing on the batch data to be processed, the method further includes:

[0013] Determining a first total number of model parameters and a second total number of graphics processors;

[0014] Obtaining a partition size according to the first total quantity and the second total quantity;

[0015] According to the partition size, each model parameter is divided to obtain multiple parameter slices; the number of model parameters contained in each parameter slice is the partition size.

[0016] In one implementation, before obtaining the partition size according to the first total number and the second total number, the method further includes:

[0017] If the first total number cannot be divided by the second total number, then fill in the parameters of each model;

[0018] A first total number of model parameters after filling is determined.

[0019] In one embodiment, before performing forward and backward propagation processing on the batch data to be processed, the method further includes:

[0020] Generate optimizer slices corresponding to each parameter slice;

[0021] Assign different optimizer slices to different GPUs;

[0022] Generate gradient slices corresponding to each parameter slice;

[0023] For each wide area network node, the following steps are performed respectively: each gradient slice is allocated to each graphics processor of the wide area network node; the parameter slice corresponding to the gradient slice allocated to each graphics processor includes the parameter slice corresponding to the optimizer slice allocated to the graphics processor.

[0024] In one embodiment, before receiving the second gradient value of the target gradient slice corresponding to each other sent by at least one first graphics processor in other wide area network nodes, the method includes:

[0025] According to the assigned gradient slices, the graphics processors are communicated and combined to obtain multiple inter-domain process groups to aggregate the gradient values ​​of the same gradient slice; the graphics processors in the same inter-domain process group are assigned the same gradient slice and are located in different wide area network nodes;

[0026] Establishing communication between each graphics processor in each inter-domain process group;

[0027] The graphics processors in each WAN node are respectively communicated and combined to obtain the intra-domain process groups corresponding to each WAN node, so as to summarize the parameter values ​​of each parameter slice in the same WAN node;

[0028] The communication between each graphics processor in the process group in each domain is established respectively.

[0029] In one implementation, performing parameter updating according to a first gradient value and at least one second gradient value corresponding to a target gradient slice to obtain a parameter value of a first parameter slice corresponding to a corresponding optimizer slice includes:

[0030] Performing gradient updating according to the first gradient value and the at least one second gradient value respectively corresponding to the at least one target gradient slice, to obtain a third gradient value for each target gradient slice respectively;

[0031] Based on the third gradient value of the target gradient slice corresponding to the first parameter slice, obtaining a first optimizer state value of the corresponding optimizer slice;

[0032] According to the third gradient value and the first optimizer state value, the parameters are updated to obtain the parameter value of the first parameter slice.

[0033] In one implementation, forward and backward propagation processing is performed on the batch data to be processed to obtain the first gradient value of the corresponding target gradient slice, including:

[0034] For each micro-batch of the batch data, the following steps are performed: forward and backward propagation processing is performed on the micro-batch data to obtain a fourth gradient value of a target gradient slice; a fifth gradient value of the target gradient slice is received from at least one first graphics processor; a sixth gradient value is obtained according to the fourth gradient value and at least one fifth gradient value;

[0035] When it is determined that the batch data processing is completed, a gradient update is performed according to the sixth gradient values ​​corresponding to each micro-batch data to obtain a first gradient value.

[0036] On the one hand, an embodiment of the present application provides a device for training a large model over a wide area network, which is applied to each target graphics processor in a distributed model training system, wherein the distributed model training system includes multiple wide area network nodes; each wide area network node includes multiple graphics processors; all model parameters of the target model to be trained are divided into multiple parameter slices; each graphics processor stores all model parameters of the target model, gradient slices corresponding to some parameter slices, and optimizer slices corresponding to one parameter slice; each gradient slice is a set of gradients corresponding to each model parameter in the corresponding parameter slice; each optimizer slice is a set of optimizer states corresponding to each model parameter in the corresponding parameter slice; the gradient slices corresponding to each parameter slice are respectively allocated to each graphics processor in the same wide area network node; the optimizer slices corresponding to different parameter slices are respectively allocated to different graphics processors; the device includes:

[0037] A processing unit, used for performing forward and backward propagation processing on the batch data to be processed to obtain a first gradient value of a corresponding target gradient slice;

[0038] A first receiving unit is used to receive the second gradient value of the target gradient slice corresponding to each other sent by at least one first graphics processor in other wide area network nodes; the first graphics processor is a graphics processor that is assigned the same gradient slice as the target graphics processor;

[0039] Update parameters, used to update parameters according to the first gradient value and at least one second gradient value corresponding to the target gradient slice, and obtain the parameter value of the first parameter slice corresponding to the corresponding optimizer slice;

[0040] A second receiving unit, configured to receive parameter values ​​of second parameter slices respectively corresponding to respective optimizer slices sent by at least one first graphics processor;

[0041] The third receiving unit is used to receive parameter values ​​of third parameter slices respectively corresponding to respective optimizer slices sent by at least one second graphics processor; the second graphics processor is a graphics processor located in the same wide area network node as the target graphics processor.

[0042] In one implementation, the processing unit is further configured to:

[0043] Determining a first total number of model parameters and a second total number of graphics processors;

[0044] Obtaining a partition size according to the first total quantity and the second total quantity;

[0045] According to the partition size, each model parameter is divided to obtain multiple parameter slices; the number of model parameters contained in each parameter slice is the partition size.

[0046] In one implementation, the processing unit is further configured to:

[0047] If the first total number cannot be divided by the second total number, then fill in the parameters of each model;

[0048] A first total number of model parameters after filling is determined.

[0049] In one implementation, the processing unit is further configured to:

[0050] Generate optimizer slices corresponding to each parameter slice;

[0051] Assign different optimizer slices to different GPUs;

[0052] Generate gradient slices corresponding to each parameter slice;

[0053] For each wide area network node, the following steps are performed respectively: each gradient slice is allocated to each graphics processor of the wide area network node; the parameter slice corresponding to the gradient slice allocated to each graphics processor includes the parameter slice corresponding to the optimizer slice allocated to the graphics processor.

[0054] In one implementation, the first receiving unit is further configured to:

[0055] According to the assigned gradient slices, the graphics processors are communicated and combined to obtain multiple inter-domain process groups to aggregate the gradient values ​​of the same gradient slice; the graphics processors in the same inter-domain process group are assigned the same gradient slice and are located in different wide area network nodes;

[0056] Establishing communication between each graphics processor in each inter-domain process group;

[0057] The graphics processors in each WAN node are respectively communicated and combined to obtain the intra-domain process groups corresponding to each WAN node, so as to summarize the parameter values ​​of each parameter slice in the same WAN node;

[0058] The communication between each graphics processor in the process group in each domain is established respectively.

[0059] In one implementation, the updating unit is used to:

[0060] Performing gradient updating according to the first gradient value and the at least one second gradient value respectively corresponding to the at least one target gradient slice, to obtain a third gradient value for each target gradient slice respectively;

[0061] Based on the third gradient value of the target gradient slice corresponding to the first parameter slice, obtaining a first optimizer state value of the corresponding optimizer slice;

[0062] According to the third gradient value and the first optimizer state value, the parameters are updated to obtain the parameter value of the first parameter slice.

[0063] In one implementation, the processing unit is used to:

[0064] For each micro-batch of the batch data, the following steps are performed: forward and backward propagation processing is performed on the micro-batch data to obtain a fourth gradient value of a target gradient slice; a fifth gradient value of the target gradient slice is received from at least one first graphics processor; a sixth gradient value is obtained according to the fourth gradient value and at least one fifth gradient value;

[0065] When it is determined that the batch data processing is completed, a gradient update is performed according to the sixth gradient values ​​corresponding to each micro-batch data to obtain a first gradient value.

[0066] On the one hand, an embodiment of the present application provides an electronic device, including:

[0067] Processor; and

[0068] A memory storing computer instructions for causing a processor to execute steps of a method provided in any of the above-mentioned various optional implementations of wide area network large model training.

[0069] On the one hand, an embodiment of the present application provides a storage medium storing computer instructions, which are used to enable a computer to execute the steps of the method provided in any of the various optional implementations of wide area network large model training as described above.

[0070] The method for training a large model in a wide area network in an embodiment of the present application includes executing the following steps for each target graphics processor in each graphics processor: performing forward and backward propagation processing on the batch data to be processed to obtain the first gradient value of the corresponding target gradient slice; receiving the second gradient value of the target gradient slice corresponding to each other sent by at least one first graphics processor in other wide area network nodes; the first graphics processor is a graphics processor that is assigned the same gradient slice as the target graphics processor; according to the first gradient value and at least one second gradient value corresponding to the target gradient slice, update the parameters to obtain the parameter value of the first parameter slice corresponding to the corresponding optimizer slice; receiving the parameter value of the second parameter slice corresponding to each optimizer slice sent by at least one first graphics processor; receiving the parameter value of the third parameter slice corresponding to each optimizer slice sent by at least one second graphics processor; the second graphics processor is a graphics processor in the same wide area network node as the target graphics processor. In this way, the gradient and optimizer state are divided and stored, and the data is merged, which can take into account the reduction of the memory usage of each GPU and the improvement of the model training performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0072] Figure 1 This is an example diagram of the architecture of a distributed model training system in an embodiment of the present application.

[0073] Figure 2 It is a flowchart of a method for allocating video memory in an embodiment of the present application.

[0074] Figure 3 This is an example diagram of gradient slicing allocation in an embodiment of the present application.

[0075] Figure 4 This is an example diagram of a communication grouping in an embodiment of the present application.

[0076] Figure 5 It is a flow chart of a method for training a large model in a wide area network in an embodiment of the present application.

[0077] Figure 6 It is a schematic diagram of forward propagation in an embodiment of the present application.

[0078] Figure 7 It is a schematic diagram of a gradient storage in an embodiment of the present application.

[0079] Figure 8 It is a schematic diagram of a gradient domain merging in an embodiment of the present application.

[0080] Fig. 9 It is a schematic diagram of an inter-domain gradient transfer in an embodiment of the present application.

[0081] Fig.10 It is a schematic diagram of an inter-domain gradient merging in an embodiment of the present application.

[0082] Fig.11 This is an example diagram of parameter update in an embodiment of the present application.

[0083] Fig.12 It is a schematic diagram of parameter merging of an inter-domain process group in an embodiment of the present application.

[0084] Fig.13 This is a schematic diagram of parameter merging of a process group within a domain in an embodiment of the present application.

[0085] Fig.14 It is a detailed flow chart of a model training method in an embodiment of the present application.

[0086] Fig.15 It is a structural block diagram of a device for training a large model in a wide area network in an embodiment of the present application.

[0087] Fig.16 It is a structural schematic diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0088] The technical solution of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described implementation methods are part of the implementation methods of the present application, rather than all of the implementation methods. Based on the implementation methods in the present application, all other implementation methods obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In addition, the technical features involved in the different implementation methods of the present application described below can be combined with each other as long as they do not conflict with each other.

[0089] First, some terms involved in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0090] Terminal device: can be a mobile terminal, a fixed terminal or a portable terminal, such as a mobile phone, a station, a unit, a device, a multimedia computer, a multimedia tablet, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a personal communication system device, a personal navigation device, a personal digital assistant, an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a gaming device or any combination thereof, including accessories and peripherals of these devices or any combination thereof. It is also foreseeable that the terminal device can support any type of interface for the user (such as a wearable device), etc.

[0091] Server: It can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0092] In the model training scenario, since model training usually requires a huge amount of data support, in order to improve training efficiency, the amount of batch data processed simultaneously during training usually needs to be increased, which will significantly increase the burden on video memory. Furthermore, model training requires frequent calculation of gradients and optimizer states during the process, which will multiply the video memory required for training. When distributed model training is performed based on a wide area network to reduce existing memory usage, since nodes usually communicate frequently, there is a large communication delay and low training efficiency.

[0093] Therefore, during the model training process, how to balance reducing the memory usage of each GPU and improving the model training performance is a problem that needs to be solved.

[0094] Based on the defects of the above-mentioned related technologies, the embodiments of the present application provide a method, device, electronic device and storage medium for training a large model in a wide area network, aiming to reduce the memory occupancy of each GPU and improve the model training performance during model training.

[0095] A method for training a large model over a wide area network is provided in an embodiment of the present application. The method can be applied to a distributed model training system, which includes multiple wide area network nodes; each wide area network node includes multiple GPUs; it is a group of multiple GPUs. Different GPUs can be located in the same electronic device or in different electronic devices. The electronic device can be any type of device, such as a terminal device or a server. The number of wide area network nodes can be expressed as N, and the number of GPUs contained in each wide area network node can be expressed as M, where N and M are both positive integers greater than 1.

[0096] Among them, all model parameters of the target model to be trained are divided into multiple parameter shards; each GPU stores all model parameters of the target model, gradient shards corresponding to some parameter shards, and optimizer shards corresponding to one parameter shard; each gradient shard is a collection of gradients corresponding to each model parameter in the corresponding parameter shard; each optimizer shard is a collection of optimizer states corresponding to each model parameter in the corresponding parameter shard; the gradient shards corresponding to each parameter shard are respectively allocated to each GPU in the same wide area network node; and the optimizer shards corresponding to different parameter shards are respectively allocated to different GPUs.

[0097] See also Figure 1 As shown, it is an example diagram of the architecture of a distributed model training system in an embodiment of the present application, and the distributed model training system includes a wide area network node 1 and a wide area network node 2. The wide area network node 1 and the wide area network node 2 each contain four GPUs. All model parameters of the target model are divided into 8 parameter slices, namely parameters [0]-parameters [7], and corresponding gradient slices, namely gradients [0]-gradients [7] and optimizer slices, namely optimizer states [0]-optimizer states [7], are generated. Each GPU is allocated with complete model parameters, namely parameters [0]-parameters [7], and is allocated with two gradient slices and one optimizer slice.

[0098] In an embodiment of the present application, a wide area network distributed training framework is provided, which includes two parts: initialization and model training process, wherein initialization is used to start the model training task, mainly including process creation, video memory allocation, communication grouping, model initialization and data loading. The model training process includes forward and backward propagation, gradient calculation and merging, optimizer state update and model parameter update and merging in batch training (which may include micro-batch training). During the model training process, initialization is mainly the initialization of the model training environment, which can be executed only once, and the training process is a periodic process for training model parameters, which can be executed cyclically.

[0099] Among them, video memory allocation refers to the segmented storage for model parameters, gradients, and optimizer states. Model parameters refer to the parameters that the model needs to learn during training. Gradients are the derivative data of model parameters on sample data that are automatically calculated during the backpropagation process for model update. Optimizer states are data about current gradients and historical gradients that are calculated and retained to optimize gradient descent. In each round of training cycle, all GPU processes are required to complete gradient updates, optimizer state updates, and model parameter updates in succession, and to ensure that the global gradients, optimizer states, and models are the same after the update.

[0100] Combine the following Figure 1 and Figure 2, the method of video memory allocation is explained. Figure 2 The flowchart of a method for allocating video memory is as follows:

[0101] Step 200: All model parameters of the target model are stored in each GPU respectively.

[0102] The target model is a model that includes forward and backward propagation processing during the training process, and can be a model built based on a neural network. The forward and backward propagation processing includes forward propagation and backward propagation (ie, back propagation).

[0103] In the embodiment of the present application, based on the idea of ​​data parallelism, complete model parameters are stored in each GPU process to ensure that the process of each GPU can independently complete forward propagation and backward propagation.

[0104] Step 201: Divide each model parameter to obtain multiple parameter slices.

[0105] In one implementation, when executing step 201, the following steps may be used:

[0106] S2011: Determine a first total number of model parameters and a second total number of graphics processors.

[0107] S2012: Obtain a partition size according to the first total number and the second total number.

[0108] In one implementation, the partition size is obtained according to the ratio of the first total number to the second total number.

[0109] Furthermore, if the first total number cannot be divided by the second total number, each model parameter is filled to ensure that the first total number can be divided by the second total number; the first total number of each model parameter after filling is determined, and the partition size is obtained based on the ratio of the new first total number to the second total number.

[0110] Among them, the following formula can be used to determine the partition size:

[0111] partition_size=(num_parameter+num_padding) / NM;

[0112] Among them, partition_size is the partition size, num_parameter is the first total number before parameter filling, num_padding is the filling parameter amount, N is the number of WAN nodes, M is the number of graphics processors included in each WAN node; NM is the second total number.

[0113] In this way, the model parameters can be padded to ensure that the first total number of the padded model parameters is divisible by the second total number of GPUs participating in the training, and then the partition size can be obtained.

[0114] S2013: Divide each model parameter according to the partition size to obtain multiple parameter slices.

[0115] In this way, the model parameters can be divided into multiple parameter slices of the same size. It should be noted that although the actual storage space of the parameters is not divided, the parameters are still sliced ​​to determine the index of the subsequent gradient slice and optimizer slice.

[0116] Step 202: Perform storage partitioning on the optimizer state.

[0117] In one implementation, optimizer slices corresponding to each parameter slice are generated; and different optimizer slices are respectively allocated to different graphics processors.

[0118] In one implementation, a corresponding process is created for each GPU participating in the training, and the GPU and the GPU process are numbered, and each parameter slice is numbered respectively, and the number of each parameter slice is used as the number of the corresponding optimizer state, and then each optimizer slice is assigned to the GPU process with the same number according to the number of each optimizer slice, and recorded. The numbers of the GPU and its corresponding GPU process are the same.

[0119] For example, in a training task with N WAN nodes and each node containing M GPUs, the GPUs can be numbered from 0 to NM-1, and the GPU processes can be numbered from 0 to NM-1. Each parameter shard and its corresponding optimizer shard can be numbered from 0 to NM-1. Then the optimizer shard numbered 0 can be assigned to the GPU process numbered 0.

[0120] This is because during the model training process, the memory usage of the optimizer state is significantly higher than that of the model parameters and gradients. Therefore, based on the parameter sharding, each optimizer shard is obtained and allocated to evenly distribute the optimizer state to all GPU processes in each WAN node, thereby greatly reducing the memory resources required for each GPU training.

[0121] Step 203: storing and segmenting the gradient.

[0122] In one implementation, a gradient slice corresponding to each parameter slice is generated; for each wide area network node, the following steps are performed: each gradient slice is assigned to each graphics processor of the wide area network node; the parameter slice corresponding to the gradient slice assigned to each graphics processor includes the parameter slice corresponding to the optimizer slice assigned to the graphics processor.

[0123] In one implementation, taking a wide area network node as an example, the numbers of each parameter slice are used as the numbers of the corresponding gradient slices, and then the gradient slices can be assigned to the GPU processes in the wide area network node in turn according to the order of the numbers of the gradient slices, and recorded. Among them, each number of the gradient slice in the same GPU process includes the number of the corresponding optimizer slice. Similarly, the storage segmentation of the gradient can be performed on other wide area network nodes, which will not be repeated here.

[0124] Combine the following Figure 3 The allocation of gradient shards is described. Figure 3 An example diagram of a gradient sharding assignment. Figure 3 In the example, M is 2, that is, a WAN node contains two GPUs, namely GPU[0] and GPU[1], and there are four gradient shards, namely gradient[0], gradient[1], gradient[2], and gradient[3]. Then, in order of numbering, gradient[0] is assigned to GPU[0], gradient[1] is assigned to GPU[1], gradient[2] is assigned to GPU[0], and gradient[3] is assigned to GPU[1].

[0125] In the memory allocation scheme during initialization, according to the characteristics of GPU, intra-domain nodes and inter-domain nodes in WAN on micro-batch data and batch number training, combined with the proportion of model parameters, gradients and optimizer state memory usage, a different storage partitioning scheme is adopted. When allocating the storage space of model parameters, the model parameters are not partitioned and are completely allocated to each GPU, that is, redundant storage is performed at the GPU level; when allocating the storage space of gradients, the gradient storage is not partitioned for all nodes in the WAN domain, but is divided equally according to the total number of GPUs contained in the WAN node based on the WAN node; when allocating the storage space of optimizer state, it is evenly divided to all GPUs participating in the training for storage, without distinguishing between intra-domain and inter-domain (i.e., intra-node and inter-node). Here, intra-domain refers to a high-speed interconnected LAN or private network with low latency and high bandwidth.

[0126] During the training process, the frequency of updating and merging gradient data is usually much higher than the frequency of data processing of model parameters and optimizer status, especially when model training is based on micro-batches, the frequency difference will be more obvious. Therefore, the complete gradient (i.e., the sum of all gradient slices) is divided into gradient slices for storage in units of one WAN node, so that it can be divided and stored in each GPU in each WAN node. In this way, the number of inter-domain communications (i.e., the number of communications between GPUs in different WAN nodes) can be reduced, and the complete gradient can be evenly distributed to each GPU process in the same WAN node, which can reduce the video memory resources required for single GPU training.

[0127] In order to reduce the number of inter-domain node communications and the memory occupancy, the optimizer state and gradient are stored and segmented according to the different degrees of influence of the mutual communication delay between GPUs within a node, between nodes within a domain, and between GPUs in a wide area network on the model training cycle, as well as the memory occupancy ratio of model parameters, gradients, and optimizer states. In this way, each GPU is allocated with complete model parameters, and is allocated with optimizer shards and gradient shards with the same numbers. If there are multiple gradient shards, other numbered gradient shards are also included. In this way, the GPU processes can be grouped according to the numbers of each GPU, GPU process, optimizer shard, and gradient shard for subsequent communication between GPUs.

[0128] In one implementation, for each WAN node, all GPU processes in the same WAN node are organized into corresponding process groups to obtain intra-domain process groups. Multiple GPU processes in each WAN node that are allocated with the same gradient slice (i.e., gradient slices with the same number) are organized into corresponding process groups, which are called inter-domain process groups.

[0129] Specifically, according to the assigned gradient slices, the graphics processors are communicated and combined to obtain multiple inter-domain process groups to summarize the gradient values ​​of the same gradient slice; the graphics processors in the same inter-domain process group are assigned the same gradient slice and are located in different wide area network nodes; the communication between the graphics processors in each inter-domain process group is established respectively; the graphics processors in each wide area network node are communicated and combined respectively to obtain the intra-domain process groups corresponding to each wide area network node to summarize the parameter values ​​of each parameter slice in the same wide area network node; the communication between the graphics processors in each intra-domain process group is established respectively.

[0130] Combine the following Figure 4 Communication grouping will be described. Figure 4 The figure is an example of a communication grouping. Figure 4 In the example, there are two WAN nodes, namely, WAN node 1 and WAN node 2. For example, the two GPUs in WAN node 1 (namely, GPU[0] and GPU[1]) are formed into intra-domain process group 1, and the two GPUs in WAN node 2 (namely, GPU[2] and GPU[3]) are formed into intra-domain process group 2. Since GPU[0] in WAN node 1 and GPU[2] in WAN node 2 are allocated the same gradient slices, namely, gradient[0] and gradient[2], GPU[0] and GPU[2] are formed into inter-domain process 1, and similarly, GPU[1] and GPU[3] are formed into inter-domain process 2.

[0131] Furthermore, after completing the storage space segmentation and communication grouping, the model can be initialized. For example, the initial parameter weights of the target model are read, the number of layers and parameter amounts of each model parameter are counted, and the size of the gradient slice and the optimizer slice can be calculated according to the video memory allocation scheme, and the complete model parameters are respectively allocated to each GPU process, and the space of the corresponding gradient slice and optimizer slice is allocated in the GPU video memory.

[0132] In this way, during the model initialization process, the model initialization completes the reading of the original model weights, creates a separate working process for each GPU, loads the model parameters into the GPU video memory according to the storage partitioning scheme, and allocates storage space for the gradient and optimizer state after partitioning on the GPU; the GPU processes are grouped for communication according to the storage partitioning scheme, all GPUs in each domain are organized into an intra-domain process group, and GPUs containing the same gradient slice in different domains are organized into an inter-domain process group (for merging gradients of the same slice between domains).

[0133] Furthermore, data loading can be used to complete the reading and division of training data. Data loading is to initialize the data set used in the training process, read the global data used for training, and divide the data into training sets and test sets. The data is evenly divided into batches according to the total number of GPUs and distributed to each GPU process.

[0134] Specifically, all training data are read, and the training data are divided into training sets and test sets. According to the training task, a hyperparameter, i.e., batch size, is pre-set, and the training data is divided according to the batch size and the number of GPU processes.

[0135] For example, in a training task with N WAN nodes and each node containing M GPUs, assuming that the training data size is K and the batch data size is B, the size of the data set allocated to each GPU process is: Among them, mathematical symbols It means taking the upper bound of the value, that is, rounding up.

[0136] After initialization, model training can be performed, which is the process of training model parameters. The model training process takes the training of a batch of data as a training cycle. In each training cycle, data of the batch size is extracted from the training data allocated by the GPU process for training.

[0137] Among them, each training cycle mainly includes the following processes: forward propagation, backward propagation, inter-domain gradient merging, optimizer state update, model parameter update and merging. Furthermore, if the amount of training data is large, in order to further reduce the video memory occupied by model training, the batch data can be divided to obtain multiple micro-batches of data. In this case, the training cycle can include the following processes: obtaining batch data and splitting, inputting micro-batch data, forward propagation, backward propagation and intra-domain gradient merging, inter-domain gradient merging, optimizer state update, model parameter update and merging.

[0138] See also Figure 5 As shown, it is a flow chart of a method for training a large model in a wide area network in an embodiment of the present application. Figure 5 To illustrate the method, for each target GPU in each graphics processor, perform the following steps:

[0139] Step 500: Perform forward and backward propagation processing on the batch data to be processed to obtain the first gradient value of the corresponding target gradient slice.

[0140] In one implementation, the target GPU acquires batch data, and based on the target model, performs forward propagation and backward propagation on the batch data to obtain first gradient values ​​corresponding to at least one target gradient slice assigned to the target GPU.

[0141] Each GPU is respectively allocated with one or more corresponding gradient slices, namely, corresponding target gradient slices. If there are multiple target gradient slices, the first gradient value corresponding to each target gradient slice can be obtained respectively.

[0142] In the embodiment of the present application, each GPU in the intra-domain process group stores complete model parameters, and the intra-domain process group stores complete gradients, so forward propagation (forward) and backward propagation (backward) of the target model can be performed autonomously.

[0143] Furthermore, in order to further reduce the video memory required for training, in the forward and backward propagation process, the batch data (batch) assigned to the process group in the domain is further subdivided into micro-batch data (mirco batch), and the gradient accumulation technology is used to determine the gradient value. Specifically, in the training of micro-batch data, the training of a micro-batch data is a step, and each GPU process forward propagates the micro-batch data, calculates the forward loss of the target model in the micro-batch data, and then performs backward propagation. In the backward propagation process, the gradient value of each gradient slice of the target model is calculated layer by layer from the output end to the output end. To ensure that the GPU process only saves the gradient data of the corresponding gradient partition size, after the gradient value of the gradient partition of each layer is calculated, the gradient value is merged in real time.

[0144] Combine the following Figure 6 , the forward propagation of micro-batch data is explained. Figure 6 A schematic diagram of forward propagation. Figure 6 In the process, the GPU process performs forward propagation on the micro-batch data and calculates the loss of the target model on the forward propagation of the micro-batch data.

[0145] During the back propagation process, after the gradient value of a certain layer of gradient slices is obtained, it is moved into a buffer of a certain size. When the buffer is full, the gradient data in the buffer can be sent to the in-domain GPU storing the gradient slice using point-to-point communication, where the in-domain GPU refers to the GPU in the same WAN node as the target GPU, which can be called the second GPU. The second GPU determines whether the gradient partition is stored locally. If so, the gradient value of the gradient slice is updated to the reserved gradient space, otherwise, the received data is discarded. As an example, this operation can be implemented by a hook function that is mounted on each layer of the model parameters to process the gradient merging in the domain.

[0146] That is to say, each GPU in each WAN node will generate gradient values ​​for all gradient slices, so each GPU will send the gradient values ​​of all generated gradient slices to each other GPU in the same WAN node, and receive the gradient values ​​of all gradient slices sent by each other GPU in the same WAN node. Each GPU will filter out the gradient value of the corresponding target gradient slice from the gradient values ​​of all received gradient slices, and perform intra-domain gradient merging based on the gradient value of the filtered target gradient slice and the gradient value of the locally generated target gradient slice, and discard the gradient values ​​of other gradient slices.

[0147] In one implementation, when executing step 500, the following steps may be used:

[0148] S5001: For each micro-batch in the batch data, perform the following steps until it is determined that the batch data processing is completed: perform forward and backward propagation processing on the micro-batch data to obtain a fourth gradient value of a target gradient slice; receive a fifth gradient value of the target gradient slice respectively sent by at least one first graphics processor; and obtain a sixth gradient value according to the fourth gradient value and at least one fifth gradient value.

[0149] The second graphics processor is a graphics processor located in the same wide area network node as the target graphics processor.

[0150] In one implementation, the average of the fourth gradient value and at least one fifth gradient value is calculated to obtain the sixth gradient value, that is, the gradient values ​​are reduced.

[0151] Combine the following Figure 7 and Figure 8 The generation, storage and in-domain merging of gradient values ​​are explained. Figure 7 A schematic diagram of gradient storage. Figure 7 In the example, GPU[0] stores four parameter slices, two gradient slices, and one optimizer slice. Each parameter slice includes parameter[0], parameter[1], parameter[2], and parameter[3]. Each gradient slice is gradient[0] and gradient[1]. The optimizer slice is optimizer state[0]. During the back-propagation process, GPU[0] generates the gradient value of the gradient slice corresponding to parameter[3], which is represented as new gradient[3]. (0) , and move it into the cache.

[0152] Figure 8 A schematic diagram of gradient domain merging. Figure 8 In the example, WAN node 1 contains two GPUs, namely GPU[0] and GPU[1]. During the back-propagation process, GPU[1] generates the gradient value of the gradient slice corresponding to parameter[3], which is represented as the new gradient[3]. (1) , and receives the new gradient [3] sent by GPU [0] (0) , and calculate the new gradient [3] (0) With the new gradient [3] (1) The average value of is obtained to obtain the new gradient [3], which is denoted as the new gradient [3]. That is, GPU [1] is taken as the target GPU, GPU [0] is the second GPU in the same WAN node as the target GPU, and the target gradient slice corresponding to GPU [1] is gradient [3]. The micro-batch data is processed forward and backward to obtain the fourth gradient value of gradient [3], which is the new gradient [3]. (1) , and receives the fifth gradient value of the gradient [3] of the second GPU, namely GPU [0], namely the new gradient [3] (0) , and calculate the new gradient [3] (0) With the new gradient [3] (1) The sixth gradient value is obtained by taking the average value of , i.e., the new gradient [3].

[0153] In this way, after back propagation and merging the in-domain gradient, take the next micro-batch of data for training, and repeat this process until the gradient accumulation boundary is reached, that is, the batch data processing is completed.

[0154] S5002: When it is determined that the batch data processing is completed, a gradient update is performed according to the sixth gradient values ​​corresponding to each micro-batch data to obtain a first gradient value.

[0155] In one implementation, a sum of at least one sixth gradient value is calculated to obtain the first gradient value.

[0156] In the embodiment of the present application, only the intra-domain gradient merging for the target GPU is described. Similarly, the intra-domain gradient merging can be performed on other GPUs respectively.

[0157] In this way, the batch data is divided to obtain a plurality of micro-batch data, and each micro-batch data in the batch data is sequentially processed by forward and backward propagation, so that the first gradient value of the target gradient slice can be obtained.

[0158] Step 501: receiving second gradient values ​​of respective corresponding target gradient slices sent by at least one first graphics processor in other wide area network nodes; the first graphics processor is allocated with the target gradient slice.

[0159] There is at least one first GPU, the first GPU and the target GPU are located in different WAN nodes, and the first GPU and the target GPU are allocated the same gradient slice, namely, the target gradient slice.

[0160] Furthermore, the target GPU also sends the first gradient value of the target gradient slice to each first GPU respectively.

[0161] That is, each GPU sends the first gradient value of the corresponding gradient slice to each other GPU in the inter-domain process group to which it belongs, and receives the second gradient value of the corresponding gradient slice sent by each other GPU in the inter-domain process group.

[0162] In this way, each GPU obtains the global gradient of the corresponding gradient slice trained in the batch of data.

[0163] Step 502: Update parameters according to the first gradient value and at least one second gradient value corresponding to the target gradient slice, and obtain the parameter value of the first parameter slice corresponding to the corresponding optimizer slice.

[0164] In one implementation, when executing step 502, the following steps may be used:

[0165] S5021: Perform a gradient update according to the first gradient value and the at least one second gradient value respectively corresponding to the at least one target gradient slice, and obtain a third gradient value of each target gradient slice respectively.

[0166] In one implementation, the following steps are performed for each target gradient slice:

[0167] The first gradient value and the second gradient value of the target gradient slice are averaged to obtain a third gradient value, wherein the second gradient value is at least one.

[0168] For example, if a first gradient value corresponding to a target gradient slice is T1, and the target gradient slice corresponds to only one second gradient value, and the second gradient value is T2, then the third gradient value of the target gradient slice is determined to be (T1+T2) / 2.

[0169] This is because after reaching the gradient accumulation boundary, that is, after the training of the batch of data is completed, although each intra-domain process group stores a complete gradient, the training data of different intra-domain process groups are only a part of the global batch data. Therefore, the gradients between different intra-domain process groups need to be merged to obtain the global gradient obtained by the training of the batch of data. According to the gradient memory allocation scheme, after the training of each batch of data is completed, it is only necessary to merge the gradient data (i.e. the gradient value of the corresponding gradient slice) held by each GPU process in each inter-domain process group to ensure that all GPU processes obtain the global gradient after the training of the batch of data.

[0170] Combine the following Fig. 9 and Fig.10 Inter-domain gradient transfer and inter-domain gradient merging are explained. Fig. 9 A schematic diagram of inter-domain gradient transfer. Fig.10 A schematic diagram of inter-domain gradient merging.

[0171] Fig. 9 The two GPUs include GPU[0] and GPU[2] located at different WAN nodes. Both of them store the same two gradient slices. GPU[0] sends the gradient values ​​corresponding to the two locally generated gradient slices to GPU[2], and GPU[2] sends the gradient values ​​corresponding to the two locally generated gradient slices to GPU[0]. Fig.10 In , GPU[0] and GPU[2] both merge gradients based on the local and received gradient values ​​corresponding to the two gradient slices. Similarly, GPU[1] and GPU[3] both merge gradients based on the local and received gradient values ​​corresponding to gradient[1] and gradient[3].

[0172] In the embodiment of the present application, in the process of inter-domain gradient merging, gradient merging is performed in units of inter-domain process groups. Each GPU process sends the gradient value of the gradient slice it holds to the GPU process with the same gradient slice, and averages the gradient value of the received gradient slice with the gradient value of its own gradient slice to obtain an updated gradient value.

[0173] S5022: Based on the third gradient value of the target gradient slice corresponding to the first parameter slice, obtain a first optimizer state value of the corresponding optimizer slice.

[0174] Among them, the update of the optimizer state is only related to the gradient. Considering the large amount of data communicated by the optimizer state, when performing parameter segmentation, the parameter slices of the gradient slices in the same GPU are overwritten by the parameter slices of the optimizer slices, so that the gradient merge can cover the optimizer state merge, and there is no need to merge the optimizer state independently again. After forward propagation, backpropagation, intra-domain gradient merge, and inter-domain gradient merge, all intra-domain process groups save the updated global gradients and can autonomously update the partial optimizer state stored in the domain.

[0175] S5023: Update parameters according to the third gradient value and the first optimizer state value to obtain parameter values ​​of the first parameter slice.

[0176] In one implementation, a parameter slice to be updated, i.e., a first parameter slice, is determined according to a corresponding optimizer slice, and according to a model parameter update calculation formula, based on a third gradient value and a first optimizer state value corresponding to the first parameter slice, and a learning rate, the first parameter slice is updated to obtain a parameter value of the first parameter slice. In the embodiment of the present application, only the model parameter update of one GPU is used for explanation. Similarly, the model parameters of other GPUs can be updated, which will not be described in detail here.

[0177] Combine the following Fig.11 Explain parameter updating. Fig.11 An example diagram of a parameter update. Fig.11 In the process, GPU[0] obtains the updated gradient values ​​of each gradient slice, i.e., new gradient[0] and new gradient[2], and the updated first optimizer state value of the optimizer slice, i.e., new optimizer state[0], and performs parameter update according to the new gradient[0], the new gradient[2], and the new optimizer state[0] to obtain the updated parameter value of the first parameter slice, i.e., new parameter[0].

[0178] In the embodiment of the present application, the merging of all gradients obtained based on batch data training is completed before the model parameters are updated. Therefore, the update of the model parameters will not cause a decrease in accuracy due to the division of the gradient and the optimizer state.

[0179] Step 503: Receive parameter values ​​of second parameter slices corresponding to respective optimizer slices sent by at least one first graphics processor.

[0180] Specifically, based on a principle similar to the target GPU obtaining the parameter value of the first parameter slice corresponding to the corresponding optimizer slice, the parameter value of the parameter slice (i.e., the second parameter slice) corresponding to the corresponding optimizer slice of each other GPU in the inter-domain process group to which the target GPU belongs is obtained, and the parameter value of the second parameter slice corresponding to the corresponding optimizer slice sent by each other GPU in the inter-domain process group, i.e., the first graphics processor, is received.

[0181] This is because according to the distribution rules of gradient and optimizer status, although each GPU process in the inter-domain process group stores different optimizer slices, the gradient slices they hold are the same, and the parameter slices corresponding to the gradient slices must include the parameter slices corresponding to the optimizer slices held. Therefore, each GPU in each inter-domain process group sends the parameter value of the updated parameter slice it holds to each other GPU holding the same gradient slice, and replaces the parameter value of the original corresponding parameter slice with the parameter value of the received parameter slice.

[0182] Combine the following Fig.12 Describes the merging of parameters of inter-zone process groups. Fig.12 A schematic diagram of parameter merging of an inter-domain process group. Fig.12 In the example, WAN node 1 includes GPU[0] and GPU[1], and WAN node 2 includes GPU[2] and GPU[3]. GPU[0] sends new parameter[0] to GPU[2], and receives new parameter[2] from GPU[2]. GPU[1] sends new parameter[1] to GPU[3], and receives new parameter[3] from GPU[3].

[0183] Step 504: receiving parameter values ​​of third parameter slices respectively corresponding to respective optimizer slices sent by at least one second graphics processor; the second graphics processor is a graphics processor located in the same wide area network node as the target graphics processor.

[0184] The third parameter slice corresponding to the second GPU is its updated parameter slice.

[0185] Combine the following Fig.13 Describes the merging of parameters for process groups within a domain. Fig.13 A schematic diagram of parameter merging for a process group within a domain. Fig.13In the example, WAN node 1 includes GPU[0] and GPU[1], and WAN node 2 includes GPU[2] and GPU[3]. GPU[0] sends new parameters[0] and[2] to GPU[1], and receives new parameters[1] and[3] from GPU[1]. GPU[2] sends new parameters[0] and[2] to GPU[3], and receives new parameters[1] and[3] from GPU[3].

[0186] Since any process group in the domain will contain all model parameters, the parameters are merged again in the process group in the domain. Each GPU process sends the parameter values ​​of the parameter slices it holds to all GPU processes in the domain, so that each GPU process can obtain the updated complete global parameters.

[0187] In this way, after the parameter update is completed, each GPU process holds a parameter slice of the updated model parameters. Therefore, combined with the intra-domain parameter merging and inter-domain parameter merging, each GPU process can obtain the complete updated model parameters, reducing the number of inter-domain communications during the parameter merging process and further optimizing the model training cycle.

[0188] Combine the following Fig.14 , the model training process is explained. Fig.14 This is a detailed flow chart of a model training method. The specific implementation process of this method includes:

[0189] Step 1400: The training cycle begins.

[0190] Step 1410: Split the acquired batch data into multiple micro-batch data.

[0191] Step 1420: Input a micro-batch of data.

[0192] Step 1430: Perform forward propagation based on the micro-batch data.

[0193] Step 1440: Backward propagation and intra-domain gradient merging.

[0194] Among them, the back propagation is layer-by-layer back propagation.

[0195] Step 1450: Determine whether the batch data has been processed. If so, execute step 1460; otherwise, execute step 1420.

[0196] Step 1460: Merge inter-domain gradients.

[0197] Step 1470: Optimizer state update.

[0198] Step 1480: Model parameter update and merging.

[0199] Step 1490: The training cycle ends and step 1400 is executed.

[0200] Specifically, when executing step 1400 to step 1490, the specific steps refer to the above-mentioned step 500 to step 504 to perform multiple rounds of training on the model, which will not be repeated here.

[0201] In the embodiment of the present application, in order to solve the problem of large communication delay and limited bandwidth between wide area network nodes, the gradient storage space is divided and merged by distinguishing between intra-domain and inter-domain, and all GPUs (without distinguishing between intra-domain and inter-domain) are used to divide, store and merge the optimizer state, and update the parameters. Specifically, the training of a batch of data is a training cycle. In each training cycle, the batch data is divided into micro-batch data of a specified size, and micro-batch data training is performed. Micro-batch data training adopts gradient accumulation technology, with the forward and backward propagation of a micro-batch data as a step. Gradient calculation is performed in the back propagation of each step, and the gradient is calculated layer by layer and merged layer by layer in the domain. The intra-domain merging of the gradient averages the gradient based on point-to-point communication. After each step of back propagation is completed, the obtained gradient is accumulated. When the gradient accumulation boundary is reached, that is, after all micro-batch data training is completed, the gradient inter-domain merging is performed, and the same gradient slices in the inter-domain process group are averaged once. After the gradient merging is completed, the optimizer state and parameters are updated. The optimizer state stored in each GPU process and the model parameters of its corresponding part are updated according to the calculation formula. Parameter merging adopts the method of first merging between domains and then merging within the domain, which further reduces the communication overhead. The model parameters of the inter-domain process groups are merged first, and then the parameters of each intra-domain process group are merged again. After the above two types of merging are completed, the overall gradient, optimizer state and model parameter merging of all computing nodes and training samples are completed. In this way, while reducing the GPU video memory usage in the wide area network large model training, the communication delay and bandwidth between the wide area network nodes can be reduced. The impact on the training cycle is reduced, and a balance is achieved between parallelism, video memory usage and communication between remote nodes, so that the performance impact of wide area delay and bandwidth on cluster training is controlled at a low level while ensuring a high degree of parallelism, and efficient wide area network large model training is achieved.

[0202] Based on the same inventive concept, a device for training a large model of a wide area network is also provided in the embodiment of the present application. Since the principle of solving the problem by the above-mentioned device and equipment is similar to that of a method for training a large model of a wide area network, the implementation of the above-mentioned device can refer to the implementation of the method, and the repeated parts will not be repeated. The device can be applied to electronic devices. The present application does not limit the type of electronic devices. It can be any type of device suitable for implementation, such as terminal devices and servers, etc., and the present application will not repeat them.

[0203] See also Fig.15As shown, it is a structural block diagram of the apparatus for training a large model of a wide area network in an embodiment of the present application. In some embodiments, the apparatus for training a large model of a wide area network in an example of the present application includes:

[0204] A processing unit 1501 is used to perform forward and backward propagation processing on the batch data to be processed to obtain a first gradient value of a corresponding target gradient slice;

[0205] The first receiving unit 1502 is configured to receive the second gradient value of the target gradient slice corresponding to each other, which is sent by at least one first graphics processor in other wide area network nodes; the first graphics processor is a graphics processor that is allocated with the same gradient slice as the target graphics processor;

[0206] Update parameter 1503, used to update the parameters according to the first gradient value and at least one second gradient value corresponding to the target gradient slice, and obtain the parameter value of the first parameter slice corresponding to the corresponding optimizer slice;

[0207] The second receiving unit 1504 is configured to receive parameter values ​​of second parameter slices respectively corresponding to respective optimizer slices sent by at least one first graphics processor;

[0208] The third receiving unit 1505 is used to receive parameter values ​​of third parameter slices respectively corresponding to respective optimizer slices sent by at least one second graphics processor; the second graphics processor is a graphics processor located in the same wide area network node as the target graphics processor.

[0209] In one implementation, the processing unit 1501 is further configured to:

[0210] Determining a first total number of model parameters and a second total number of graphics processors;

[0211] Obtaining a partition size according to the first total quantity and the second total quantity;

[0212] According to the partition size, each model parameter is divided to obtain multiple parameter slices; the number of model parameters contained in each parameter slice is the partition size.

[0213] In one implementation, the processing unit 1501 is further configured to:

[0214] If the first total number cannot be divided by the second total number, then fill in the parameters of each model;

[0215] A first total number of model parameters after filling is determined.

[0216] In one implementation, the processing unit 1501 is further configured to:

[0217] Generate optimizer slices corresponding to each parameter slice;

[0218] Assign different optimizer slices to different GPUs;

[0219] Generate gradient slices corresponding to each parameter slice;

[0220] For each wide area network node, the following steps are performed respectively: each gradient slice is allocated to each graphics processor of the wide area network node; the parameter slice corresponding to the gradient slice allocated to each graphics processor includes the parameter slice corresponding to the optimizer slice allocated to the graphics processor.

[0221] In one implementation, the first receiving unit 1502 is further configured to:

[0222] According to the assigned gradient slices, the graphics processors are communicated and combined to obtain multiple inter-domain process groups to aggregate the gradient values ​​of the same gradient slice; the graphics processors in the same inter-domain process group are assigned the same gradient slice and are located in different wide area network nodes;

[0223] Establishing communication between each graphics processor in each inter-domain process group;

[0224] The graphics processors in each WAN node are respectively communicated and combined to obtain the intra-domain process groups corresponding to each WAN node, so as to summarize the parameter values ​​of each parameter slice in the same WAN node;

[0225] The communication between each graphics processor in the process group in each domain is established respectively.

[0226] In one implementation, the updating unit is used to:

[0227] Performing gradient updating according to the first gradient value and the at least one second gradient value respectively corresponding to the at least one target gradient slice, to obtain a third gradient value for each target gradient slice respectively;

[0228] Based on the third gradient value of the target gradient slice corresponding to the first parameter slice, obtaining a first optimizer state value of the corresponding optimizer slice;

[0229] According to the third gradient value and the first optimizer state value, the parameters are updated to obtain the parameter value of the first parameter slice.

[0230] In one implementation, the processing unit 1501 is configured to:

[0231] For each micro-batch of the batch data, the following steps are performed: forward and backward propagation processing is performed on the micro-batch data to obtain a fourth gradient value of a target gradient slice; a fifth gradient value of the target gradient slice is received from at least one first graphics processor; a sixth gradient value is obtained according to the fourth gradient value and at least one fifth gradient value;

[0232] When it is determined that the batch data processing is completed, a gradient update is performed according to the sixth gradient values ​​corresponding to each micro-batch data to obtain a first gradient value.

[0233] The method for training a large model in a wide area network in an embodiment of the present application includes executing the following steps for each target graphics processor in each graphics processor: performing forward and backward propagation processing on the batch data to be processed to obtain the first gradient value of the corresponding target gradient slice; receiving the second gradient value of the target gradient slice corresponding to each other sent by at least one first graphics processor in other wide area network nodes; the first graphics processor is a graphics processor that is assigned the same gradient slice as the target graphics processor; according to the first gradient value and at least one second gradient value corresponding to the target gradient slice, update the parameters to obtain the parameter value of the first parameter slice corresponding to the corresponding optimizer slice; receiving the parameter value of the second parameter slice corresponding to each optimizer slice sent by at least one first graphics processor; receiving the parameter value of the third parameter slice corresponding to each optimizer slice sent by at least one second graphics processor; the second graphics processor is a graphics processor in the same wide area network node as the target graphics processor. In this way, the gradient and optimizer state are divided and stored, and the data is merged, which can take into account the reduction of the memory usage of each GPU and the improvement of the model training performance.

[0234] In an embodiment of the present application, an electronic device is provided, including:

[0235] Processor; and

[0236] The memory stores computer instructions, where the computer instructions are used to enable the processor to execute the method of any of the above embodiments.

[0237] In an embodiment of the present application, a storage medium is provided, which stores computer instructions, and the computer instructions are used to enable a computer to execute a method in any of the above-mentioned embodiments. Fig.16 FIG. 1 shows a schematic diagram of the structure of an electronic device 1600. Fig.16 As shown, the electronic device 1600 includes: a processor 1610 and a memory 1620 , and optionally, may also include a power supply 1630 , a display unit 1640 , and an input unit 1650 .

[0238] The processor 1610 is the control center of the electronic device 1600. It connects various components using various interfaces and lines, and performs various functions of the electronic device 1600 by running or executing software programs and / or data stored in the memory 1620, thereby monitoring the electronic device 1600 as a whole.

[0239] In the embodiment of the present application, the processor 1610 executes the various steps in the above embodiment when calling the computer program stored in the memory 1620.

[0240] Optionally, the processor 1610 may include one or more processing units; preferably, the processor 1610 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and applications, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 1610. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips.

[0241] The memory 1620 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, various applications, etc.; the data storage area may store data created according to the use of the electronic device 1600, etc. In addition, the memory 1620 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices, etc.

[0242] The electronic device 1600 also includes a power source 1630 (such as a battery) for supplying power to various components. The power source can be logically connected to the processor 1610 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0243] The display unit 1640 may be used to display information input by the user or information provided to the user and various menus of the electronic device 1600, and in the embodiment of the present application, is mainly used to display the display interface of each application in the electronic device 1600 and objects such as text and pictures displayed in the display interface. The display unit 1640 may include a display panel 1641. The display panel 1641 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0244] The input unit 1650 may be used to receive information such as numbers or characters input by the user. The input unit 1650 may include a touch panel 1651 and other input devices 1652. The touch panel 1651, also known as a touch screen, may collect touch operations of the user on or near it (such as operations of the user using a finger, a stylus, or any other suitable object or accessory on or near the touch panel 1651).

[0245] Specifically, the touch panel 1651 can detect the user's touch operation, detect the signal brought by the touch operation, convert these signals into touch point coordinates, send them to the processor 1610, and receive and execute the command sent by the processor 1610. In addition, the touch panel 1651 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. Other input devices 1652 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, power on / off keys, etc.), a trackball, a mouse, a joystick, etc.

[0246] Of course, the touch panel 1651 can cover the display panel 1641. When the touch panel 1651 detects a touch operation on or near it, it is transmitted to the processor 1610 to determine the type of touch event, and then the processor 1610 provides a corresponding visual output on the display panel 1641 according to the type of touch event. Fig.16 In the embodiment, the touch panel 1651 and the display panel 1641 are used as two independent components to implement the input and output functions of the electronic device 1600, but in some embodiments, the touch panel 1651 and the display panel 1641 can be integrated to implement the input and output functions of the electronic device 1600.

[0247] The electronic device 1600 may also include one or more sensors, such as a pressure sensor, a gravity acceleration sensor, a proximity light sensor, etc. Of course, according to the needs of specific applications, the electronic device 1600 may also include other components such as a camera. Since these components are not the key components used in the embodiments of the present application, Fig.16 It is not shown and will not be described in detail.

[0248] Those skilled in the art will understand that Fig.16 The electronic device is merely an example and does not limit the electronic device, and may include more or less components than those shown in the figure, or may combine certain components, or may include different components.

[0249] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0250] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the embodiments. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. The obvious changes or modifications derived therefrom are still within the scope of protection created by this application.

Claims

1. A method for training a large model in a wide area network, characterized in that: Applied to a distributed model training system, the distributed model training system includes multiple wide area network nodes; each wide area network node includes multiple graphics processors; all model parameters of the target model to be trained are divided into multiple parameter slices; each graphics processor stores all model parameters of the target model, gradient slices corresponding to some parameter slices, and optimizer slices corresponding to one parameter slice; each gradient slice is a collection of gradients corresponding to each model parameter in the corresponding parameter slice; Each optimizer shard is a collection of optimizer states corresponding to each model parameter in the corresponding parameter shard; The gradient slices corresponding to each parameter slice are respectively allocated to each graphics processor in the same WAN node; Optimizer slices corresponding to different parameter slices are respectively allocated to different graphics processors; the method includes: For each target GPU in each GPU, perform the following steps: Perform forward and backward propagation on the batch data to be processed to obtain the first gradient value of the corresponding target gradient slice; receiving second gradient values ​​of respective corresponding target gradient slices sent by at least one first graphics processor in other wide area network nodes; the first graphics processor is a graphics processor to which the same gradient slice as the target graphics processor is allocated; Perform parameter updating according to the first gradient value and at least one second gradient value corresponding to the target gradient slice, to obtain a parameter value of the first parameter slice corresponding to the corresponding optimizer slice; receiving parameter values ​​of second parameter slices respectively corresponding to respective optimizer slices sent by the at least one first graphics processor; Receiving parameter values ​​of third parameter slices respectively corresponding to respective optimizer slices sent by at least one second graphics processor; the second graphics processor is a graphics processor located in the same wide area network node as the target graphics processor; The performing forward and backward propagation processing on the batch data to be processed to obtain the first gradient value of the corresponding target gradient slice includes: performing the following steps for each micro-batch data in the batch data: performing forward and backward propagation processing on the micro-batch data to obtain a fourth gradient value of the target gradient slice; receiving a fifth gradient value of the target gradient slice respectively sent by the at least one first graphics processor; obtaining a sixth gradient value according to the fourth gradient value and at least one fifth gradient value; and when it is determined that the batch data processing is completed, performing gradient update according to the sixth gradient values ​​respectively corresponding to each micro-batch data to obtain the first gradient value; The step of performing parameter updating according to the first gradient value and at least one second gradient value corresponding to the target gradient slice to obtain the parameter value of the first parameter slice corresponding to the corresponding optimizer slice includes: performing gradient updating according to the first gradient value and at least one second gradient value respectively corresponding to at least one target gradient slice to obtain the third gradient value of each target gradient slice respectively; obtaining the first optimizer state value of the corresponding optimizer slice based on the third gradient value of the target gradient slice corresponding to the first parameter slice; and performing parameter updating according to the third gradient value and the first optimizer state value to obtain the parameter value of the first parameter slice.

2. The method according to claim 1, characterized in that: Before performing forward and backward propagation processing on the batch data to be processed, the method further includes: Determining a first total number of model parameters and a second total number of graphics processors; Obtaining a partition size according to the first total number and the second total number; According to the partition size, each model parameter is divided to obtain a plurality of parameter slices; the number of model parameters contained in each parameter slice is the partition size.

3. The method according to claim 2, characterized in that Before obtaining the partition size according to the first total number and the second total number, the method further includes: If the first total number cannot be divided by the second total number, performing parameter filling on each model parameter; A first total number of model parameters after filling is determined.

4. The method according to claim 1, characterized in that: Before performing forward and backward propagation processing on the batch data to be processed, the method further includes: Generate optimizer slices corresponding to each parameter slice; Assign different optimizer slices to different GPUs; Generate gradient slices corresponding to each parameter slice; For each wide area network node, the following steps are performed respectively: each gradient slice is allocated to each graphics processor of the wide area network node; the parameter slice corresponding to the gradient slice allocated to each graphics processor includes the parameter slice corresponding to the optimizer slice allocated to the graphics processor.

5. The method according to claim 4, characterized in that Before receiving the second gradient value of the target gradient slice corresponding to each other sent by at least one first graphics processor in other wide area network nodes, the method includes: According to the assigned gradient slices, the graphics processors are communicated and combined to obtain multiple inter-domain process groups to aggregate the gradient values ​​of the same gradient slice; the graphics processors in the same inter-domain process group are assigned the same gradient slice and are located in different wide area network nodes; Establishing communication between each graphics processor in each inter-domain process group; The graphics processors in each WAN node are respectively communicated and combined to obtain the intra-domain process groups corresponding to each WAN node, so as to summarize the parameter values ​​of each parameter slice in the same WAN node; The communication between each graphics processor in the process group in each domain is established respectively.

6. A device for training a large model in a wide area network, characterized in that: Each target graphics processor applied to a distributed model training system, the distributed model training system includes multiple wide area network nodes; each wide area network node includes multiple graphics processors; all model parameters of the target model to be trained are divided into multiple parameter slices; each graphics processor stores all model parameters of the target model, gradient slices corresponding to some parameter slices, and optimizer slices corresponding to one parameter slice; each gradient slice is a collection of gradients corresponding to each model parameter in the corresponding parameter slice; Each optimizer shard is a collection of optimizer states corresponding to each model parameter in the corresponding parameter shard; The gradient slices corresponding to each parameter slice are respectively allocated to each graphics processor in the same WAN node; The optimizer slices corresponding to different parameter slices are assigned to different GPUs; The device comprises: A processing unit, used for performing forward and backward propagation processing on the batch data to be processed to obtain a first gradient value of a corresponding target gradient slice; A first receiving unit is configured to receive second gradient values ​​of respective corresponding target gradient slices sent by at least one first graphics processor in other wide area network nodes; the first graphics processor is a graphics processor to which the same gradient slice as the target graphics processor is allocated; Update parameters, used to update parameters according to the first gradient value and at least one second gradient value corresponding to the target gradient slice, and obtain the parameter value of the first parameter slice corresponding to the corresponding optimizer slice; A second receiving unit, configured to receive parameter values ​​of second parameter slices respectively corresponding to respective optimizer slices sent by the at least one first graphics processor; A third receiving unit is used to receive parameter values ​​of third parameter slices respectively corresponding to respective optimizer slices sent by at least one second graphics processor; the second graphics processor is a graphics processor located in the same wide area network node as the target graphics processor; The processing unit is configured to: for each micro-batch data in the batch data, respectively, perform the following steps: perform forward and backward propagation processing on the micro-batch data to obtain a fourth gradient value of the target gradient slice; receive a fifth gradient value of the target gradient slice respectively sent by the at least one first graphics processor; obtain a sixth gradient value according to the fourth gradient value and at least one fifth gradient value; when it is determined that the batch data processing is completed, perform gradient update according to the sixth gradient values ​​respectively corresponding to each micro-batch data to obtain the first gradient value; The update parameters are used to: perform gradient update according to a first gradient value and at least one second gradient value respectively corresponding to at least one target gradient slice, and obtain a third gradient value of each target gradient slice respectively; obtain a first optimizer state value of a corresponding optimizer slice based on the third gradient value of the target gradient slice corresponding to the first parameter slice; perform parameter update according to the third gradient value and the first optimizer state value, and obtain a parameter value of the first parameter slice.

7. An electronic device, characterized in that: include: processor; as well as A memory storing computer instructions, wherein the computer instructions are used to enable the processor to execute the method according to any one of claims 1 to 5.

8. A storage medium, characterized in that: Computer instructions are stored, and the computer instructions are used to make a computer execute the method according to any one of claims 1 to 5.

Images