Model training method based on distributed system, computing node and system

Abstract

A model training method, computing nodes and system based on a distributed system, the distributed system comprises n groups of computing nodes, each group of computing nodes comprises m computing nodes, the m computing nodes in each group correspond to m parameter sets included in a target model respectively, the method comprises the following steps: the m computing nodes in each group are jointly trained for multiple times, enabling each computing node in each group to obtain a first value of each parameter in the corresponding parameter set; the n first computing nodes corresponding to the first parameter in the n groups determine a second value corresponding to the first parameter based on n first values of the first parameter; and the n first computing nodes synchronize the value of the first parameter to the second value respectively.

Description

Technical Field

[0001] The embodiments of this specification belong to the field of machine learning, and in particular, to a model training method, computing node, and system based on a distributed system. Background Art

[0002] The Distributed Data Parallel (DDP) framework is an important framework for implementing distributed training, which allows deep learning models to be trained in parallel on multiple processing units (such as GPUs). Specifically, the training data can be distributed to multiple processing units, each of which has a complete parameter copy of the model, and each processing unit can use its training samples to train the model copy. Each processing unit synchronizes the gradients of the model parameters (All-reduce) after each back-propagation process, so that the model parameters of each processing unit are consistent. However, this training method synchronizes the gradients in each training, and there is a problem of frequent communication. In addition, there are always random slow nodes in multiple processing units, and the multiple processing units need to wait for the slow nodes to complete the training before they can synchronize the gradients in each training, resulting in a slow training speed.

[0003] To address the above problems, in a local stochastic gradient descent (Local SGD) method, each node independently and in parallel executes multiple local update steps, and then all nodes synchronize the parameters to reduce the communication frequency and the impact of random slow nodes.

[0004] However, the above Local SGD method cannot be applied to the training of large language models (LLM). This is because the above Local SGD method needs to save a complete parameter copy, that is, a complete model parameter, in a single node, but the storage capacity of a single node cannot store tens of billions of parameter copies of LLM. Summary of the invention

[0005] The purpose of the present invention is to provide a model training method based on a distributed system, so that model training can be performed more efficiently for LLM.

[0006] In a first aspect, the present specification provides a model training method based on a distributed system, wherein the distributed system includes n groups of computing nodes, each group of computing nodes includes m computing nodes, and the m computing nodes in each group correspond to m parameter sets included in the target model, respectively. The method includes:

[0007] The m computing nodes in each group jointly perform multiple trainings so that each computing node in each group obtains the first value of each parameter in its corresponding parameter set;

[0008] The n first computing nodes corresponding to the first parameter in the n groups determine a second value corresponding to the first parameter based on the n first values ​​of the first parameter;

[0009] The n first computing nodes respectively synchronize the value of the first parameter to the second value.

[0010] In one embodiment, the m computing nodes in each group jointly perform multiple trainings, including:

[0011] In each training, each computing node in each group obtains the values ​​of other parameters of the target model from other computing nodes in the group to obtain the values ​​of all parameters of the target model, and updates the values ​​of each parameter corresponding to the computing node based on the values ​​of all parameters of the target model.

[0012] In one embodiment, the target model is divided into a plurality of modules arranged in series, the first parameter is a parameter in a first module among the plurality of modules,

[0013] The n first computing nodes corresponding to the first parameter in the n groups determine a second value corresponding to the first parameter based on the n first values ​​of the first parameter, including:

[0014] The n first computing nodes respectively determine, during a forward propagation process of a next training of the multiple trainings, a second value corresponding to the first parameter based on the n first values ​​of the first parameter.

[0015] In one embodiment, the method further comprises:

[0016] Each of the first computing nodes performs calculations on the first module in the forward propagation based on the second value of the first parameter, and sends the second value to other computing nodes in the same group for use by other computing nodes in calculating the first module in the forward propagation.

[0017] In one implementation, the n first computing nodes respectively determine, during a forward propagation process of a next training of the multiple trainings, based on the n first values ​​of the first parameter, a second value corresponding to the first parameter, including:

[0018] The first computing node determines a second value corresponding to the first parameter in parallel with the calculation of a second module in the forward propagation process, and the second module is a previous module of the first module.

[0019] In one implementation, the n first computing nodes corresponding to the first parameter in the n groups determine the second value corresponding to the first parameter based on the n first values ​​of the first parameter, including:

[0020] The n first computing nodes calculate the difference between each first value and the current synchronization value of the first parameter, determine the abnormal value among the n first values ​​based on the difference, and determine the second value based on multiple first values ​​among the n first values ​​excluding the abnormal value.

[0021] In one implementation, determining the second value based on a plurality of first values ​​excluding the abnormal value among the n first values ​​comprises:

[0022] The weight of each difference is determined based on the multiple difference values ​​corresponding to the multiple first values, a third value is calculated based on the multiple difference values ​​and the weight of each difference, and the second value is calculated based on the synchronization value and the third value.

[0023] In one embodiment, the calculating of the third value based on the plurality of differences and the weight of each difference comprises:

[0024] A fourth value is obtained based on the multiple differences and weights of the differences, and the fourth value is processed based on a preset threshold and the multiple differences to obtain the third value.

[0025] In one implementation, the m computing nodes in each group jointly perform multiple trainings, including: the number of times each group of computing nodes performs joint training is determined based on a preset first time interval.

[0026] In one embodiment, the communication speed between the m computing nodes in each group is faster than the communication speed between computing nodes in different groups.

[0027] In one embodiment, the method further includes a warm-up training phase, which includes backpropagation. The method further includes: in the warm-up training phase, the n first computing nodes in the n groups corresponding to the first parameters synchronize the model gradients in the backpropagation.

[0028] A second aspect of the present specification provides a model training method based on a distributed system, wherein the distributed system includes n groups of computing nodes, each group of computing nodes includes m computing nodes, and the m computing nodes in each group correspond to the m parameter sets included in the target model, respectively. The method is executed by a second computing node in a first group of computing nodes, and the second computing node corresponds to a first parameter of the model. The method includes:

[0029] Performing multiple trainings in conjunction with other computing nodes in the first group of computing nodes to obtain a first value of the first parameter;

[0030] Determine, with n-1 first computing nodes corresponding to the first parameter in other groups based on the n first values ​​of the first parameter, a second value corresponding to the first parameter;

[0031] The value of the first parameter is synchronized to the second value.

[0032] In a third aspect, the present specification provides a second computing node in a distributed system, the distributed system comprising n groups of computing nodes, each group of computing nodes comprising m computing nodes, the m computing nodes in each group respectively corresponding to m parameter sets included in a target model, the second computing node corresponding to a first parameter of the model, and the second computing node comprising:

[0033] A training unit, configured to perform multiple trainings in conjunction with other computing nodes in the first group of computing nodes to obtain a first value of the first parameter;

[0034] A determining unit, configured to determine, together with n-1 first computing nodes corresponding to the first parameter in other groups, a second value corresponding to the first parameter based on the n first values ​​of the first parameter;

[0035] A synchronization unit is used to synchronize the value of the first parameter to the second value.

[0036] A fourth aspect of the present specification provides a distributed system, the distributed system comprising n groups of computing nodes, each group of computing nodes comprising m computing nodes, the m computing nodes in each group respectively corresponding to the m parameter sets included in the target model,

[0037] The m computing nodes in each group of computing nodes are used to jointly perform multiple trainings, so that each computing node in each group obtains the first value of each parameter in its corresponding parameter set;

[0038] The n first computing nodes corresponding to the first parameter in the n groups are used to determine a second value corresponding to the first parameter based on the n first values ​​of the first parameter;

[0039] The n first computing nodes are further configured to respectively synchronize the value of the first parameter to the second value.

[0040] A fifth aspect of the present specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method described in the first aspect or the second aspect.

[0041] A sixth aspect of the specification provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in the first aspect or the second aspect is implemented.

[0042] A seventh aspect of the present specification provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method described in the first aspect or the second aspect.

[0043] Through the model training scheme provided by the embodiment of this specification, by adopting the hierarchical distribution structure of the two-dimensional device network, the m computing nodes in a single model division group respectively store part of the parameters of the model, thereby solving the problem of storage overflow of a single computing node. In addition, during the model training process, the n computing nodes in a single model replication group only need to synchronize their corresponding part of the parameters, and do not need to synchronize all the parameters of the model between the computing nodes, which reduces the communication volume of a single computing node, and multiple model replication groups synchronize the model parameters corresponding to each group in parallel, thereby improving the efficiency of parameter synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0045] Figure 1 This is a system architecture diagram of distributed training in an embodiment of this specification;

[0046] Figure 2 This is a schematic diagram of the process of the warm-up phase of distributed model training in the embodiments of this specification;

[0047] Figure 3 This is a flowchart of a model training method based on a distributed system in an embodiment of this specification;

[0048] Figure 4 This is a schematic diagram of the process of the periodic synchronization training stage in the embodiment of this specification;

[0049] Figure 5 A schematic diagram of setting the number of joint training times in an embodiment of this specification;

[0050] Figure 6 A schematic diagram of a process of performing a reduction calculation on a parameter value in an embodiment of this specification;

[0051] Figure 7 Schematic diagram of the process of synchronizing parameters in forward propagation in an embodiment of this specification;

[0052] Figure 8 An architectural diagram of a computing node provided in an embodiment of this specification. DETAILED DESCRIPTION

[0053] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0054] In order to solve the above problems existing in the prior art, the embodiments of this specification propose a model training method based on a distributed system for LLM, so as to perform efficient distributed training for LLM.

[0055] Figure 1 This is a system architecture diagram of distributed training in the embodiments of this specification. Figure 1 As shown in , the system may include n*m ​​computing nodes. A computing node refers to a unit that can perform computing independently, which may include a separate computing device, a virtual computing device, or a processing unit (such as a GPU), etc., without limitation.

[0056] For ease of understanding, the n*m ​​computing nodes may be arranged into an array of n rows*m columns, wherein any i-th row in the array includes m computing nodes W i1 ~W im , the m computing nodes W i1 ~W im A model partition group is formed (i.e., the i-th model partition group among n model partition groups), and each computing node W in the group ik Corresponding to some parameters θ of the model (e.g. LLM) k , and the m computing nodes maintain a parameter copy of all the parameters of the model. It can be understood that the parameter θ k It is used to represent some parameters of the model, which may include one or more parameters. In one embodiment, the model may include L modules (module 1 to module L), and each computing node W ik The corresponding partial parameters θ k It can include some parameters θ of each module in L modules k1 ~θ kL .

[0057] In each model training, the m computing nodes in a single model partition group can locally update the values ​​of the model parameters through joint training. In one embodiment, the m computing nodes in a single model partition group can be m GPUs in a single computing device, thereby reducing the communication time between the m computing nodes during the model training process. It can be understood that the m computing nodes in a single model partition group are not limited to being m GPUs in a single computing device. For example, it can be m computing devices in a local area network, and the communication speed between the m computing devices is faster than the communication speed between computing nodes in different model partition groups.

[0058] The jth column in the array includes n computing nodes W 1j ~W nj , the n computing nodes are computing nodes corresponding to the same model parameters in the n model partitioning groups, which constitute a model replication group (i.e., the j-th model replication group in the m model replication groups). Figure 1 As shown in 11 ~W n1 With parameter θ 1 Correspondingly, W 12 ~W n2 With parameter θ 2 Each model replication group can periodically synchronize parameters between computing nodes in the same group similar to the above-mentioned Local SGD to reduce the communication overhead caused by parameter synchronization.

[0059] That is to say, the system in the embodiment of this specification adopts a hierarchical distribution structure of a two-dimensional device network. By adopting this structure, the m computing nodes in a single model division group respectively store part of the parameters of the model, thereby solving the problem of storage overflow of a single computing node. In addition, during the model training process, the n computing nodes in a single model replication group only need to synchronize their corresponding part of the parameters, and there is no need to synchronize all the parameters of the model between the computing nodes, which reduces the communication volume of a single computing node, and multiple model replication groups synchronize the model parameters corresponding to each group in parallel, thereby improving the efficiency of parameter synchronization.

[0060] The model training process in the embodiment of this specification may include two stages: an initial warm-up stage and a periodic synchronous training stage. In the warm-up stage, the number of warm-up steps t may be preset. w , before model training w In each training, the gradients of the model parameters are synchronized by each computing node in each model replication group, so that the initial training stage is more stable. After that, it can be switched to the periodic synchronization training stage. The training process of the above two stages will be described below with reference to the accompanying drawings.

[0061] Figure 2This is a schematic diagram of the process of the warm-up phase of distributed model training in an embodiment of this specification. Figure 2 Only the computation nodes W in the first model partition group are shown. 11 and computing nodes W 12 , and the computational nodes W in the second model partition group 21 and computing nodes W 22 , among which, Figure 1 As shown in 11 and computing nodes W 21 Duplicate the group in the first model.

[0062] like Figure 2 As shown, each model training process may include three parts: forward propagation, backward propagation, and local parameter update.

[0063] Among them, in one training, the node W is calculated 11 For example, in the forward propagation part, the calculation node W 11 The processing is done sequentially from module 1 to module L. Specifically, for module 1, node W is calculated. 11 First, the other parameters (θ) in module 1 of the model are aggregated from other computing nodes in the model partition group (i.e., the first model partition group) i1 ,i≠1), thereby obtaining all the parameters of module 1 (θ i1 , i=[1,m]), the value of the calculation node W 11 The corresponding training samples are input into module 1, that is, the training samples are calculated based on all the parameters of module 1 to obtain the output of module 1, and the parameters (θ i1 , i≠1) is released, and only computing node W is retained 11 The corresponding parameter θ of module 1 11 , to save storage space (such as video memory), and input the output of module 1 to module 2. Similar to module 1, in module 2, the computing node W 11 The other parameters of module 2 are gathered from other computing nodes in the model partition group to obtain all the parameters of module 2, the output of module 1 is calculated based on the parameters of module 2 to obtain the output of module 2, the parameters corresponding to other computing nodes are released, and the output of module 2 is input to module 3. This process is repeated until the calculation of module L is completed to obtain the output of module L, which is also the output of the complete model.

[0064] Then enter the back propagation part of this training. Figure 2 , in the back propagation part, the calculation node W 11 The processing is performed sequentially from module L to module 1. Specifically, in module L, the computing node W 11First, the values ​​of other parameters (θ) of module L in the model are gathered from other computing nodes in the module partition group. iL ,i≠1), thereby obtaining all the parameters of module L (θ iL , i=[1,m]), and backward gradient calculation is performed based on all parameters of module L, the output of the above model, and the labels in the above training samples to obtain the parameters θ in each module L. L Then, we calculate the gradient of node W 11 The other computing nodes in the model partition group are aggregated with computing node W 11 The corresponding parameter θ 1L The gradient of the parameter θ 1L The m gradient values ​​of θ are reduced (e.g., averaged) to obtain the updated parameter θ. 1L The gradient value of . Among them, reduction is an operation used to combine multiple elements in a set into a single result through some operation. The reduction operation in the embodiments of this specification may include summation, product, average, maximum, minimum and other operations, which are not limited in the embodiments of this specification.

[0065] Afterwards, calculate node W 11 The other computing nodes W in the model replication group to which it belongs k1 ,k=[2,m] synchronization parameterθ 1L The gradient value of the parameter θ calculated for each computing node in the model replication group 1L The gradient value of is reduced and the final parameter θ is obtained. 1L Then, calculate the gradient value of node W 11 Release the model parameter values ​​corresponding to other computing nodes in the model partition group, and calculate the parameters θ calculated by module L 1L The gradient value of is input to module L-1. In module L-1, the calculation node W 11 Similarly, the parameters are aggregated, backward calculations are performed, gradients are reduced, gradients are synchronized, and parameters are released, and then module L-2 is entered. And so on, until module 1 is processed.

[0066] After completing the above back propagation part, the calculation node W 11 Update the parameters locally, that is, use the parameters θ calculated for each module 11 ~θ 1L Each gradient value updates the parameter θ 11 ~θ 1L The value of .

[0067] As mentioned above, each computing node performs wAfter the warm-up training, you can enter the cycle synchronization training phase.

[0068] Figure 3 Flow chart of the model training method based on the distributed system in the embodiment of this specification. The process corresponds to the above-mentioned periodic synchronous training stage. In order to facilitate drawing, only the computing node W in the first model division group is shown in this figure. 11 and computing nodes W 12 , and the computational nodes W in the second model partition group 21 and computing nodes W 22 , among which, Figure 1 As shown in 11 and computing nodes W 21 In the first model copy group. It can be understood that in Figure 3 In the illustrated embodiment, a model partition group may include two or more computing nodes, and a model replication group may include two or more computing nodes. Figure 3 The method shown is also applicable to the case where a model partition group includes multiple computing nodes and / or a model replication group includes multiple computing nodes.

[0069] like Figure 3 As shown, in step S301, the m computing nodes in each model division group jointly perform multiple trainings.

[0070] For n model partition groups, multiple trainings are performed in each model partition group to update the model parameters locally. For example, the m computing nodes (including computing node W) in the first model partition group 11 and computing nodes W 12 ) are jointly trained multiple times, and the m computing nodes in the second model partition group (including computing node W 21 and computing nodes W 22 ) are combined for multiple training sessions, and so on.

[0071] Figure 4 Schematic diagram of the process of the periodic synchronization training stage in the embodiment of this specification.

[0072] like Figure 4As shown, the difference between the periodic synchronization stage and the above-mentioned warm-up stage is that each computing node in the same model replication group no longer synchronizes the gradient during the backward propagation process, but periodically synchronizes the parameters before the forward calculation of each module during the forward propagation process. Similarly, compared to Local SGD, in the embodiment of this specification, each computing node in the model replication group only needs to synchronize part of the parameters corresponding to the model replication group, and does not need to synchronize all the parameters of the model, thereby reducing the communication volume of each computing node, and multiple model replication groups synchronize the parameters of the model in parallel, which speeds up the synchronization speed of the model parameters.

[0073] Specifically, in a model training of each computing node in a single model partition group, in the forward propagation, the operations for each module of the model include aggregation parameters, forward calculation and release parameters, that is, the operations of synchronization parameters are not included, where: Figure 4 The "synchronization parameters" in the forward propagation are shown with dotted lines, indicating that this operation is not performed in every training, but only periodically. In the backward propagation, the operations for each module of the model include aggregation parameters, backward calculation, gradient reduction, and gradient release. After the backward propagation, each computing node locally updates the corresponding part of the model parameters of the computing node.

[0074] In one of the multiple joint trainings, the node W is calculated. 11 For example, in the forward propagation part, the calculation node W 11 The processing is done sequentially from module 1 to module L. Specifically, for module 1, node W is calculated. 11 First, the other parameters (θ) in module 1 of the model are aggregated from other computing nodes in the model partition group (i.e., the first model partition group) i1 ,i≠1), thereby obtaining all the parameters of module 1 (θ i1 , i=[1,m]), the value of the calculation node W 11 The corresponding training samples are input into module 1, that is, the training samples are calculated based on all the parameters of module 1 to obtain the output of module 1, and the parameters (θ i1 , i≠1) is released, and only computing node W is retained 11 The corresponding parameter θ of module 1 11 , to save storage space (such as video memory), and input the output of module 1 to module 2. Similar to module 1, in module 2, the computing node W 11The other parameters of module 2 are gathered from other computing nodes in the model partition group to obtain all the parameters of module 2, the output of module 1 is calculated based on the parameters of module 2 to obtain the output of module 2, the parameters corresponding to other computing nodes are released, and the output of module 2 is input to module 3. This process is repeated until the calculation of module L is completed to obtain the output of module L, which is also the output of the complete model.

[0075] Then enter the back propagation part of this training. Figure 4 , in the back propagation part, the calculation node W 11 The processing is performed sequentially from module L to module 1. Specifically, in module L, the computing node W 11 First, the values ​​of other parameters (θ) of module L in the model are gathered from other computing nodes in the module partition group. iL ,i≠1), thereby obtaining all the parameters of module L (θ iL , i=[1,m]), and backward gradient calculation is performed based on all parameters of module L, the output of the above model, and the labels in the above training samples to obtain the parameters θ in each module L. L Then, we calculate the gradient of node W 11 The other computing nodes in the model partition group are aggregated with computing node W 11 The corresponding parameter θ 1L The gradient of the parameter θ 1L The m gradient values ​​of θ are reduced (e.g., averaged) to obtain the updated parameter θ. 1L The gradient value of .

[0076] Afterwards, calculate node W 11 Release the model parameter values ​​corresponding to other computing nodes in the model partition group, and calculate the parameters θ calculated by module L 1L The gradient value of is input to module L-1. In module L-1, the calculation node W 11 Similarly, the parameters are aggregated, backward calculations are performed, gradients are reduced, and parameters are released, and then module L-2 is entered. And so on, until module 1 is processed.

[0077] After completing the above back propagation part, the calculation node W 11 Update the parameters locally, that is, use the parameters θ calculated for each module 11 ~θ 1L Each gradient value updates the parameter θ 11 ~θ 1L The value of .

[0078] The number of times each model partition group is jointly trained can be set in a variety of ways. Figure 5FIG. 1 is a schematic diagram of setting the number of joint training times in the embodiment of this specification. Figure 5 As shown in the upper part, in one embodiment, the number of joint training times t can be preset s , for example s = 2, each computing node in a single model partition group has been trained twice since the last synchronization ended before the next synchronization. In this case, the fast nodes in a single model partition group need to spend more time (such as Figure 5 The node waits for the slow node to complete two trainings (as shown in “Idle” in the upper middle part), and then all nodes in a single model partition group are synchronized together.

[0079] like Figure 5 As shown in the lower part, in another embodiment, the time interval T of joint training can be preset, and each computing node in a single model partition group starts training from the end time of the last synchronization and performs the next synchronization after the training time reaches T. In this case, the fast nodes in a single model partition group are faster than those in Figure 5 The upper part takes less time (such as Figure 5 The node waits for the slow node to complete training (as shown in the “Idle” in the lower middle section), and then all nodes in a single model partition group are synchronized together.

[0080] In step S303, after each model division group has been trained multiple times, each computing node obtains an updated value of its corresponding parameter. 11 Get its corresponding parameter θ 1 The value V1, the computational node W in the second model partition group 21 Get its corresponding parameter θ 1 The value V2.

[0081] In step S305, each model replication group determines a synchronization value of the parameter based on n values ​​of the corresponding parameter.

[0082] like Figure 1 As shown in , each computing node in the model replication group corresponds to the same model parameters. For example, n computing nodes W in the first model replication 11 ~W n1 Both with parameter θ 1 Corresponding. The n computing nodes respectively calculate the parameters θ in the above steps 1 The values ​​V1~Vn of the n computing nodes W 11 ~W n1 The parameter θ can be 1 The values ​​V1~Vn are calculated to obtain the parameter θ 1 The synchronization value.

[0083] Figure 6The present invention is a schematic diagram of a process of reducing and calculating parameter values ​​in an embodiment of the present specification.

[0084] like Figure 6 As shown in FIG, it is assumed that the first model replication group includes computing nodes W 11 ~W 41 , which calculates the parameters θ 1 Subtract the parameter θ from the parameter values ​​V1 to V4. 1 The current synchronization value V (that is, the θ determined by the last synchronization) 1 ), and obtain virtual gradients g1~g4 respectively.

[0085] The outliers in the virtual gradients g1 to g4 can be determined. In one embodiment, the norm of the virtual gradient can be used as a measurement indicator. Specifically, the z-test method can be used for statistical analysis. Let G i =‖g i ‖ 2 Represents node W i1 The virtual gradient norm of , the corresponding score z-score is:

[0086]

[0087] in, Represents G i The average value of G i When the node W i1 Satisfy z i >δ, it is determined as an outlier, where δ is a manually set threshold. and σ are updated using exponential moving average during training as and σ′:

[0088]

[0089] Where α is a weight factor. In this process, if all computing nodes in the model replication group are judged as abnormal nodes, all parameters will be rolled back to the parameters V of the last synchronization.

[0090] Continue to refer Figure 6 After removing the abnormal nodes in the model replication group, considering that the larger virtual gradient may affect the overall direction, the remaining virtual gradients can be weighted based on the norm of the virtual gradient. For each normal node, the weight of each node can be set as:

[0091]

[0092] Then, the weighted sum of each virtual gradient is performed to obtain the synchronized virtual gradient g:

[0093] g=Σw i g i

[0094] In addition, the update step size can be limited by the gradient clipping strategy. The clipping factor β can be set as follows:

[0095]

[0096] Where G = ‖g‖ 2 , represents the synchronized virtual gradient norm, is a preset threshold, ∈ is a small positive number. The virtual gradient can then be clipped as follows:

[0097]

[0098] Afterwards, based on the virtual gradient Update the current parameter synchronization value V to obtain the updated synchronization value

[0099] In step S307, the computing nodes in each model replication group synchronize their parameter values ​​to synchronized values.

[0100] Specifically, reference Figure 6 , each computing node in the first model replication group calculates the parameter θ 1 Synchronization value After that, we will get the parameters θ 1 The value of is synchronized to the synchronization value

[0101] In one embodiment, each computing node W in the model replication group 11 ~W n1 After completing multiple joint trainings within the above-mentioned model division groups, the synchronization value can be calculated and the parameters can be updated to the synchronization value.

[0102] In another embodiment, each computing node W in the model replication group 11 ~W n1 The parameters θ of each module can be adjusted in the forward propagation of the next training after the above multiple joint trainings. 11 ~θ 1L The synchronization value is calculated and the parameter values ​​of each module are updated to the synchronization value.

[0103] Specifically, Figure 4 As shown, in the training including the synchronization parameter operation, in the forward propagation, the parameters are first synchronized in each model replication group. Taking the first group of model replication groups as an example, each computing node W 11 ~W n1 In module 1, the parameter θ11 Multiple parameter values ​​V 11 ~V n1 Perform the reduction calculation and obtain the parameter θ 11 Synchronization value Then calculate the node W 11 ~W n1 Based on the synchronization value The subsequent operations of aggregation parameters, forward calculation and release parameters in module 1 are performed. Similarly, each computing node W 11 ~W n1 In module 2, the parameter θ 12 Multiple parameter values ​​V 12 ~V n2 Perform the reduction calculation and obtain the parameter θ 12 Synchronization value Then calculate the node W 11 ~W n1 Based on the synchronization value The subsequent operations of parameter aggregation, forward calculation and parameter release in module 2 are performed. This process is repeated until the synchronization parameters, parameter aggregation, forward calculation and parameter release are completed for module L.

[0104] Figure 7 Schematic diagram of the process of synchronizing parameters in forward propagation in an embodiment of this specification.

[0105] like Figure 7 As shown, the computing node W in the first model replication group 11 ~W n1 For example ( Figure 7 The computing node W is schematically shown in FIG. 11 and computing nodes W 21 ), in step S701, each computing node in the model replication group performs forward propagation in module 1 based on the parameter θ 11 Multiple values ​​of the parameter θ are determined 11 Synchronization value

[0106] In step S703, each computing node in the model replication group sets the parameter θ 11 The value of is synchronized to the value

[0107] In step S705, in parallel with the forward propagation in module 1 in step S707, the computing nodes W in the model replication group 11 ~W n1 Based on the parameters θ corresponding to module 2 12 Multiple values ​​of the parameter θ are determined 12 Synchronization value

[0108] In step S709, each computing node in the model replication group sets the parameter θ 12 The value of is synchronized to the value

[0109] exist Figure 7 In the process shown, each computing node in the model replication group can perform synchronization parameter operations in module 2 while processing the forward calculation of module 1, thereby saving the time cost of synchronizing parameters and further accelerating the model training process.

[0110] Figure 8 An architecture diagram of a computing node provided in an embodiment of this specification, the computing node belongs to a distributed system, the distributed system includes n groups of computing nodes, each group of computing nodes includes m computing nodes, the m computing nodes in each group correspond to the m parameter sets included in the target model, and the computing nodes include:

[0111] A training unit 81 is used to perform multiple trainings in conjunction with other computing nodes in the first group of computing nodes to obtain a first value of the first parameter;

[0112] A determining unit 82, configured to determine, with n-1 first computing nodes corresponding to the first parameter in other groups based on the n first values ​​of the first parameter, a second value corresponding to the first parameter;

[0113] The synchronization unit 83 is configured to synchronize the value of the first parameter to the second value.

[0114] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0115] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0116] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, the present application does not exclude that with the development of computer technology in the future, the computer that implements the functions of the above embodiments may be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0117] Although one or more embodiments of the present specification provide method operation steps as described in the embodiments or flow charts, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps, and does not represent the only execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements. For example, if the words first, second, etc. are used to represent the name, they do not represent any specific order.

[0118] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing one or more of the present specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0119] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0120] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0122] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0123] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0124] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage, graphene storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0125] It should be understood by those skilled in the art that one or more embodiments of the present specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0126] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0127] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0128] The above description is only an example of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. For those skilled in the art, one or more embodiments of the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims.

Claims

1. A model training method based on a distributed system, wherein the distributed system includes n groups of computing nodes, each group of computing nodes includes m computing nodes, and the m computing nodes in each group correspond to m parameter sets included in the target model, respectively, and the method includes: The m computing nodes in each group of computing nodes jointly perform multiple trainings so that each computing node in each group obtains the first value of each parameter in its corresponding parameter set; The n first computing nodes corresponding to the first parameter in the n groups determine a second value corresponding to the first parameter based on the n first values ​​of the first parameter; The n first computing nodes respectively synchronize the value of the first parameter to the second value.

2. The method according to claim 1, wherein the m computing nodes in each group jointly perform multiple trainings, comprising: In each training, each computing node in each group obtains the values ​​of other parameters of the target model from other computing nodes in the group to obtain the values ​​of all parameters of the target model, and updates the values ​​of each parameter corresponding to the computing node based on the values ​​of all parameters of the target model.

3. The method according to claim 1, wherein the target model is divided into a plurality of modules arranged in series, the first parameter is a parameter in a first module of the plurality of modules, The n first computing nodes corresponding to the first parameter in the n groups determine a second value corresponding to the first parameter based on the n first values ​​of the first parameter, including: The n first computing nodes respectively determine, during a forward propagation process of a next training of the multiple trainings, a second value corresponding to the first parameter based on the n first values ​​of the first parameter.

4. The method according to claim 3, further comprising: Each of the first computing nodes performs calculations on the first module in the forward propagation based on the second value of the first parameter, and sends the second value to other computing nodes in the same group for use by other computing nodes in calculating the first module in the forward propagation.

5. The method according to claim 3, wherein the n first computing nodes respectively determine, during a forward propagation process of a next training of the multiple trainings, a second value corresponding to the first parameter based on the n first values ​​of the first parameter, comprising: The first computing node determines a second value corresponding to the first parameter in parallel with the calculation of a second module in the forward propagation process, and the second module is a previous module of the first module.

6. The method according to claim 1, wherein the n first computing nodes corresponding to the first parameter in the n groups determine the second value corresponding to the first parameter based on the n first values ​​of the first parameter, comprising: The n first computing nodes calculate the difference between each first value and the current synchronization value of the first parameter, determine the abnormal value among the n first values ​​based on the difference, and determine the second value based on multiple first values ​​among the n first values ​​excluding the abnormal value.

7. The method according to claim 6, wherein determining the second value based on a plurality of first values ​​excluding the abnormal value among the n first values ​​comprises: The weight of each difference is determined based on the multiple difference values ​​corresponding to the multiple first values, a third value is calculated based on the multiple difference values ​​and the weight of each difference, and the second value is calculated based on the synchronization value and the third value.

8. The method according to claim 7, wherein the calculating the third value based on the plurality of difference values ​​and the weights of the respective difference values ​​comprises: A fourth value is obtained based on the multiple differences and weights of the differences, and the fourth value is processed based on a preset threshold and the multiple differences to obtain the third value.

9. The method according to claim 1, wherein the m computing nodes in each group jointly perform multiple trainings, comprising: The number of times each group of computing nodes performs joint training is determined based on a preset first time interval.

10. The method according to claim 1, wherein: The communication speed between the m computing nodes in each group is faster than the communication speed between computing nodes in different groups.

11. The method according to claim 1, further comprising a warm-up training phase, wherein the warm-up training phase comprises back propagation, and the method further comprises: In the warm-up training phase, the n first computing nodes corresponding to the first parameters in the n groups synchronize the gradients of the model parameters in the backward propagation.

12. A model training method based on a distributed system, the distributed system comprising n groups of computing nodes, each group of computing nodes comprising m computing nodes, the m computing nodes in each group respectively corresponding to the m parameter sets included in the target model, the method being executed by a second computing node in the first group of computing nodes, the second computing node corresponding to the first parameter of the model, the method comprising: Performing multiple trainings in conjunction with other computing nodes in the first group of computing nodes to obtain a first value of the first parameter; Determine, with n-1 first computing nodes corresponding to the first parameter in other groups based on the n first values ​​of the first parameter, a second value corresponding to the first parameter; The value of the first parameter is synchronized to the second value.

13. A second computing node in a distributed system, the distributed system comprising n groups of computing nodes, each group of computing nodes comprising m computing nodes, the m computing nodes in each group respectively corresponding to m parameter sets included in a target model, the second computing node corresponding to a first parameter of the model, the second computing node comprising: A training unit, configured to perform multiple trainings in conjunction with other computing nodes in the first group of computing nodes to obtain a first value of the first parameter; A determining unit, configured to determine, together with n-1 first computing nodes corresponding to the first parameter in other groups, a second value corresponding to the first parameter based on the n first values ​​of the first parameter; A synchronization unit is used to synchronize the value of the first parameter to the second value.

14. A distributed system, comprising n groups of computing nodes, each group of computing nodes comprising m computing nodes, the m computing nodes in each group corresponding to the m parameter sets included in the target model, respectively, The m computing nodes in each group of computing nodes are used to jointly perform multiple trainings, so that each computing node in each group obtains the first value of each parameter in its corresponding parameter set; The n first computing nodes corresponding to the first parameter in the n groups are used to determine a second value corresponding to the first parameter based on the n first values ​​of the first parameter; The n first computing nodes are further configured to respectively synchronize the value of the first parameter to the second value.

15. A computing device comprising a memory and a processor, wherein the memory stores executable codes, and when the processor executes the executable codes, the method of claim 12 is implemented.

Images