Parallel training method and device of global model, equipment, medium and product

Abstract

The invention discloses a parallel training method and device of a global model, equipment, a medium and a product, and the method comprises the steps: carrying out the grouping of each calculation node, and obtaining a plurality of calculation groups; training the computing nodes in each computing group by adopting an asynchronous strategy so as to update the local model corresponding to each computing group; determining an outdated group in all the calculation groups, and adding the outdated group into an outdated group; when the number of outdated groups in the outdated group reaches a preset first threshold value, respectively calculating the training precision of each outdated group, calculating the training precision mean value of the outdated group, and recording the training precision mean value as a first training precision mean value; and updating the model parameters of the global model according to the local model parameters of the outdated group and other non-outdated groups with the training precision exceeding the first training precision mean value. By adopting the method, the parameter updating condition and updating means of the global model are improved, and the balance between the model training precision and the model updating interval is realized.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a method, device, equipment, medium and product for parallel training of a global model. Background Art

[0002] In recent years, deep learning has been widely used in various fields. The scale and complexity of models have continued to increase, resulting in a sharp increase in the time and computing resource requirements of the training process. In order to effectively meet this challenge, deep learning distributed training technology has emerged.

[0003] Common deep learning parallel techniques include data parallelism. The idea of ​​data parallelism is to split the training data into different fragments, and each computing node is responsible for training the gradient calculation of one or more data fragments. Data parallelism can be performed in synchronous or asynchronous mode. In synchronous training, each node needs to collect and aggregate the gradients of others and update the model weights in each iteration. In asynchronous training, nodes do not need to synchronize with each other. The parameter server is responsible for collecting gradients from computing nodes, updating global model weights, and sending new parameters to computing nodes.

[0004] However, the inventors found that the prior art has at least the following problems: In the real world, most clusters are naturally heterogeneous, consisting of different models of CPU (central processing unit) and GPU (graphics processing unit). Because in the data parallel asynchronous optimization scenario, nodes do not need to wait for the gradients of other nodes to update global parameters, so there is a large difference between the training progress calculated on the fastest node and the slowest node, resulting in the gradient calculated by the slowest node and the version conflict on the parameter server, affecting the convergence of the model. In the prior art, when the version difference between the fastest node and the slowest node reaches a certain threshold, the parameter update of the fastest node is stopped and global synchronization is performed. However, the problem with this solution is that the slowest node has not been updated in time due to its outdated parameters, which is likely to cause a decrease in training accuracy. Therefore, parameter fusion at this time will affect the existing excellent parameters and reduce the model training accuracy. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a method, apparatus, device, medium and product for parallel training of a global model, which completes the parameter update process of the global model by dynamically grouping computing nodes and screening model parameters, thereby effectively achieving a balance between model training accuracy and model update interval.

[0006] To achieve the above object, an embodiment of the present invention provides a parallel training method for a global model, comprising:

[0007] Group each computing node into several computing groups;

[0008] Using an asynchronous strategy to train the computing nodes in each computing group to update the local model corresponding to each computing group;

[0009] determining a lagging group among all the calculated groups and adding the group to the lagging group;

[0010] When the number of lagging groups in the lagging group reaches a preset first threshold, respectively calculating the training accuracy of each of the lagging groups, and calculating the mean of the training accuracy of the lagging groups, which is recorded as a first training accuracy mean;

[0011] The model parameters of the global model are updated according to the local model parameters of the lagging group and other non-lagging groups whose training accuracy exceeds the first training accuracy mean.

[0012] As an improvement of the above solution, the updating of the model parameters of the global model according to the local model parameters of the lagging group and other non-lagging groups whose training accuracy exceeds the first training accuracy mean includes:

[0013] Acquire the lagging group and other non-lagging groups whose training accuracy exceeds the first training accuracy mean as target groups participating in the global model parameter update;

[0014] Obtaining the local model parameters of the fastest group in the target group to obtain a first model parameter group; wherein the fastest group is a computing group with the largest local model version;

[0015] Performing an average fusion process on the local model parameters of other calculation groups in the target group that are not the fastest group, to obtain a second model parameter group;

[0016] The model parameters of the global model are updated according to the first model parameter group, the second model parameter group and corresponding weight values.

[0017] As an improvement of the above solution, updating the model parameters of the global model according to the first model parameter group, the second model parameter group and the corresponding weight values ​​includes:

[0018] Calculating the training accuracy of each computing group in the target group;

[0019] Calculating a second training accuracy mean according to the training accuracy of other computing groups in the target group that are not the fastest group;

[0020] According to the training accuracy of the fastest group and the second training accuracy mean, the weight value of the first model parameter group is calculated using the following calculation formula:

[0021]

[0022] According to the first model parameter group, the second model parameter group and the weight value, the updated model parameters of the global model are calculated using the following calculation formula:

[0023] Z i =λx i +(1-λ)y i ;

[0024] Among them, z i is the updated model parameter of the global model, x i is the model parameter in the first model parameter group, y i is the model parameter in the second model parameter group, λ is the weight value of the first model parameter group, ACC F is the training accuracy of the fastest group, ACC D is the second training accuracy mean.

[0025] As an improvement of the above solution, determining a lagging group among all the computing groups and adding the lagging group to the lagging group includes:

[0026] When the difference between the local model version of any of the computing groups and the local model version of the fastest group reaches a preset second threshold, the computing group is added to the lagging group as a lagging group; wherein the local model version is increased by one after each local model update; and the fastest group is the computing group with the largest local model version.

[0027] As an improvement of the above solution, the method further includes:

[0028] When the number of lagging groups in the lagging group does not reach the first threshold, and the difference between the local model versions of the slowest group and the fastest group reaches a preset third threshold,

[0029] If the number of lagging groups in the lagging group is equal to one, updating the model parameters of the global model according to the local model parameters of all non-lagging groups;

[0030] If the number of lagging groups in the lagging group is greater than one, updating the model parameters of the global model according to the local model parameters of the lagging groups whose training accuracy exceeds the first training accuracy mean and other non-lagging groups;

[0031] The slowest group is the calculation group with the smallest local model version, and the third threshold is greater than the second threshold.

[0032] As an improvement of the above solution, the method further includes:

[0033] When the difference between the local model version of any of the computing groups and the global model version of the global model reaches a preset third threshold, the model parameters of the global model are updated according to the local model parameters of all the computing groups; wherein the local model version is incremented by one after each local model update, and the global model version is incremented by one after each global model update.

[0034] As an improvement of the above solution, after updating the model parameters of the global model, the method further includes:

[0035] Distributing the updated model parameters of the global model to the computing nodes in the computing group that did not participate in the global model parameter update, as the latest local model parameters for training;

[0036] This cycle is iterated until the global model converges.

[0037] As an improvement of the above solution, the computing nodes are grouped to obtain several computing groups, including:

[0038] Obtain performance data of each computing node in real time;

[0039] Each of the computing nodes is divided into a plurality of computing groups according to the performance data, so that the sum of variances of the performance data of each of the computing groups is minimized.

[0040] The present invention also provides a parallel training device for a global model, comprising:

[0041] A computing group division module is used to group each computing node into several computing groups;

[0042] A local model updating module, used for training the computing nodes in each computing group using an asynchronous strategy to update the local model corresponding to each computing group;

[0043] A lagging group building module, used for determining a lagging group among all the computing groups and adding the lagging group to the lagging group;

[0044] A training accuracy calculation module, used for calculating the training accuracy of each lagging group respectively when the number of lagging groups in the lagging group reaches a preset first threshold, and calculating the training accuracy mean of the lagging group, which is recorded as a first training accuracy mean;

[0045] The global model updating module is used to update the model parameters of the global model according to the local model parameters of the lagging group and other non-lagging groups whose training accuracy exceeds the first training accuracy mean.

[0046] The present invention also provides a parallel training device for a global model, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements the parallel training method for the global model as described in any one of the above items.

[0047] The present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the parallel training method of the global model as described in any one of the above items.

[0048] The present invention also provides a computer program product, which includes a computer program or computer instructions, and when the computer program or the computer instructions are executed by a processor, the parallel training method of the global model as described in any one of the above items is implemented.

[0049] Compared with the prior art, the parallel training method, apparatus, equipment, medium and product of the global model disclosed in the present invention, in the asynchronous training scenario of the global model, dynamically groups the computing nodes, determines the parameter update conditions of the global model according to the training speed and training accuracy of each group, and screens the local model parameters to complete the parameter update of the global model, solves the problem of slowing down the model convergence process due to large accuracy differences during training, effectively achieves a balance between model training accuracy and model update interval, and improves the training accuracy of the global model. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flowchart of a parallel training method for a global model provided by an embodiment of the present invention;

[0051] Figure 2 It is a schematic diagram of a trigger condition for updating global model parameters in an embodiment of the present invention;

[0052] Figure 3 is a schematic diagram of another triggering condition for updating global model parameters in an embodiment of the present invention;

[0053] Figure 4 It is a structural schematic diagram of a parallel training device for a global model provided by an embodiment of the present invention;

[0054] Figure 5 It is a structural schematic diagram of a parallel training device for a global model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] In the description of the present application, it should be understood that the terms "center", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0057] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0058] See also Figure 1 , is a flow chart of a parallel training method for a global model provided by an embodiment of the present invention. The embodiment of the present invention provides a parallel training method for a global model, and the method includes steps S11 to S15:

[0059] S11, grouping each computing node to obtain a number of computing groups;

[0060] S12, adopting an asynchronous strategy to train the computing nodes in each of the computing groups to update the local model corresponding to each of the computing groups;

[0061] S13, determining a lagging group among all the calculation groups, and adding the group to the lagging group;

[0062] S14, when the number of lagging groups in the lagging group reaches a preset first threshold, respectively calculating the training accuracy of each of the lagging groups, and calculating the mean of the training accuracy of the lagging groups, which is recorded as a first training accuracy mean;

[0063] S15. Update the model parameters of the global model according to the local model parameters of the lagging group and other non-lagging groups whose training accuracy exceeds the first training accuracy mean.

[0064] It should be noted that the embodiment of the present invention is applied to the scenario of data parallel asynchronous training of the global model, by setting multiple computing nodes to train part of the training data respectively, and adopting the asynchronous training mode, the computing nodes do not need to be synchronized with each other. The computing nodes are training machines, such as heterogeneous distributed GPU servers, etc.

[0065] In the embodiment of the present invention, each of the computing nodes is grouped to obtain a plurality of computing groups, each of the computing groups includes at least one computing node, and the grouping result is sent to all computing nodes.

[0066] As an optional implementation, step S11, that is, grouping the computing nodes to obtain a plurality of computing groups, includes:

[0067] S111. Obtain performance data of each computing node in real time;

[0068] S112: Cluster computing nodes with similar performance using a natural breakpoint clustering algorithm based on the performance data, so as to divide each computing node into a plurality of computing groups.

[0069] Specifically, the performance data of each computing node is collected in real time for performance evaluation. This process is performed after the first epoch (iterative round) of training. No grouping strategy is used in this stage, aiming to obtain the most original performance data. The performance data includes the training time.

[0070] Computing nodes are grouped based on performance data, using a natural breakpoint clustering algorithm for grouping, and the grouping results are sent to each computing node. Different groups are distinguished using different keys, where the key is an integer variable. The groups are denoted as key_0, key_1, ..., key_n, where n is the number of computing groups. Each computing group maintains a local model and a local parameter server, and uses a parameter server architecture for gradient communication. The local parameter server is used to aggregate the training results of each computing node in the computing group, maintain the final training parameters of the local model, and update the local model. The local model version number is denoted as k, which is initialized to 0. After each local model update, the k value increases by 1.

[0071] More preferably, the variance is used to measure the performance similarity of the computing nodes, that is, the computing nodes are divided into a plurality of computing groups according to the performance data, so that the sum of the variances of the performance data of each computing group is minimized.

[0072] Specifically, nodes with similar performance are placed together, all computing nodes in the cluster are divided into several groups, and variance is used to measure similarity. The variance of each computing group is calculated, and then the sum of the variances of these computing groups is calculated. The size of the sum of the variances is used to compare the quality of the grouping results. When the sum of the variances is the smallest, the corresponding grouping result is the optimal grouping result, and the optimal grouping result is obtained.

[0073] Based on the optimal grouping results, the computing nodes in each computing group use an asynchronous strategy to independently update the local models between groups. In addition, during the training process, a global parameter server is also maintained for the final global model to maintain the final training parameters of the global model. The global model version number is denoted as p, which is initialized to 0. After each global model update, the p value increases by 1.

[0074] Furthermore, a rule for determining the lagging group is preset to determine the lagging group from among all the computing groups during the model training process to form a lagging group.

[0075] As a preferred implementation, step S13, i.e., determining a lagging group among all the computing groups and adding the lagging group to the lagging group, includes:

[0076] S131. When the difference between the local model version of any of the computing groups and the local model version of the fastest group reaches a preset second threshold value θ, the computing group is added to the lagging group as a lagging group.

[0077] Specifically, a second threshold value θ is pre-set to characterize the difference between the local model versions of each computing group in the cluster and the fastest group. The fastest group is the computing group with the largest local model version. When the difference between the local model version of any computing group and the local model version of the fastest group reaches the second threshold value θ, it is considered that the training process of the computing group is slow, and it is added to the lagging group as a lagging group.

[0078] The number of lagging groups contained in the lagging group is counted in real time. When the number of lagging groups in the lagging group reaches a preset first threshold γ, the local parameter update process of each calculation group is stopped, and a parameter update condition of the global model is triggered.

[0079] See also Figure 2 , is a schematic diagram of the principle of a trigger condition for updating the global model parameters in an embodiment of the present invention. Taking 6 computing nodes forming 3 computing groups as an example, the first threshold γ is set to 2, and the second threshold θ is set to 4. The changes of each group and the parameter update mechanism of the dynamic grouping during the calculation iteration are introduced.

[0080] Among them, the 6 computing nodes are workers 0 to 5, and the 3 computing groups are recorded as key_0, key_1 and key_2. It can be seen that when the local model version number of computing group key_2 is 8, the local model version of computing group key_1 is 4, and the version number of computing group key_0 is 6, computing group key_2 is the fastest group. At this time, the difference between the version number of computing group key_1 and the version number of computing group key_2 reaches the second threshold value θ, then computing group key_1 is added to the lagging group, and the lagging value γ'=1. When the local model version number of computing group key_2 is 17 and the local model version number of computing group key_0 is 12, the difference between the two versions is greater than the second threshold value θ, computing group key_0 is also added to the lagging group, and the lagging value γ'=2, reaching the first threshold γ, and a global model parameter update is triggered.

[0081] When the parameter update condition of the global model is triggered, it is necessary to determine the computing nodes that participate in the parameter update of the global model. Calculate the training accuracy of each of the lagging groups separately, and calculate the mean training accuracy of the lagging group based on the training accuracy of each of the lagging groups, which is recorded as the first training accuracy mean. Obtain the lagging group whose training accuracy exceeds the first training accuracy mean. Only the lagging groups whose training accuracy exceeds the first training accuracy mean and other non-lagging groups in the world can participate in the parameter update of the global model. Other lagging groups whose training accuracy does not exceed the first training accuracy mean will not participate in the parameter update of the global model, so as to prevent the lagging groups with poor local model parameters from slowing down the progress of model training in the entire cluster.

[0082] Furthermore, the model parameters of the global model are updated according to the local model parameters of the lagging group and other non-lagging groups whose training accuracy exceeds the first training accuracy mean.

[0083] By adopting the technical means of the embodiments of the present invention, in the asynchronous training scenario of the global model, the computing nodes are dynamically grouped, and the parameter update conditions of the global model are determined according to the training speed and training accuracy of each group, and the local model parameters are screened to complete the parameter update of the global model. This solves the problem of slowing down the model convergence process due to large accuracy differences during training, effectively achieves a balance between model training accuracy and model update interval, and improves the training accuracy of the global model.

[0084] As a preferred implementation, step S15, that is, updating the model parameters of the global model according to the local model parameters of the lagging group and other non-lagging groups whose training accuracy exceeds the first training accuracy mean, includes steps S151 to S154:

[0085] S151, obtaining the lagging group whose training accuracy exceeds the first training accuracy mean and other non-lagging groups as target groups participating in the global model parameter update;

[0086] S152, obtaining the local model parameters of the fastest group in the target group to obtain a first model parameter group; wherein the fastest group is a calculation group with the largest local model version;

[0087] S153, performing an average fusion process on the local model parameters of other calculation groups in the target group other than the fastest group to obtain a second model parameter group;

[0088] S154. Update the model parameters of the global model according to the first model parameter group, the second model parameter group and corresponding weight values.

[0089] The embodiment of the present invention proposes a model parameter update rule for a global model, where a computing group that can participate in the global model parameter update is recorded as a target group, and the local model parameters of the fastest group with the largest local model version number in the target group are recorded as the first model parameter group A={x 1 , x 2 , ..., x n}, where x i represents the local model parameter value of the fastest group, and n represents the number of model parameters. The local model parameters of the other calculation groups in the target group except the fastest group are averaged and fused to obtain the second model parameter group B={y 1 ,y 2 , ..., y n}, where y i represents the average fused model parameter value, and n represents the number of model parameters.

[0090] Optionally, the average fusion process is: taking the average of the same local model parameters of all the calculation groups to form fused model parameters.

[0091] Preferably, step S154, that is, updating the model parameters of the global model according to the first model parameter group, the second model parameter group and the corresponding weight values, includes:

[0092] Calculating the training accuracy of each computing group in the target group;

[0093] Calculate the second training accuracy mean ACC according to the training accuracy of other computing groups in the target group other than the fastest group D ;

[0094] According to the training accuracy of the fastest group ACCF And the second training accuracy mean ACC D , the weight value λ of the first model parameter group is calculated using the following calculation formula:

[0095]

[0096] According to the first model parameter group ACC F The second model parameter group ACC D and the weight value λ, the updated model parameters of the global model are calculated using the following calculation formula:

[0097] z i =λx i +(1-λ)y i ;

[0098] Among them, z i is the updated model parameter of the global model, x i is the model parameter in the first model parameter group, y i is the model parameter in the second model parameter group, λ is the weight value of the first model parameter group, ACC F is the training accuracy of the fastest group, ACC D is the second training accuracy mean.

[0099] The updated model parameter set of the global model is thus obtained as C={z 1 , z 2 , ..., z n}.

[0100] By adopting the technical means of the embodiments of the present invention, in the process of updating the global model parameters, the local model parameters of the computing groups with poor training speed and training accuracy are discarded, and the global model parameters are updated according to the local model parameters of the computing groups with better training speed and training accuracy, and the weights of the local model parameters of the fastest group are given priority, thereby effectively improving the accuracy of updating the global model parameters.

[0101] As a preferred implementation, the embodiment of the present invention is further implemented on the basis of the above embodiment, and the method further includes steps S21 to S23:

[0102] S21, when the number of lagging groups in the lagging group does not reach the first threshold, and the difference between the local model versions of the slowest group and the fastest group reaches a preset third threshold,

[0103] S22, if the number of lagging groups in the lagging group is equal to one, updating the model parameters of the global model according to the local model parameters of all non-lagging groups;

[0104] S23. If the number of lagging groups in the lagging group is greater than one, the model parameters of the global model are updated according to the local model parameters of the lagging groups whose training accuracy exceeds the first training accuracy mean and other non-lagging groups.

[0105] In an embodiment of the present invention, another parameter update condition for the global model is added. During the model training process, if the number of lagging groups in the lagging group reaches the second threshold θ, but the difference in local model versions between the slowest group and the fastest group has reached the third threshold α, the slowest group is the calculation group with the smallest local model version, and the third threshold α is greater than the second threshold θ, then the parameter update condition for the global model is also triggered at this time.

[0106] See also Figure 3 , is a schematic diagram of the principle of another triggering condition for updating the global model parameters in an embodiment of the present invention. Taking 6 computing nodes forming 3 computing groups as an example, the first threshold γ is set to 2, the second threshold θ is set to 4, and the third threshold α is set to 7. The changes of each group and the parameter update mechanism of the dynamic grouping during the calculation iteration are introduced.

[0107] Among them, the 6 computing nodes are workers 0 to 5, and the 3 computing groups are key_0, key_1, and key_2. It can be seen that when the local model version number of computing group key_2 is 8, the local model version of computing group key_1 is 4, and the version number of computing group key_0 is 6, computing group key_2 is the fastest group and computing group key_1 is the slowest group. At this time, the difference between the version number of computing group key_1 and the version number of computing group key_2 reaches the second threshold θ, then computing group key_1 is added to the lagging group, and the lagging value γ'=1. When the local model version number of computing group key_2 is 13 and the local model version number of computing group key_0 is 6, computing group key_2 is the fastest group and computing group key_0 is the slowest group. At this time, the difference between the version number of computing group key_0 and the version number of computing group key_2 reaches the third threshold α. Even if the lagging value does not reach the first threshold γ, it still triggers a parameter update of the global model.

[0108] Furthermore, when the parameter update condition of the global model is triggered, it is necessary to determine the computing nodes that participate in the parameter update of the global model. There are two cases for discussion:

[0109] If there is only one lagging group in the lagging group at this time, the local model parameters of the lagging group are discarded, that is, it does not participate in the updating process of the global model parameters. The model parameters of the global model are updated according to the local model parameters of all non-lagging groups.

[0110] If the number of lagging groups in the lagging group is greater than one, the training accuracy of each lagging group is also calculated, and the first training accuracy mean of the lagging group is calculated according to the training accuracy of each lagging group. The model parameters of the global model are updated according to the local model parameters of the lagging group whose training accuracy exceeds the first training accuracy mean and other non-lagging groups.

[0111] It should be noted that the parameter updating rules of the global model can refer to the above embodiment and will not be elaborated here.

[0112] As a preferred implementation, the embodiment of the present invention is further implemented on the basis of the above embodiment, and the method further includes step S24:

[0113] S24. When the difference between the local model version of any of the computing groups and the global model version of the global model reaches a preset third threshold, the model parameters of the global model are updated according to the local model parameters of all the computing groups.

[0114] In an embodiment of the present invention, another parameter update condition of the global model is added. During the model training process, if the difference between the local model version of the calculation group and the global model version of the global model reaches a preset third threshold α, the parameter update condition of the global model is also triggered.

[0115] It should be noted that the parameter updating rules of the global model can refer to the above embodiment and will not be elaborated here.

[0116] By adopting the technical means of the embodiments of the present invention, the parameter update conditions for the global model are set by taking into account various situations such as the local model version difference between the calculation group and the fastest group, the local model version difference between the slowest group and the fastest group, and the difference between the local model version and the global model version, so as to effectively achieve a balance between the model training accuracy and the model update interval, and improve the training accuracy of the global model.

[0117] As a preferred implementation manner, the embodiment of the present invention is further implemented on the basis of any of the above embodiments. After the model parameters of the global model are updated, the method further includes:

[0118] Distributing the updated model parameters of the global model to the computing nodes in the computing group that did not participate in the global model parameter update, as the latest local model parameters for training;

[0119] This cycle is iterated until the global model converges.

[0120] Specifically, after the model parameters of the global model are updated once, the lagging group that did not participate in the global model parameter update in this round will pull a copy of the latest model parameters from the global parameter server for training. This cycle is repeated until the global model converges, the model parameters of the final global model are obtained, the global model training is completed, and the trained global model is obtained.

[0121] See also Figure 4 , is a schematic diagram of the structure of a parallel training device for a global model provided by an embodiment of the present invention. The embodiment of the present invention also provides a parallel training device 30 for a global model, including:

[0122] A computing group division module 31 is used to group each computing node to obtain a plurality of computing groups;

[0123] A local model updating module 32, used for training the computing nodes in each computing group using an asynchronous strategy to update the local model corresponding to each computing group;

[0124] A lagging group building module 33, used to determine a lagging group among all the computing groups and add the lagging group to the lagging group;

[0125] A training accuracy calculation module 34 is used to calculate the training accuracy of each lagging group when the number of lagging groups in the lagging group reaches a preset first threshold, and calculate the training accuracy mean of the lagging group, which is recorded as a first training accuracy mean;

[0126] The global model updating module 35 is used to update the model parameters of the global model according to the local model parameters of the lagging group and other non-lagging groups whose training accuracy exceeds the first training accuracy mean.

[0127] As a preferred implementation, the global model updating module 35 is specifically used for:

[0128] Acquire the lagging group and other non-lagging groups whose training accuracy exceeds the first training accuracy mean as target groups participating in the global model parameter update;

[0129] Obtaining the local model parameters of the fastest group in the target group to obtain a first model parameter group; wherein the fastest group is a computing group with the largest local model version;

[0130] Performing an average fusion process on the local model parameters of other calculation groups in the target group that are not the fastest group, to obtain a second model parameter group;

[0131] The model parameters of the global model are updated according to the first model parameter group, the second model parameter group and corresponding weight values.

[0132] Preferably, updating the model parameters of the global model according to the first model parameter group, the second model parameter group and the corresponding weight values ​​comprises:

[0133] Calculating the training accuracy of each computing group in the target group;

[0134] Calculating a second training accuracy mean according to the training accuracy of other computing groups in the target group that are not the fastest group;

[0135] According to the training accuracy of the fastest group and the second training accuracy mean, the weight value of the first model parameter group is calculated using the following calculation formula:

[0136]

[0137] According to the first model parameter group, the second model parameter group and the weight value, the updated model parameters of the global model are calculated using the following calculation formula:

[0138] z i =λx i +(1-λ)y i ;

[0139] Among them, z i is the updated model parameter of the global model, x i is the model parameter in the first model parameter group, y i is the model parameter in the second model parameter group, λ is the weight value of the first model parameter group, ACC F is the training accuracy of the fastest group, ACC D is the second training accuracy mean.

[0140] As a preferred implementation, the lagging group building module 33 is specifically used for:

[0141] When the difference between the local model version of any of the computing groups and the local model version of the fastest group reaches a preset second threshold, the computing group is added to the lagging group as a lagging group; wherein the local model version is increased by one after each local model update; and the fastest group is the computing group with the largest local model version.

[0142] It should be noted that a parallel training device for a global model provided in an embodiment of the present invention is used to execute all process steps of a parallel training method for a global model in the above embodiment. The working principles and beneficial effects of the two correspond one to one, and thus will not be elaborated upon.

[0143] An embodiment of the present invention also provides a parallel training device 40 for a global model, comprising a processor 41, a memory 42, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the parallel training method for the global model as described in any one of the above embodiments is implemented.

[0144] It should be noted that the parallel training device apparatus for a global model provided in an embodiment of the present invention is used to execute all the process steps of a parallel training method for a global model in the above-mentioned embodiment. The working principles and beneficial effects of the two correspond one to one, and thus will not be elaborated upon.

[0145] An embodiment of the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the parallel training method of the global model as described in any one of the above embodiments.

[0146] An embodiment of the present invention also provides a computer program product, which includes a computer program or computer instructions. When the computer program or the computer instructions are executed by a processor, the parallel training method of the global model as described in any one of the above embodiments is implemented.

[0147] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0148] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A parallel training method for a global model, characterized in that: include: Group each computing node into several computing groups; Using an asynchronous strategy to train the computing nodes in each computing group to update the local model corresponding to each computing group; determining a lagging group among all the calculated groups and adding the group to the lagging group; When the number of lagging groups in the lagging group reaches a preset first threshold, respectively calculating the training accuracy of each of the lagging groups, and calculating the mean of the training accuracy of the lagging groups, which is recorded as a first training accuracy mean; The model parameters of the global model are updated according to the local model parameters of the lagging group and other non-lagging groups whose training accuracy exceeds the first training accuracy mean.

2. The parallel training method of the global model according to claim 1, characterized in that: The updating of the model parameters of the global model according to the local model parameters of the lagging group and other non-lagging groups whose training accuracy exceeds the first training accuracy mean includes: Acquire the lagging group and other non-lagging groups whose training accuracy exceeds the first training accuracy mean as target groups participating in the global model parameter update; Obtaining the local model parameters of the fastest group in the target group to obtain a first model parameter group; wherein the fastest group is a computing group with the largest local model version; Performing an average fusion process on the local model parameters of other calculation groups in the target group that are not the fastest group, to obtain a second model parameter group; The model parameters of the global model are updated according to the first model parameter group, the second model parameter group and corresponding weight values.

3. The parallel training method of the global model as claimed in claim 2, characterized in that: The updating of the model parameters of the global model according to the first model parameter group, the second model parameter group and the corresponding weight values ​​comprises: Calculating the training accuracy of each computing group in the target group; Calculating a second training accuracy mean according to the training accuracy of other computing groups in the target group that are not the fastest group; Calculating a weight value of the first model parameter group according to the training accuracy of the fastest group and the second training accuracy mean; The updated model parameters of the global model are calculated according to the first model parameter group, the second model parameter group and the weight values.

4. The parallel training method of the global model according to claim 1, characterized in that: The step of determining a lagging group among all the computing groups and adding the lagging group to the lagging group includes: When the difference between the local model version of any of the computing groups and the local model version of the fastest group reaches a preset second threshold, the computing group is added to the lagging group as a lagging group; wherein the local model version is increased by one after each local model update; and the fastest group is the computing group with the largest local model version.

5. The parallel training method of the global model according to claim 4, characterized in that: The method further comprises: When the number of lagging groups in the lagging group does not reach the first threshold, and the difference between the local model versions of the slowest group and the fastest group reaches a preset third threshold, If the number of lagging groups in the lagging group is equal to one, updating the model parameters of the global model according to the local model parameters of all non-lagging groups; If the number of lagging groups in the lagging group is greater than one, updating the model parameters of the global model according to the local model parameters of the lagging groups whose training accuracy exceeds the first training accuracy mean and other non-lagging groups; The slowest group is the calculation group with the smallest local model version, and the third threshold is greater than the second threshold.

6. The parallel training method of the global model according to claim 1, characterized in that: The method further comprises: When the difference between the local model version of any of the computing groups and the global model version of the global model reaches a preset third threshold, the model parameters of the global model are updated according to the local model parameters of all the computing groups; wherein the local model version is incremented by one after each local model update, and the global model version is incremented by one after each global model update.

7. The parallel training method of the global model according to any one of claims 1 to 6, characterized in that: After updating the model parameters of the global model, the method further includes: Distributing the updated model parameters of the global model to the computing nodes in the computing group that did not participate in the global model parameter update, as the latest local model parameters for training; This cycle is iterated until the global model converges.

8. The parallel training method of the global model according to claim 1, characterized in that: The computing nodes are grouped to obtain several computing groups, including: Obtain performance data of each computing node in real time; Each of the computing nodes is divided into a plurality of computing groups according to the performance data, so that the sum of variances of the performance data of each of the computing groups is minimized.

9. A parallel training device for a global model, characterized in that: include: A computing group division module is used to group each computing node into several computing groups; A local model updating module, used for training the computing nodes in each computing group using an asynchronous strategy to update the local model corresponding to each computing group; A lagging group building module, used for determining a lagging group among all the computing groups and adding the lagging group to the lagging group; A training accuracy calculation module, used for calculating the training accuracy of each lagging group respectively when the number of lagging groups in the lagging group reaches a preset first threshold, and calculating the training accuracy mean of the lagging group, which is recorded as a first training accuracy mean; The global model updating module is used to update the model parameters of the global model according to the local model parameters of the lagging group and other non-lagging groups whose training accuracy exceeds the first training accuracy mean.

10. A parallel training device for a global model, characterized in that: It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, it implements the parallel training method of the global model as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the parallel training method of the global model as described in any one of claims 1 to 8.

12. A computer program product, characterized in that The computer program product comprises a computer program or computer instructions, and when the computer program or the computer instructions are executed by a processor, the parallel training method of the global model as described in any one of claims 1 to 8 is implemented.

Images