Method for distributed training of neural network model and computing node

Abstract

Embodiments of the present application provide a method for distributed training of a neural network model in a computing device cluster, a computing node, and a computing device cluster. Computing units of the computing node can obtain weight coefficients of all network layers by means of intra-node communication prior to forward computation and backward computation, without the need for inter-node communication, thereby reducing the number of global communications, and improving model training efficiency. The method comprises: sequentially performing global collection on weight coefficients of network layers in computing units of a node; performing forward computation on input data of the network layers on the basis of a global collection result; upon obtaining a prediction result by means of the forward computation, sequentially performing global collection on the weight coefficients of the network layers in the computing units of the node; performing backward computation on input gradient data of the network layers on the basis of the global collection result; and updating the weight coefficients of the computing units on the basis of a gradient obtained by means of the backward computation, and iteratively executing the described steps until training is finished.

Description

Method and computing node for distributed training of neural network model

[0001] The present application claims priority to the Chinese patent application No. 202410545003.7, filed on April 30, 2024, and entitled "Method and computing node for distributed training of neural network model", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, and in particular to a method and a computing node for distributed training of neural network model. BACKGROUND

[0003] With the development of artificial intelligence technology, the neural network model and the training data volume are increasing, and the single card training mode has been unable to meet the training needs of super large models. In order to improve the training efficiency of the neural network model, a distributed parallel training mode can be used.

[0004] Referring to FIG. 1, a method for distributed training of a neural network model is roughly as follows: taking four computing nodes in a computing device cluster as an example, each computing node includes two accelerator cards, the weight coefficients (i.e., full weight coefficients) of the neural network model are divided into eight parts, and each accelerator card stores one part. After dividing the training data into eight parts, each accelerator card collects the weight coefficients of all accelerator cards globally, performs forward calculation on the training data subset according to the full weight coefficients collected globally, collects the weight coefficients of all accelerator cards globally again, performs backward calculation according to the full weight coefficients collected globally and the forward calculation result, and then performs reduction scattering on the backward calculation results of all accelerator cards. The weight coefficients of each accelerator card are updated according to the gradient obtained by the reduction scattering, and the above training steps are iteratively executed until the training is completed.

[0005] During the global collection, each computing node needs to obtain the weight coefficients of other computing nodes through a low-bandwidth link, which has the problem of slow communication speed, which leads to slow training model speed. SUMMARY

[0006] The present application provides a method for distributed training of a neural network model, each computing node stores weight coefficients of the neural network model, and the weight coefficients of the neural network model can be obtained through intra-node communication before forward calculation and backward calculation, so that the weight coefficients of the neural network model do not need to be obtained through cross-node communication, thereby reducing the number of global communications and improving the training model efficiency. The present application also provides a computing node, a computing device cluster, a computer readable storage medium and a computer program product capable of implementing the above method.

[0007] The first aspect provides a method for distributed training of a neural network model in a computing device cluster, comprising: sequentially collecting, by a computing unit of a computing node, weight coefficients of a network layer in all computing units of the computing node, and performing forward calculation on input data of the network layer according to the globally collected weight coefficients of the network layer; after obtaining a prediction result through the forward calculation, sequentially collecting, by the computing unit of the computing node, the weight coefficients of the network layer in all computing units of the computing node, performing backward calculation on input gradient data of the network layer according to the globally collected weight coefficients of the network layer, and then updating the weight coefficients of the computing unit according to the gradient obtained through the backward calculation, and iteratively performing the above steps until the training is completed. The computing device cluster comprises M computing nodes, each of which comprises K computing units, and M and K are positive integers. Each computing node stores weight coefficients of a neural network model, and the weight coefficients of each network layer in the neural network model are uniformly distributed in the K computing units of the computing node. The communication bandwidth between the computing units in the computing node is greater than the communication bandwidth between the computing nodes. The bus between the computing units can include but is not limited to an NVlink bus, and the communication link between the computing nodes can include but is not limited to a peripheral component interconnect express (PCIE) bus.

[0008] According to the implementation, each computing unit of the computing node can obtain the weight coefficients of each network layer through intra-node communication, and then perform forward calculation and backward calculation according to the weight coefficients of the network layer. Therefore, no cross-node communication is required before the forward calculation and the backward calculation. Since the number of global communications is reduced, the forward calculation speed and the backward calculation speed can be improved, thereby improving the training model efficiency.

[0009] In combination with the first aspect, in a first possible implementation, after the computing unit of the computing node performs forward calculation on the input data of the network layer according to the globally collected weight coefficients of the network layer, the computing unit of the computing node releases the weight coefficients from other computing units in the globally collected weight coefficients of the network layer; and after the computing unit of the computing node performs backward calculation on the input gradient data of the network layer according to the globally collected weight coefficients of the network layer, the computing unit of the computing node releases the weight coefficients from other computing units in the globally collected weight coefficients of the network layer. In this way, the storage overhead of the weight coefficients can be reduced, so that the method for distributed training of a neural network model according to the present application can also be performed by a computing unit with small storage capacity.

[0010] In a second possible implementation manner of the first aspect or the first possible implementation manner, the computing node updates the weight coefficient of the computing unit according to the gradient result obtained by the back calculation comprises: the computing node performs reduction scattering on the gradient obtained by the back calculation of all computing units of the computing node to obtain a reduction scattering result of the computing node, the computing unit of the computing node acquires the reduction scattering result of the computing unit in the reduction scattering result of the computing node, performs global reduction on the reduction scattering result of the computing unit group to which the computing unit belongs, and updates the weight coefficient of the computing unit according to the gradient obtained by the global reduction. The computing unit group comprises computing units corresponding to the computing unit in all computing nodes. Optionally, the computing units corresponding to the computing unit in all computing nodes have the same serial number.

[0011] According to the implementation, the global reduction is performed on the reduction scattering result of the computing unit group to which the computing unit belongs, that is, the cross-node communication is performed once, so that the sum of the gradients of multiple computing units in different computing nodes can be acquired, and the average gradient can be calculated according to the gradient sum. The weight coefficient updated based on the average gradient can more accurately fit the data than the weight coefficient updated based on the gradient of a single computing node.

[0012] In a third possible implementation manner of the second possible implementation manner of the first aspect, the computing unit of the computing node updates the weight coefficient of the computing unit according to the gradient obtained by the global reduction comprises: the computing unit of the computing node determines a first-order matrix estimation value and a second-order matrix estimation value according to the gradient obtained by the global reduction; and the weight coefficient of the computing unit is updated according to the first-order matrix estimation value and the second-order matrix estimation value. The adaptive learning rate can be calculated according to the first-order matrix estimation value and the second-order matrix estimation value, the gradient direction can be more accurately adjusted according to the adaptive learning rate, the number of times of updating the weight coefficient can be reduced, and the training speed can be improved.

[0013] In another possible implementation manner of the second possible implementation manner of the first aspect, the weight coefficient of the computing unit and the gradient obtained by the global reduction satisfy the following formula: θ t =θ t-1 -α*g t . θ t is the weight coefficient after the tth update, θ t-1 is the weight coefficient after the (t-1)th update, which can be regarded as the weight coefficient before the tth update, α is a learning rate, g t is the gradient obtained by the global reduction in the tth cycle. In this way, another method for updating the weight coefficient is provided, and the flexibility of implementation of the scheme is improved.

[0014] In a possible implementation manner of the first aspect or the first implementation manner of the first aspect, in a possible implementation manner, the computing unit of the computing node updates the weight coefficient of the computing unit according to the gradient obtained by the back calculation, including: the computing unit of the computing node obtains the reduction scattering result of the computing unit in the global reduction scattering result, and determines the reference gradient of the computing unit according to the reduction scattering result of the computing unit; the reference gradients of the computing unit group to which the computing unit belongs are collected globally; and the weight coefficient of the computing unit is updated according to the globally collected reference gradients. The computing unit group includes the computing units corresponding to the computing unit in all computing nodes. The global reduction scattering result is obtained by reducing and scattering the gradient obtained by the back calculation of all computing nodes, so that the optimizer state (such as the gradient) can be cut and stored, and the storage overhead of the optimizer state can be reduced. In this way, the gradient of the other computing nodes is obtained through 2 times of cross-node communication, which is less than the number of times of cross-node communication in the prior art, and therefore the speed of training the model can be improved.

[0015] In a possible implementation manner of the first aspect or the first implementation manner of the first aspect, in a possible implementation manner, the computing unit of the computing node updates the weight coefficient of the computing unit according to the gradient obtained by the back calculation, including: the computing unit of the computing node obtains the reduction scattering result of the computing unit in the global reduction scattering result, and determines the reference gradient of the computing unit according to the reduction scattering result of the computing unit; the reference gradients of the computing unit group to which the computing unit belongs are collected globally; and the weight coefficient of the computing unit is updated according to the globally collected reference gradients. The computing unit group includes the computing units corresponding to the computing unit in all computing nodes. The global reduction scattering result is obtained by reducing and scattering the gradient obtained by the back calculation of all computing nodes, so that the optimizer state (such as the gradient) can be cut and stored, and the storage overhead of the optimizer state can be reduced. In this way, the gradient of the other computing nodes is obtained through 2 times of cross-node communication, which is less than the number of times of cross-node communication in the prior art, and therefore the speed of training the model can be improved.

[0016] In a possible implementation manner of the first aspect or the first implementation manner of the first aspect, in a possible implementation manner, the computing unit of the computing node updates the weight coefficient of the computing unit according to the gradient obtained by the back calculation, including: the computing unit of the computing node obtains the reduction scattering result of the computing unit in the global reduction scattering result, and determines the reference gradient of the computing unit according to the reduction scattering result of the computing unit; the reference gradients of the computing unit group to which the computing unit belongs are collected globally; and the weight coefficient of the computing unit is updated according to the globally collected reference gradients. The computing unit group includes the computing units corresponding to the computing unit in all computing nodes. The global reduction scattering result is obtained by reducing and scattering the gradient obtained by the back calculation of all computing nodes, so that the optimizer state (such as the gradient) can be cut and stored, and the storage overhead of the optimizer state can be reduced. In this way, the gradient of the other computing nodes is obtained through 2 times of cross-node communication, which is less than the number of times of cross-node communication in the prior art, and therefore the speed of training the model can be improved.

[0017] In a possible implementation manner of the first aspect or the first implementation manner of the first aspect, in a possible implementation manner, the computing unit of the computing node updates the weight coefficient of the computing unit according to the gradient obtained by the back calculation, including: the computing unit of the computing node obtains the reduction scattering result of the computing unit in the global reduction scattering result, and determines the reference gradient of the computing unit according to the reduction scattering result of the computing unit; the reference gradients of the computing unit group to which the computing unit belongs are collected globally; and the weight coefficient of the computing unit is updated according to the globally collected reference gradients. The computing unit group includes the computing units corresponding to the computing unit in all computing nodes. The global reduction scattering result is obtained by reducing and scattering the gradient obtained by the back calculation of all computing nodes, so that the optimizer state (such as the gradient) can be cut and stored, and the storage overhead of the optimizer state can be reduced. In this way, the gradient of the other computing nodes is obtained through 2 times of cross-node communication, which is less than the number of times of cross-node communication in the prior art, and therefore the speed of training the model can be improved.

[0018] The second aspect provides a computing node, comprising a plurality of computing units, the computing units being configured to sequentially collect global network layer weight coefficients in all computing units of the computing node; perform forward calculation on input data of the network layer according to the global network layer weight coefficients; sequentially collect global network layer weight coefficients in all computing units of the computing node after obtaining a prediction result through the forward calculation; perform backward calculation on input gradient data of the network layer according to the global network layer weight coefficients; update the weight coefficients of the computing units according to the gradient obtained through the backward calculation; and iteratively perform the above steps until the training is completed.

[0019] With reference to the second aspect, in a first possible implementation manner of the second aspect, the computing unit is further configured to release the weight coefficients from other computing units in the global network layer weight coefficients after the computing unit performs the forward calculation on the input data of the network layer according to the global network layer weight coefficients; and release the weight coefficients from other computing units in the global network layer weight coefficients after the computing unit performs the backward calculation on the input gradient data of the network layer according to the global network layer weight coefficients.

[0020] With reference to the second aspect, in a second possible implementation manner of the second aspect, the computing unit is configured to obtain the reduction scattering result of the computing unit in the reduction scattering result of the computing node, perform global reduction on the reduction scattering result of the computing unit group to which the computing unit belongs, and update the weight coefficients of the computing unit according to the gradient obtained through the global reduction.

[0021] With reference to the second possible implementation manner of the second aspect, in a third possible implementation manner of the second aspect, the computing unit is specifically configured to determine a first-order matrix estimation value and a second-order matrix estimation value according to the gradient obtained through the global reduction, and update the weight coefficients of the computing unit according to the first-order matrix estimation value and the second-order matrix estimation value.

[0022] With reference to the second aspect or the first possible implementation manner of the second aspect, in a fourth possible implementation manner of the second aspect, the computing unit is configured to obtain the reduction scattering result of the computing unit in the global reduction scattering result, determine the reference gradient of the computing unit according to the reduction scattering result of the computing unit, perform global collection on the reference gradients of the computing unit group to which the computing unit belongs, and update the weight coefficients of the computing unit according to the global collected reference gradient.

[0023] With reference to the fourth possible implementation manner of the second aspect, in another possible implementation manner of the second aspect, the computing unit is configured to determine a first-order matrix estimation value and a second-order matrix estimation value according to the reduction scattering result of the computing unit, and determine the reference gradient of the computing unit according to the first-order matrix estimation value and the second-order matrix estimation value.

[0024] With reference to the second aspect or any possible implementation manner of the second aspect, in a possible implementation manner of the second aspect, the computing unit is further configured to determine a loss function return value according to the loss function, the prediction result and the preset label value; when the loss function return value is less than or equal to a first threshold, ending the training; when the loss function return value is greater than the first threshold, triggering the step of sequentially collecting, by the computing unit before the forward calculation, the weight coefficients of the network layers in all the computing units of the computing node.

[0025] With reference to the second aspect or any possible implementation manner of the second aspect, in a possible implementation manner of the second aspect, the computing unit is further configured to, when the number of iterations is equal to a second threshold, end the training.

[0026] The terms in the second aspect, the steps performed by the computing unit and the beneficial effects can be referred to the corresponding descriptions of the first aspect.

[0027] The third aspect provides a computing device cluster, the computing device cluster comprising at least one computing device, the at least one computing device comprising at least one processor and at least one memory, the at least one memory storing computer readable instructions; the at least one processor executes the computer readable instructions to enable the computing device cluster to perform the method in the first aspect or any possible implementation manner of the first aspect.

[0028] The fourth aspect, the present application provides a computer readable storage medium, the computer readable storage medium stores instructions, the instructions instruct the computing device or the computing device cluster to perform the method for training the distributed neural network model in the first aspect or any implementation manner of the first aspect.

[0029] The fifth aspect, the present application provides a computer program product comprising instructions, when executed on a computing device or a computing device cluster, enable the computing device or the computing device cluster to perform the method for training the distributed neural network model in the first aspect or any implementation manner of the first aspect.

[0030] On the basis of the implementation manners of the above aspects provided by the present application, further combinations can be made to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0031] FIG. 1 is a schematic diagram of an existing distributed training neural network model;

[0032] FIG. 2 is a schematic diagram of an application scenario of the distributed training neural network model in the embodiment of the present application;

[0033] FIG. 3 is a schematic diagram of a training model in the embodiment of the present application;

[0034] FIG. 4 is a flowchart of a method for training a neural network model in a distributed manner according to an embodiment of the present application;

[0035] FIG. 5 is a flowchart of a method for updating weight coefficients according to a result of back propagation according to an embodiment of the present application;

[0036] FIG. 6 is a schematic diagram of a method for training a neural network model in a distributed manner according to an embodiment of the present application;

[0037] FIG. 7 is a flowchart of another method for updating weight coefficients according to a result of back propagation according to an embodiment of the present application;

[0038] FIG. 8 is a schematic diagram of another method for training a neural network model in a distributed manner according to an embodiment of the present application;

[0039] FIG. 9 is a structural diagram of a computing node according to an embodiment of the present application;

[0040] FIG. 10 is a structural diagram of a computing device according to an embodiment of the present application;

[0041] FIG. 11 is a structural diagram of a cluster of computing devices according to an embodiment of the present application. DETAILED DESCRIPTION

[0042] The method for training a neural network model in a distributed manner according to the present application can be applied to a cluster of computing devices. As shown in FIG. 2, in one example, the cluster of computing devices includes M computing nodes, and each computing node includes K computing units. The computing units can be, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), neural processing units (NPUs), virtual machine instances, or container instances. In the present application, the computing nodes can also be referred to as nodes or computing devices. Each computing node globally collects weight coefficients of all computing units in the computing node; performs forward calculation on training data of the computing units according to the globally collected weight coefficients; globally collects weight coefficients of all computing units in the computing node; performs back propagation according to the globally collected weight coefficients, a predicted result, and a preset label value; updates the weight coefficients of all computing units according to the gradient obtained by the back propagation; and iteratively performs the above steps until the training is completed.

[0043] A neural network model generally includes linear layers and nonlinear layers, which are alternately stacked. The linear layers can perform linear transformation on data, such as convolution layers or fully connected layers, which have learnable weight parameters. The nonlinear layers are also referred to as activation layers, such as rectified linear unit (ReLU) layers and Sigmoid layers.

[0044] After inputting x into the neural network model, the x is subjected to a plurality of layer function transformations of the neural network model, and then y is output. In supervised training, the output data y needs to be fitted with the artificially labeled labels, and the relationship between y and x is fitted by adjusting the weight coefficients. The process of adjusting the weight coefficients to fit x and y is the process of training the neural network model.

[0045] The training of the neural network model can adopt a back propagation method, which calculates the gradient of the weight parameters in the reverse direction according to the difference between the forward predicted output data y and the labels, and then adjusts the value of the weight through the gradient, so that the output data gradually approaches the labels. Such a process of one forward, reverse, and weight adjustment is called one training iteration, and after several ten thousand to several million iterations, the model parameters that meet the conditions are obtained.

[0046] Referring to FIG. 3, in the training phase, a training set for the deep learning model needs to be constructed based on the target, and the training set includes a plurality of training data, and each training data is provided with a label. The label of the training data is the correct answer of the training data on a specific problem, and the label can represent the target of training the deep learning model using the training data. For example, for training a deep learning model that can be used to identify different animals, the training set can include a plurality of images of different animals (i.e., training data), and each image can have a label identifying the type of animal contained therein, for example, cat, dog, and in this example, the type of animal corresponding to each image is the label of the training data.

[0047] When training the deep learning model, the training data can be input into the deep learning model after the parameters are initialized in batches, and the deep learning model calculates (i.e., infers) the training data to obtain a prediction result for the training data. The prediction result obtained by inference and the label corresponding to the training data are used as data for calculating the loss according to the loss function. The loss function is a function used to calculate the gap (i.e., loss value) between the prediction result of the model for the training data and the label of the training data in the model training phase. The loss function can be implemented by different mathematical functions, and common loss functions include mean square error loss function, logarithmic loss function, least squares loss function, etc.

[0048] The loss value calculated based on the loss function can be used to update the parameters of the deep learning model, and the gradient descent method is commonly used for parameter updating. The training of the model is a repeated iterative process, and different training data is inferred and the loss value is calculated in each iteration. The goal of multiple iterations is to continuously update the parameters of the deep learning model to find the parameter configuration that makes the loss value of the loss function the lowest or tends to be stable.

[0049] In the training stage, in order to make the training efficiency of the model and the performance of the model after training more optimal, some reasonable hyperparameters need to be set for training. The hyperparameters of the deep learning model refer to a kind of parameters that cannot be obtained by learning training data or cannot be changed by training data driving in the training process, which is a concept relative to the parameters in the model. The hyperparameters of the deep learning model are usually set by artificial experience or experiment, and the hyperparameters include: learning rate, batch size, network structure hyperparameters (such as network layer number (also called depth), network layer interaction mode, convolution kernel number and convolution kernel size, activation function) and the like. Among them, the learning rate as a hyperparameter is used to control the amplitude of the parameter weight update of the model in the training process, which greatly affects the speed and accuracy of the training.

[0050] The trained deep learning model can be used for inference on input data. In the inference stage, the data of the actual application scene is usually used as the input data, and the inference of the trained deep learning model can obtain the inference result. The inference stage is the actual application of the trained deep learning model, which can quickly use the AI capability to solve specific technical problems. Nowadays, there are many AI application scenarios, and the inference of the deep learning model can also be used in various application scenarios, such as personnel identification scene for access control security system, video pornography and violence detection, express delivery order number detection and identification, etc.

[0051] The above only takes the training of the most typical deep learning model as an example for introduction, and the training of other types of models has slight differences, but the principle is similar, which is to infer the training data, adjust the parameters in the model according to the inference result, and obtain the parameter combination that makes the performance of the model stable as the goal.

[0052] Training is also mainly divided into supervised training and unsupervised training, and the training process of the foregoing deep learning model belongs to supervised training. Taking images as an example, if a neural network model is trained in an unsupervised manner, the training images in the training image set used for training are not labeled, and the training images in the training image set are sequentially input into the neural network model, and the neural network model gradually identifies the association and potential rules between the training images in the training image set until the neural network model can be used to judge or identify the type or characteristics of the input image. For example, clustering, after a neural network model used for clustering receives a large number of training images, it can learn the characteristics of each training image and the association and difference between the training images, and automatically divide the training images into multiple types. Different task types can use different neural network models, some neural network models can only be trained in a supervised learning manner, some neural network models can only be trained in an unsupervised learning manner, and some neural network models can be trained in both a supervised learning manner and an unsupervised learning manner. Generally speaking, the neural network model in machine learning often needs to be trained in a supervised learning manner, and the neural network model trained in a supervised learning manner can more specifically learn the association between the training data and the corresponding labels in the training set, so that the neural network model trained can have a higher accuracy when used to predict other input data.

[0053] After obtaining the gradient, the strategy for updating the weight includes a stochastic gradient descent (SGD) optimization strategy, an adaptive moment estimation (ADMA) optimization strategy, and the like. The SGD optimization strategy multiplies the gradient by a learning rate, and then uses it to adjust the weight. The ADMA optimization strategy calculates and updates the step size by using a first-order moment estimation value m and a second-order moment estimation value v.

[0054] In order to meet the demand of the neural network model for computing power, some parallel training methods are currently proposed, such as data parallelism, model parallelism, hybrid parallelism, pipeline parallelism, and the like, to provide greater computing power.

[0055] FIG. 2 is a data parallel training scenario. Referring to FIG. 2, each node includes K computing units, for example, node 0 includes computing unit 0 to computing unit K-1, and node M-1 includes computing unit N-K+1 to computing unit N. The link between the computing units is a high-bandwidth link, and the link between the computing nodes is a low-bandwidth link.

[0056] The same neural network model to be trained is deployed on each node, and the data set for training is divided into multiple data subsets and distributed to the respective computing units. Each computing unit independently trains the neural network model using the data in its corresponding data subset, and in each training iteration, the gradient value calculated by each computing unit can be synchronized to other computing units, so that each computing unit can obtain the gradient mean value of one iteration process, and each computing unit can update the parameters of the respective neural network model according to the gradient mean value. This is equivalent to training the neural network model by aggregating the data of many mini-batches on multiple computing units to form a large batch data, so that the convergence speed of the model is faster, and the efficiency of model training is improved.

[0057] There are various methods for calculating the average gradient in data parallelism, such as using a parameter server architecture to calculate the average gradient, or calculating the average gradient through collective communication. In collective communication, the average gradient can be calculated through global reduction (allreduce), and the data can be collected through global collection (allgather) and scattered through reduction scatter (reducescatter).

[0058] During the training of the model, the memory overhead includes two parts. The first part is the model state, including the model parameters, the gradient, and the optimizer state. The model parameters include the weight coefficients of the neural network model, and the optimizer state includes the learning rate, the batch sample size, and the decay coefficient. The second part includes the activation, the memory buffer, and the memory fragmentation. In a training scenario in which the number of parameters of a model is φ and the optimizer is an ADAM optimizer, the model parameters in FP16 format require 2φ storage space, and the gradient in FP16 format requires 2φ storage space. The optimizer state includes the optimizer parameters in FP32 format, the first moment in FP32 format, and the second moment in FP32 format, all of which require 4φ storage space. When φ = 7.5B, the model state of this training scenario requires 16*7.5 = 120GB of memory. The current specifications of graphics cards are 16GB, 32GB, or 64GB, and it can be seen that a single card cannot meet the memory requirements of this training scenario. F16 format refers to half-precision floating-point format, and the bit width of F16 format is 16. F32 format refers to single-precision floating-point format, and the bit width of F32 format is 32. It should be understood that the data of the present application is not limited to the above formats, and can be set according to actual conditions.

[0059] In the data parallel method, when each computing unit obtains the same gradient, there is redundancy in subsequent gradient-based calculation results (such as first moment estimation and second moment estimation). The optimizer state splitting and gradient splitting method can reduce the calculation amount of the computing unit, but the communication amount transmitted between the computing units remains unchanged. The communication amount refers to the size of the data transmitted between nodes, which can be denoted as γ.

[0060] In the distributed training method shown in FIG. 1, the model parameters, optimizer states and gradients are all divided into all computing units, which can reduce the storage requirement of the acceleration card, but 3 global communications are required, and the overhead of global communication is 1.5γ. Thus, the training speed is not high, and the communication amount between nodes is large. The present application proposes a distributed training method of multiple copy parameter sharing. Each computing node is deployed with the weight coefficients of the entire neural network model, and the weight coefficients of each network layer in the neural network model are uniformly distributed in the computing units of the computing node. Through global communication within the node, the weight coefficients of each network layer can be obtained, so that each network layer weight coefficient does not need to be obtained through cross-node communication. In the distributed training process, all computing nodes of the computing device cluster perform the method of distributed training of the neural network model in the present application, so that a large-scale neural network model can be trained. The process of a single computing node performing the above method is introduced below with reference to FIG. 4. In an embodiment, the method of distributed training of the neural network model in the present application comprises the following steps:

[0061] S401, the computing units of the computing node sequentially collect the weight coefficients of the network layer in all computing units of the computing node globally.

[0062] In the embodiment, the computing device cluster includes M computing nodes, and each computing node stores the same weight coefficients of the neural network model. Each computing node includes K computing units, and each computing unit can store 1 / K of the weight coefficients of the neural network model. M and K are both integers greater than 1, which can be set according to actual conditions, and the present application does not make any limitation. It should be understood that the computing node performing the method of the present application can be any one of the computing nodes in the computing device cluster.

[0063] The computing units of the computing node collect the weight coefficients of the network layer in all computing units of the computing node globally according to the first sequence. The first sequence is the arrangement order of the network layer in the neural network model. For each network layer of the neural network model, each computing unit stores part of the weight coefficients of the network layer. After collecting the weight coefficients of the network layer in all computing units of the computing node globally, each computing unit can obtain all the weight coefficients of the network layer.

[0064] S402, the computing units of the computing node perform forward calculation on the input data of the network layer according to the globally collected weight coefficients of the network layer.

[0065] Specifically, the neural network model includes T linear layers, and all weight coefficients of the neural network model include weight coefficients of the T linear layers. The value of T can be set according to actual conditions. In the forward calculation process, the training data of the calculation unit is the input data of the first network layer in the neural network model, and the input data of each network layer in the remaining network layers is the output data of the previous network layer. For each linear layer, the calculation unit obtains all weight coefficients of the linear layer through global collection, and performs operation on the input data according to the all weight coefficients of the linear layer, so as to obtain the output data of the linear layer. After the forward calculation, the output result of the last network layer is the prediction result of the neural network model.

[0066] Optionally, the calculation unit corresponds to the training data one by one, and each calculation unit is used to train a training data. The training data of each calculation unit can be considered as a subset of all training data. For example, the calculation device cluster includes N calculation units, and all training data can be divided into N parts, and the training data of each calculation unit is 1 / N of all training data.

[0067] S403, after obtaining the prediction result through the forward calculation, the calculation unit of the calculation node globally collects the weight coefficients of the network layer in all calculation units of the calculation node in turn.

[0068] The calculation unit of the calculation node globally collects the weight coefficients of the network layer in all calculation units of the calculation node according to the second order, and the second order is opposite to the first order. The specific process of globally collecting the weight coefficients of the network layer in all calculation units of the calculation node by the calculation unit of the calculation node is similar to S401.

[0069] Optionally, after obtaining the prediction result through the forward calculation, it can be judged whether the difference between the prediction result and the preset label value is less than a preset threshold. If the difference between the prediction result and the preset label value is less than the preset threshold, it indicates that the current neural network model weight coefficient meets the condition of ending training. If the difference between the prediction result and the preset label value is greater than or equal to the preset threshold, it indicates that the current neural network model weight coefficient does not meet the condition of ending training, and S404 is executed.

[0070] The condition of ending training can also be that the number of iterations reaches a second threshold, or the condition of ending training is that the return value of the loss function is less than or equal to a first threshold, and the input data of the loss function includes the forward calculation result and the preset label value. Alternatively, the calculation node records the return value of the loss function of each training, and the condition of ending training is that the difference between the return values of the loss function in the last two trainings is less than or equal to a preset difference. The loss function can be but is not limited to a mean square error loss function and a cross entropy loss function.

[0071] It should be noted that the condition for ending the training is not limited to the above examples, and can be set according to actual conditions, and the present application is not limited. The preset threshold, the first threshold, the second threshold and the preset difference can be set according to actual conditions, and the present application is not limited.

[0072] S404, the computing unit of the computing node reversely calculates the input gradient data of the network layer according to the globally collected network layer weight coefficients.

[0073] After each computing unit obtains the prediction result, the prediction result and the preset label value are taken as input data of a loss function, the return value of the loss function can reflect the difference between the prediction data and the preset label value, and the input gradient data of each network layer in the computing unit is obtained by reversely calculating the globally collected weight coefficients and the return value of the loss function. Specifically, in the reverse calculation process, the return value of the loss function is taken as the initial gradient, for each linear layer, the computing unit obtains all the weight coefficients of the linear layer through global collection, the gradient output by the previous linear layer is the input gradient data of the linear layer, the output gradient of the linear layer is calculated according to all the weight coefficients of the linear layer and the input gradient data of the linear layer, and the output gradient of all the linear layers can be calculated in this way.

[0074] S405, the computing unit of the computing node updates the weight coefficients of the computing unit according to the gradient obtained by reversely calculating, and jumps to execute S401.

[0075] The specific implementation process of S405 can refer to the embodiment shown in FIG. 5 or the embodiment shown in FIG. 7.

[0076] In this embodiment, in the data parallel process, the weight coefficients of each network layer can be obtained by performing a set communication operation in the computing node during forward calculation and reverse calculation, without global communication across nodes, which can reduce the number of global set communications, thereby improving the efficiency of training a large model.

[0077] Secondly, the weight coefficients of the neural network model are averagely stored in the plurality of computing units in the computing node, compared with storing all the weight coefficients of the neural network model in each computing unit, the method of the present embodiment has low requirements on the storage capacity of the computing unit, and improves the feasibility of the scheme implementation.

[0078] The weight coefficients of the computing unit can be updated by various methods, which will be introduced below. Referring to FIG. 5, in an optional embodiment, S405 includes the following steps:

[0079] S501, the computing unit of the computing node obtains the reduction scattering result of the computing unit in the reduction scattering result of the computing node.

[0080] The gradient data of the computing node includes the gradients obtained by the reverse calculation of all the computing units in the computing node. The computing node performs reduction scattering on the gradient data of the computing node, and each computing unit can obtain the reduction scattering result of the computing unit. For the gradient data of each network layer, each computing unit can divide the gradient data of the network layer into K parts, and then the K computing units of each computing node perform reduction scattering on the K*K gradient data. The reduction scattering result of each computing unit includes parts of gradient data, and each part of gradient data is the gradient sum of K gradient data and the K gradient data comes from the K computing units of the same computing node. K is the total number of computing units in a single computing node.

[0081] S502, the computing unit of the computing node globally reduces the reduction scattering result of the computing unit group to which the computing unit belongs. For example, the computing units with the same serial number in all computing nodes belong to the same computing unit group. After each computing unit in the computing unit group obtains the reduction scattering result of the computing unit, the reduction scattering result of the computing unit group is globally reduced, so that the gradient sum of the computing units in the multiple computing nodes can be obtained. Specifically, the computing unit group includes M computing units and the M computing units come from M computing nodes. The reduction scattering results of the M computing units are globally reduced, that is, the reduction scattering results of the M computing units are summed up, and the sum result can be considered as the gradient sum of N gradient data. It should be noted that the global reduction does not change the number of gradient data in the reduction scattering result of the computing unit.

[0082] It should be understood that the computing nodes with the corresponding relationship in all computing nodes are not limited to the computing nodes with the same serial number, and the computing node group can be set according to the actual situation.

[0083] S503, the computing unit of the computing node updates the weight coefficient of the computing unit according to the gradient obtained by the global reduction.

[0084] Since the gradient data and the weight coefficient globally reduced by each computing unit are parts, the gradient data and the weight coefficient globally reduced by the computing unit are one-to-one corresponding. The gradient data globally reduced by each computing unit is the gradient sum of N gradient data. When updating the weight coefficient, the gradient globally reduced is divided by N to obtain the average gradient. The gradient descent operation according to the average gradient can make the gradient descent relatively smooth.

[0085] In this embodiment, during the step of updating the weight coefficients of all computing units based on the gradient obtained from backpropagation, each computing unit performs one cross-node aggregate communication, that is, one cross-node aggregate communication is performed per training iteration. Compared with the prior art, which performs three cross-node aggregate communications per training iteration, this application can reduce the number of cross-node communications and improve the efficiency of training the model.

[0086] Secondly, after reducing and scattering the gradient data of the network layers, each computational unit stores... The gradient data is in multiple parts, therefore the storage overhead is small. Furthermore, according to... The computational overhead of updating the weight coefficients from the gradient data is relatively small.

[0087] The following describes a method for updating weight coefficients based on the ADMA method. In an optional embodiment, S503 includes: the computing unit of the computing node determines the first-order matrix estimate and the second-order matrix estimate based on the gradient obtained by global reduction; determines the target gradient based on the first-order matrix estimate and the second-order matrix estimate; and updates the weight coefficients of the computing unit based on the target gradient.

[0088] Optionally, the first-order matrix estimate and the gradient obtained from global reduction satisfy the following formula: m t =beta1*m t-1 +(1-beta1)*g t Where beta1 is the first decay rate, m t Let g be the first-order matrix estimate for the t-th iteration. t Let be the gradient obtained by the global reduction in the t-th iteration.

[0089] The second-order matrix estimate and the gradient obtained from global reduction satisfy the following formula: Where beta2 is the second decay rate, v t This is the estimated value of the second-order matrix in the t-th iteration. Let be the square of the gradient obtained by global reduction in the t-th iteration. t is a positive integer.

[0090] The first-order matrix estimate, the second-order matrix estimate, and the weight coefficients of the computational unit satisfy the following formula: Where, θ t Let θ be the weight coefficient after the t-th update. t-1 Let be the weight coefficients after the (t-1)th update, α be the learning rate, and ∈ be the value used to stabilize the weight coefficients. The values ​​of the first decay rate, the second decay rate, the learning rate, and ∈ can be set according to the actual situation, and this application does not impose any restrictions. This allows for automatic adjustment of the learning rate based on historical gradients, improving convergence speed and reducing overfitting, providing an efficient scheme for updating weight coefficients.

[0091] The method of updating the weight coefficient based on the SGD method is introduced as follows. In another optional embodiment, the weight coefficient of the global reduction gradient and the calculation unit satisfies the following formula: θ t = θ t-1 - α * g t . θ t is the weight coefficient after the tth update, θ t-1 is the weight coefficient after the t-1th update, α is the learning rate, g t is the global reduction gradient in the tth cycle. In this way, another method of updating the weight coefficient is provided, and the flexibility of the scheme implementation is improved. It should be noted that the method of updating the weight coefficient is not limited to the above examples.

[0092] For ease of understanding, the method of the distributed training neural network model of the present application is introduced below with an embodiment. Referring to FIG. 6, in an example, the computing device cluster includes M computing nodes, each computing node includes K calculation units, the training data is divided into N parts, each calculation unit processes one part of the training data, for example, the training data of the calculation unit 0 of the computing node 0 is data 0, the training data of the calculation unit K-1 of the computing node 0 is data K-1, the training data of the calculation unit 0 of the computing node M is data N-K, the training data of the calculation unit K of the computing node M is data N, and the training data of other calculation units can be similarly deduced. In each computing node, each calculation unit stores 1 / K of the weight coefficients of the network model.

[0093] In the computing node 0, the computing node 0 collects the weight coefficients of the i layer of all the calculation units in the order of the network layer, so that each calculation unit obtains the weight coefficients of the i layer of the neural network model. After the calculation unit 0 performs forward calculation according to the weight coefficients of the i layer of the neural network model, the weight coefficients of the i layer of other calculation units are released, until the predicted result is calculated. Then, the computing node 0 collects the weight coefficients of the i layer of all the calculation units again in the reverse order of the network layer, so that each calculation unit obtains the weight coefficients of the i layer of the neural network model. The calculation unit 0 performs backward calculation according to the weight coefficients of the i layer of the neural network model, the predicted result and the label value, to obtain the backward calculation result (i.e., the gradient). Similarly, the process of obtaining the backward calculation result by the calculation unit 1 to the calculation unit K-1 is similar to the process of obtaining the backward calculation result by the calculation unit 0, and the process of obtaining the backward calculation result by each calculation unit in the computing node 1 to the last computing node is similar to the process of obtaining the backward calculation result by each calculation unit in the computing node 0.

[0094] In the computing node 0, the computing node 0 performs reduction scattering on the gradients obtained by reverse calculation of all the computing units, and then the computing unit 0 performs global reduction on the reduction scattering results of the computing unit 0 in all the computing nodes. The first-order matrix estimate value and the second-order matrix estimate value are calculated according to the gradients obtained by global reduction, the target gradient of the weight coefficient is determined according to the first-order matrix estimate value and the second-order matrix estimate value, and the weight coefficient of the computing unit 0 is updated by using the target gradient. Similarly, the method for updating the weight coefficient of other computing units in the computing node 0 is similar to the method for updating the weight coefficient of the computing unit 0, and the method for updating the weight coefficient of other computing nodes is similar to the method for updating the weight coefficient of the computing node 0. The above process is iteratively executed, and if the difference between the predicted data of each computing unit in the forward calculation result and the preset label value is less than a threshold value, the training is ended. It should be understood that the values of the gradients obtained by global reduction, the first-order matrix estimate value, the second-order matrix estimate value and the target gradient in the training process depend on the actual operation result.

[0095] Referring to FIG. 7, in another optional embodiment, S405 includes the following steps:

[0096] S701, the computing unit of the computing node obtains the reduction scattering result of the computing unit in the global reduction scattering result.

[0097] After the computing units of all the computing nodes execute S401 to S404, the gradients obtained by reverse calculation of all the computing nodes are performed reduction scattering to obtain a global reduction scattering result. The global reduction scattering result includes the reduction scattering results of all the computing units, and each computing unit can obtain the reduction scattering result of the computing unit. Specifically, for the gradient data of each network layer, each computing unit divides the gradient data of the network layer into N parts, and then N computing units perform reduction scattering on the N*N parts of gradient data. The reduction scattering result obtained by each computing unit includes parts of gradient data, and each part of gradient data is the gradient sum of N gradient data and the N gradient data comes from N computing units respectively, and N is the total number of computing units in the computing device cluster.

[0098] S702, the computing unit of the computing node determines the reference gradient of the computing unit according to the reduction scattering result of the computing unit.

[0099] S703, the computing unit of the computing node globally collects the reference gradients of the computing unit groups to which the computing unit belongs.

[0100] If the computing unit group includes M computing units and the M computing units come from M computing nodes, the global collection result of the computing unit includes (i.e. ) parts of reference gradients.

[0101] S704. The computational units of the computation nodes update the weight coefficients of the computational units based on the globally collected reference gradients.

[0102] The gradient data obtained by global reduction in each computational unit is the sum of the gradients of N gradient data. When updating the weight coefficients, the reference gradient collected globally is divided by N to obtain the average gradient. Performing gradient descent operations based on the average gradient can make gradient descent smoother.

[0103] In this embodiment, the computing unit of the computing node needs to perform two global operations: global reduction scattering and global collection. This is less than the three global operations performed in existing methods, thus reducing the number of global communications between nodes and improving training efficiency.

[0104] Secondly, after reducing and scattering the gradient data of the network layers, each computational unit stores... This provides a subset of gradient data, thus further reducing the storage overhead of gradient data. Furthermore, according to... Updating the weight coefficients using gradient data can further reduce computational overhead.

[0105] Furthermore, this provides another method for updating weight coefficients, increasing the flexibility of the scheme implementation.

[0106] The following describes a method for calculating a reference gradient based on the ADMA method. In an optional embodiment, S702 includes: the computing unit of the computing node determines the first-order matrix estimate and the second-order matrix estimate based on the gradient obtained by global reduction; and determines the reference gradient based on the first-order matrix estimate and the second-order matrix estimate.

[0107] In this embodiment, the first-order matrix estimate and the gradient obtained from global reduction satisfy the following formula: m t =beta1*m t-1 +(1-beta1)*g t Where beta1 is the first decay rate, m t Let g be the first-order matrix estimate for the t-th iteration. t Let be the gradient obtained by the global reduction in the t-th iteration.

[0108] The second-order matrix estimate and the gradient obtained from global reduction satisfy the following formula: Where beta2 is the second decay rate, v t This is the estimated value of the second-order matrix in the t-th iteration. Let be the square of the gradient obtained by global reduction in the t-th iteration. t is a positive integer.

[0109] The first-order matrix estimate, the second-order matrix estimate, and the reference gradient satisfy the following formula: wherein, is the reference gradient after the t+1th update, a is the learning rate, and ∈ is a value used to stabilize the weight coefficient, which can be but is not limited to 0, and can be set according to actual conditions, which is not limited in the present application. In this way, an efficient method for calculating the reference gradient is provided.

[0110] The method for calculating the reference gradient based on the SGD method is introduced below. In another optional embodiment, the reference gradient and the reduction scattering result of the calculation unit satisfy the following formula: g t = a * g t-1 . g t is the reference gradient after the tth update, a is the learning rate, g t-1 is the reference gradient after the t-1th update. In this way, another method for calculating the reference gradient is provided, and the flexibility of the scheme implementation is improved. It should be noted that the method for calculating the reference gradient in the present application is not limited to the above examples.

[0111] In another optional embodiment, after S402 and S404, each calculation unit of the computing node retains the local weight coefficient and releases the weight coefficient of other calculation units. Each calculation unit stores the weight coefficient of 1 / K, and the training method of the present embodiment can be implemented. Compared with the method in which each calculation unit stores all the weight coefficients of the neural network model, the method of the present embodiment has low requirements on the storage capacity of the calculation unit, and facilitates the implementation of the distributed training method of the neural network model in the present application by small-capacity calculation units, thereby improving the feasibility of the scheme implementation.

[0112] For ease of understanding, the method for distributed training of the neural network model in the present application is introduced below with an embodiment. Referring to FIG. 8, in an example, the computing device cluster includes M computing nodes, each computing node includes K calculation units, the training data is divided into N parts, each calculation unit processes one part of the training data, for example, the training data of the calculation unit 0 of the computing node 0 is data 0, the training data of the calculation unit K-1 of the computing node 0 is data K-1, the training data of the calculation unit 0 of the computing node M is data N-K, and the training data of the calculation unit K-1 of the computing node M is data N. The training data of other calculation units can be deduced in the same way. In each computing node, each calculation unit stores 1 / K weight coefficients of the neural network model.

[0113] In the computing node 0, the computing node 0 globally collects the weight coefficients of the i-th layer of all the computing units in the order of network layers, so that each computing unit obtains the weight coefficients of the i-th layer of the neural network model. After the computing unit 0 performs forward calculation according to the weight coefficients of the i-th layer of the neural network model, the weight coefficients of the i-th layer of other computing units are released, until the forward calculation obtains the prediction result. Then, the computing node 0 globally collects the weight coefficients of the i-th layer of all the computing units in the reverse order of network layers, so that each computing unit obtains the weight coefficients of the i-th layer of the neural network model. The computing unit 0 performs backward calculation according to the weight coefficients of the i-th layer of the neural network model to obtain the backward calculation result. Similarly, the computing unit 1 to the computing unit K-1 obtain the backward calculation result in the same way as the computing unit 0, and each computing unit in the computing node 1 to the last computing node obtains the backward calculation result in the same way as each computing unit in the computing node 0.

[0114] In the computing node 0, the computing node 0 reduces and scatters the gradients obtained by the backward calculation of all the computing units. The computing unit 0 calculates the first-order matrix estimate and the second-order matrix estimate according to the reduction and scattering result of the computing unit 0, determines the reference gradient of the weight coefficient according to the first-order matrix estimate and the second-order matrix estimate, globally collects the reference gradients of the computing units 0 of all the computing nodes, and updates the weight coefficients of the computing units 0 with the globally collected reference gradients. Similarly, the computing unit 1 to the computing unit K-1 update the weight coefficients in the same way as the computing unit 0, and the other computing nodes update the weight coefficients in the same way as the computing node 0. The above process is iteratively performed, and if the difference between the prediction data of each computing unit and the preset label value is less than a threshold value, the training is ended. It should be noted that the values of the gradients obtained by the reduction and scattering, the first-order matrix estimate, the second-order matrix estimate and the reference gradient depend on the actual operation result.

[0115] Referring to FIG. 9, the present application provides a computing node 900, which includes a plurality of computing units 901. The computing unit 901 is configured to sequentially globally collect the weight coefficients of the network layer in all the computing units of the computing node, perform forward calculation on the input data of the network layer according to the globally collected weight coefficients of the network layer, sequentially globally collect the weight coefficients of the network layer in all the computing units of the computing node after the forward calculation obtains the prediction result, perform backward calculation on the input gradient data of the network layer according to the globally collected weight coefficients of the network layer, update the weight coefficients of the computing unit according to the gradient obtained by the backward calculation, and iteratively perform the above steps until the training is ended.

[0116] In an optional embodiment, the computing unit 901 is further configured to release the weight coefficients of the global collected network layer weight coefficients from other computing units after the computing unit performs forward calculation on the input data of the network layer according to the global collected network layer weight coefficients; and release the weight coefficients of the global collected network layer weight coefficients from other computing units after the computing unit performs backward calculation on the input gradient data of the network layer according to the global collected network layer weight coefficients.

[0117] In another optional embodiment, the computing unit 901 is specifically configured to obtain the reduction scattering result of the computing unit in the reduction scattering result of the computing node, the reduction scattering result of the computing node being obtained by performing reduction scattering on the gradients obtained by performing backward calculation on all computing units of the computing node; perform global reduction on the reduction scattering result of the computing unit group to which the computing unit belongs, the computing unit group including the computing units corresponding to the computing unit in all computing nodes; and update the weight coefficients of the computing unit according to the gradient obtained by the global reduction. Optionally, the computing unit 901 is specifically configured to determine the first-order matrix estimation value and the second-order matrix estimation value according to the gradient obtained by the global reduction; and update the weight coefficients of the computing unit according to the first-order matrix estimation value and the second-order matrix estimation value.

[0118] In another optional embodiment, the computing unit 901 is specifically configured to obtain the reduction scattering result of the computing unit in the global reduction scattering result, and determine the reference gradient of the computing unit according to the reduction scattering result of the computing unit; perform global collection on the reference gradients of the computing unit group to which the computing unit belongs; and update the weight coefficients of the computing unit according to the global collected reference gradient. Optionally, the computing unit 901 is specifically configured to determine the first-order matrix estimation value and the second-order matrix estimation value according to the reduction scattering result of the computing unit; and determine the reference gradient of the computing unit according to the first-order matrix estimation value and the second-order matrix estimation value.

[0119] In another optional embodiment, the computing unit 901 is further configured to determine the loss function return value according to the loss function, the prediction result and the preset label value; end the training when the loss function return value is less than or equal to the first threshold value; and trigger the computing unit to perform the step of globally collecting the weight coefficients of the network layer in all computing units of the computing node when the loss function return value is greater than the first threshold value.

[0120] In another optional embodiment, the computing unit 901 is further configured to end the training when the number of iterations is equal to the second threshold value.

[0121] It should be noted that the computing unit in the present application can be a software module, for example, an optimizer or a graph engine component in a training framework, and the training framework can be but is not limited to TensorFlow, PyTorch, and the graph engine component can be but is not limited to a split deployment component.

[0122] The present application also provides a computing device 1000. As shown in FIG. 10, the computing device 1000 includes a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, the memory 1006, and the communication interface 1008 communicate via the bus 1002. It should be understood that the present application does not limit the number of processors and the number of memories in the computing device 1000.

[0123] The bus 1002 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is shown in FIG. 10, but it does not mean that there is only one bus or only one type of bus. The bus 1002 can include a path for transmitting information between various components (e.g., the memory 1006, the processor 1004, the communication interface 1008) of the computing device 1000.

[0124] The processor 1004 can include any one or more of a CPU, a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc. The processor includes a plurality of processing cores.

[0125] The memory 1006 can include a volatile memory (e.g., a random access memory (RAM)) and / or a non-volatile memory (e.g., a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD)). In some embodiments, the memory 1006 stores executable program code, and the processor 1004 executes the executable program code to respectively implement the functions of the aforementioned computing units 901, thereby implementing the method of training a neural network model in a distributed manner.

[0126] The communication interface 1008 uses a transceiver module such as, but not limited to, a network interface card, a transceiver, etc., to implement communication between the computing device 1000 and other devices or communication networks.

[0127] The embodiments of the present application also provide a computing device cluster, which can realize the functions of the data processing apparatus. The computing device cluster comprises at least one computing device. The computing device can be a server, for example, a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer, or a smart phone.

[0128] As shown in FIG. 11, the computing device cluster comprises at least one computing device 1000. The same instructions for performing the method of distributed training of a neural network model can be stored in the memory 1006 of one or more computing devices 1000 in the computing device cluster.

[0129] The embodiments of the present application also provide a computer program product comprising instructions. The computer program product can be a software or program product comprising instructions, which can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device is caused to perform the method of distributed training of a neural network model.

[0130] The embodiments of the present application also provide a computer readable storage medium. The computer readable storage medium can be any available medium that the computing device can store or a data storage device such as a data center comprising one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), etc. The computer readable storage medium comprises instructions, which instruct the computing device to perform the method of distributed training of a neural network model.

[0131] The above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the above-mentioned embodiments of the present application are described in detail, those skilled in the art should understand that the technical solutions recorded in the above-mentioned embodiments can be modified, or some technical features can be replaced by equivalent ones; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for distributed training of a neural network model in a cluster of computing devices, the method comprising: The computing device cluster comprises a plurality of computing nodes, each of the computing nodes comprises a plurality of computing units, a communication bandwidth between the computing units in the computing node is greater than a communication bandwidth between the computing nodes, each of the computing nodes stores weight coefficients of a neural network model, and the weight coefficients of each network layer in the neural network model are uniformly distributed in the plurality of computing units of the computing node, and the method comprises: The computing units of the computing node sequentially collect the weight coefficients of the network layer in all the computing units of the computing node globally; The computing units of the computing node perform forward calculation on input data of the network layer according to the globally collected weight coefficients of the network layer; After obtaining a prediction result through the forward calculation, the computing units of the computing node sequentially collect the weight coefficients of the network layer in all the computing units of the computing node globally; The computing units of the computing node perform backward calculation on input gradient data of the network layer according to the globally collected weight coefficients of the network layer; The computing units of the computing node update the weight coefficients of the computing units according to the gradients obtained through the backward calculation, and iteratively execute the above steps until the training is completed.

2. The method of claim 1, wherein, The method further comprises: After the computing units of the computing node perform forward calculation on the input data of the network layer according to the globally collected weight coefficients of the network layer, the computing units of the computing node release the weight coefficients from other computing units in the globally collected weight coefficients of the network layer; After the computing units of the computing node perform backward calculation on the input gradient data of the network layer according to the globally collected weight coefficients of the network layer, the computing units of the computing node release the weight coefficients from other computing units in the globally collected weight coefficients of the network layer.

3. The method according to claim 1 or 2, characterized in that, The computing units of the computing node updating the weight coefficients of the computing units according to the gradients obtained through the backward calculation comprises: The computing units of the computing node obtain the reduction scattering result of the computing units in the reduction scattering result of the computing node, and the reduction scattering result of the computing node is obtained by performing reduction scattering on the gradients obtained through the backward calculation of all the computing units of the computing node; The computing units of the computing node globally reduce the reduction scattering result of the computing unit group to which the computing units belong, and the computing unit group comprises the computing units corresponding to the computing units in all the computing nodes; The computing units of the computing node update the weight coefficients of the computing units according to the gradients obtained through the global reduction.

4. The method of claim 3, wherein, The computing units of the computing node updating the weight coefficients of the computing units according to the gradients obtained through the global reduction comprises: The computing units of the computing node determine a first-order matrix estimation value and a second-order matrix estimation value according to the gradients obtained through the global reduction; The computing units of the computing node update the weight coefficients of the computing units according to the first-order matrix estimation value and the second-order matrix estimation value.

5. The method according to claim 1 or 2, characterized in that, The computing units of the computing node updating the weight coefficients of the computing units according to the gradients obtained through the backward calculation comprises: The computing unit of the computing node obtains the reduction scattering result of the computing unit in the global reduction scattering result, the global reduction scattering result being obtained by reducing and scattering the gradient obtained by reverse calculation of all computing nodes; The computing unit of the computing node determines the reference gradient of the computing unit according to the reduction scattering result of the computing unit; The computing unit of the computing node globally collects the reference gradients of the computing unit group to which the computing unit belongs; The computing unit of the computing node updates the weight coefficient of the computing unit according to the globally collected reference gradients.

6. The method of claim 5, wherein, The computing unit of the computing node determines the reference gradient of the computing unit according to the reduction scattering result of the computing unit, including: The computing unit of the computing node determines the first-order matrix estimation value and the second-order matrix estimation value according to the reduction scattering result of the computing unit; The computing unit of the computing node determines the reference gradient of the computing unit according to the first-order matrix estimation value and the second-order matrix estimation value.

7. The method according to any one of claims 1 to 6, characterized in that, After the prediction result is obtained by forward calculation, the method further includes: The computing unit of the computing node determines the loss function return value according to the loss function, the prediction result and the preset label value; When the loss function return value is less than or equal to the first threshold value, the computing unit of the computing node ends the training; When the loss function return value is greater than the first threshold value, the computing unit of the computing node triggers the step of globally collecting the weight coefficients of the network layer in all computing units of the computing node in turn before the forward calculation.

8. The method according to any one of claims 1 to 6, characterized in that, The method further includes: When the number of iterations is equal to the second threshold value, the computing unit of the computing node ends the training.

9. A computing node, characterized in that, The computing node includes a plurality of computing units, and the computing units are used to globally collect the weight coefficients of the network layer in all computing units of the computing node in turn; The input data of the network layer is forward calculated according to the globally collected weight coefficients of the network layer, and the weight coefficients of the network layer in all computing units of the computing node are globally collected in turn after the prediction result is obtained by forward calculation; The input gradient data of the network layer is reverse calculated according to the globally collected weight coefficients of the network layer; The weight coefficients of the computing unit are updated according to the gradient obtained by reverse calculation, and the above steps are iteratively executed until the training ends.

10. The computing node of claim 9, wherein, The computing unit is also used to release the weight coefficients from other computing units in the globally collected weight coefficients of the network layer after the computing unit performs forward calculation on the input data of the network layer according to the globally collected weight coefficients of the network layer; The computing unit is also used to release the weight coefficients from other computing units in the globally collected weight coefficients of the network layer after the computing unit performs reverse calculation on the input gradient data of the network layer according to the globally collected weight coefficients of the network layer.

11. The computing node according to claim 9 or 10, c h a r a c t e r i z e d b y The computing unit is specifically used to obtain the reduction scattering result of the computing unit in the reduction scattering result of the computing node, the reduction scattering result of the computing node being obtained by reducing and scattering the gradient obtained by reverse calculation of all computing units of the computing node; performing global reduction on the reduction scattering result of the computing unit group to which the computing unit belongs, the computing unit group including all computing units corresponding to the computing unit in all computing nodes; updating the weight coefficient of the computing unit according to the gradient obtained through global reduction.

12. The computing node of claim 11, wherein, The computing unit is specifically configured to determine a first-order matrix estimation value and a second-order matrix estimation value according to the gradient obtained through global reduction; and update the weight coefficient of the computing unit according to the first-order matrix estimation value and the second-order matrix estimation value.

13. The computing node of claim 9 or 10, wherein, The computing unit is specifically configured to obtain the reduction scattering result of the computing unit in the global reduction scattering result, the global reduction scattering result being obtained by performing reduction scattering on the gradient obtained through reverse calculation of all computing nodes; determine the reference gradient of the computing unit according to the reduction scattering result of the computing unit; perform global collection on the reference gradients of the computing unit group to which the computing unit belongs; update the weight coefficient of the computing unit according to the reference gradient collected globally.

14. The computing node of claim 13, wherein, The computing unit is specifically configured to determine a first-order matrix estimation value and a second-order matrix estimation value according to the reduction scattering result of the computing unit; and determine the reference gradient of the computing unit according to the first-order matrix estimation value and the second-order matrix estimation value.

15. The computing node according to any of claims 9 to 14, characterized in that, The computing unit is further configured to determine a loss function return value according to a loss function, the prediction result and a preset label value; end the training when the loss function return value is less than or equal to a first threshold value; and trigger the step of sequentially performing global collection by the computing unit on the weight coefficients of the network layers of all computing units of the computing node before forward calculation when the loss function return value is greater than the first threshold value.

16. The computing node according to any one of claims 9 to 14, wherein, The computing unit is further configured to end the training when the number of iterations is equal to a second threshold value.

17. A cluster of computing devices, characterized in that, The computing device cluster includes at least one computing device, the at least one computing device including at least one processor and at least one memory, the at least one memory storing computer readable instructions; the at least one processor executes the computer readable instructions, so that the computing device cluster performs the method in any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that, including computer readable instructions; the computer readable instructions are used to implement the method in any one of claims 1 to 8.

19. A computer program product, characterised in that, including computer readable instructions; the computer readable instructions are used to implement the method in any one of claims 1 to 8.

Images