A model training method and device, a storage medium and an apparatus

Abstract

The specification discloses a model training method and device, a storage medium and equipment. The current node and other nodes determine outputs by using a to-be-trained model based on different training samples, calculate gradients of dense parameters and gradients of sparse parameters of the to-be-trained model according to the outputs respectively. After the nodes interact the gradients calculated by themselves, the dense parameters are updated based on the gradients corresponding to the dense parameters, and the sparse parameters required to be updated by each node are updated based on the gradients corresponding to the sparse parameters. The sparse parameters required to be updated by different nodes are not completely the same. After the parameters are updated, the current node continues to jointly train the to-be-trained model with other nodes according to the updated dense parameters and sparse parameters. Through distributed training, the sparse parameters and the dense parameters of the to-be-trained model can be synchronously updated in different ways, and the training efficiency of the model is improved.

Description

Technical Field

[0001] This specification relates to the field of machine learning technology, and in particular to a model training method, apparatus, storage medium and device. Background Technology

[0002] Users are increasingly concerned about the privacy of their data. Currently, deep learning models are being used more and more widely. To improve the training efficiency of deep learning models, existing technologies propose a distributed training method using a parameter server / worker (PS / Worker) model. In this model, the parameter server adjusts the parameters based on the gradient, while the computation nodes calculate the gradient based on the model output and labels.

[0003] To improve the efficiency of distributed training, this manual provides an alternative model training method. Summary of the Invention

[0004] This specification provides a model training method, apparatus, storage medium, and device to at least partially solve the above-mentioned technical problems.

[0005] The following technical solution is adopted in this specification:

[0006] This specification provides a model training method applied to nodes performing distributed training of a model to be trained. The model to be trained contains dense parameters and sparse parameters, and the sparse parameters that different nodes need to update are not exactly the same. The method includes:

[0007] The current node inputs its own training samples into the model to be trained, and determines the first gradient corresponding to the dense parameters and the second gradient corresponding to the sparse parameters based on the output of the model to be trained and the labels of the training samples.

[0008] The first gradient and the second gradient are sent to other nodes, and the third gradient corresponding to the dense parameters and the fourth gradient corresponding to the sparse parameters are received from the other nodes. The third gradient and the fourth gradient are determined by the other nodes based on their own training samples through the model to be trained.

[0009] Update the dense parameters of the model to be trained based on the first gradient and the third gradient;

[0010] The sparse parameters that the current node needs to update are updated based on at least a portion of the gradient in the second gradient and / or at least a portion of the gradient in the fourth gradient.

[0011] The updated sparse parameters are sent to the other nodes, and the updated sparse parameters sent by the other nodes are received.

[0012] Based on the updated dense and sparse parameters, the other nodes are used to continue training the model until the model convergence condition is met.

[0013] This specification provides a model training apparatus for use on nodes performing distributed training of a model to be trained. The model to be trained contains dense parameters and sparse parameters, and the sparse parameters that different nodes need to update are not exactly the same. The apparatus includes:

[0014] The determination module is used to input the training samples of the current node itself into the model to be trained, and determine the first gradient corresponding to the dense parameters and the second gradient corresponding to the sparse parameters based on the output of the model to be trained and the labeling of the training samples.

[0015] The first interaction module is used to send the first gradient and the second gradient to other nodes, and to receive the third gradient corresponding to the dense parameters and the fourth gradient corresponding to the sparse parameters sent by the other nodes, wherein the third gradient and the fourth gradient are determined by the other nodes based on their own training samples through the model to be trained.

[0016] The first update module is used to update the dense parameters of the model to be trained based on the first gradient and the third gradient.

[0017] The second update module is used to update the sparse parameters of the current node according to at least a portion of the gradient in the second gradient and / or at least a portion of the gradient in the fourth gradient.

[0018] The second interaction module is used to send the updated sparse parameters to the other nodes and receive the updated sparse parameters sent by the other nodes.

[0019] The training module is used to continue training the model to be trained in conjunction with the other nodes based on the updated dense and sparse parameters until the model convergence condition is met.

[0020] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described model training method.

[0021] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the model training method described above.

[0022] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:

[0023] In the above model training method, after the current node inputs its own training samples into the model to be trained, it can determine the first gradient of the dense parameters and the second gradient of the sparse parameters of the model to be trained based on the model output and the annotation of the training samples. The current node then sends the first and second gradients to other nodes and receives the third gradient of the dense parameters and the fourth gradient of the sparse parameters determined and sent by other nodes based on their own training samples. Afterward, the current node can update the dense parameters based on the first and third gradients, and update its own sparse parameters based on at least a portion of the second gradient and / or at least a portion of the fourth gradient. Other nodes update their dense parameters and their own sparse parameters based on the received gradients. The current node sends the updated sparse parameters to other nodes and receives the sparse parameters that other nodes need to update, and then continues distributed training of the model to be trained with other nodes based on the update results.

[0024] As can be seen from the above, the model training method provided in this manual can improve the training efficiency of the model by synchronously updating the sparse and dense parameters of the model to be trained in different ways through distributed training. Attached Figure Description

[0025] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and their descriptions, serving to explain this specification and do not constitute an undue limitation thereof.

[0026] In the picture:

[0027] Figure 1 This is a flowchart illustrating a model training method provided in this specification;

[0028] Figure 2 This is a schematic diagram of a model training system provided in this specification.

[0029] Figure 3 This is a schematic diagram of one of the algorithm modules provided in this specification;

[0030] Figure 4 This is a schematic diagram illustrating the construction of a forward computation graph provided in this specification;

[0031] Figure 5 This is a schematic diagram illustrating the operation rewriting of a reverse process provided in this specification;

[0032] Figure 6This is a schematic diagram of a model training device described in this specification;

[0033] Figure 7 This is a schematic diagram of an electronic device provided in this specification. Detailed Implementation

[0034] In the field of deep learning, improving model training efficiency is essential. Especially with the ever-increasing volume of training data for deep learning today, the time required to train a model is also growing, hindering model development and iteration. Therefore, the challenge of improving model training efficiency cannot be ignored.

[0035] Current distributed training methods typically employ the PS / Worker model to improve the training efficiency of deep learning models.

[0036] To improve the training efficiency of deep models, this manual provides a model training method.

[0037] In this specification, each node participating in distributed training can use this model training method to perform distributed training on the model to be trained.

[0038] In the training mode corresponding to the model training method provided in this specification, there is no distinction between parameter servers and computing nodes. For each node participating in distributed training, the node can be used to both calculate gradients and update parameters.

[0039] Furthermore, considering the differences in parameters among deep learning models, this specification divides the parameters of the model to be trained into sparse parameters and dense parameters, and updates dense and sparse parameters in different ways. Sparse parameters are those that may not be updated in every training iteration, while dense parameters are those that are updated in every training iteration.

[0040] In one or more embodiments of this specification, the sparse parameters of the model to be trained can be the parameters of the embedding layer of the model to be trained, and the dense parameters can be the parameters of the fully connected layer.

[0041] The embedding layer is used for vector mapping, that is, mapping different inputs to different vectors, each vector being a sparse parameter. Therefore, the embedding layer contains a large number of vectors that can match different inputs, resulting in a massive number of sparse parameters, typically tens of thousands. However, an input data set of the model being trained can only match a subset of the sparse parameters of the embedding layer; therefore, a training sample can only be used to update a portion of the sparse parameters.

[0042] Taking the product recommendation model as an example, if the input of the model to be trained is the products in a search page that the user has browsed, then each product can be matched with a vector of the embedding layer, that is, a sparse parameter.

[0043] Dense parameters are parameters that are inevitably used in the forward computation of the model. Therefore, no matter how the input of the model to be trained changes, any training sample can be used to update all dense parameters.

[0044] In the training mode corresponding to the model training method provided in this specification, the nodes participating in distributed training jointly train the model to be trained based on different training samples. For example, each node can train the model to be trained using different batches of training samples. Furthermore, each node can determine the gradients of each parameter of the model to be trained based on one training sample in its own batch and then update the parameters of the model to be trained; alternatively, each node can determine the gradients of each parameter of the model to be trained based on all training samples in its own batch and then update the parameters of the model to be trained. That is, each node can jointly perform one round of distributed training on the model to be trained based on one training sample in its own batch, or each node can jointly perform one round of distributed training on the model to be trained based on all training samples in its own batch.

[0045] Furthermore, each node can update each parameter synchronously. That is, after each node has calculated the gradient of the same parameter, the parameter is updated based on the gradients calculated by each node for that parameter. Compared to asynchronous updates, synchronous updates can avoid the impact of differences in node computation speeds on training efficiency.

[0046] Asynchronous updates mean that for each parameter, as soon as a node calculates its gradient, the parameter can be updated, resulting in the updated parameter. This is done without waiting for all nodes to calculate their gradients before updating that node. However, when different nodes calculate gradients at different speeds, this difference in computation speed can limit the model's training efficiency. For example, a faster node might update the parameter to the new version, while a slower node might update it back to the old version, causing some nodes to perform unnecessary work and thus limiting the model's training efficiency.

[0047] In this specification, the dense parameters of the model to be trained are of a magnitude that a single node can handle; that is, their quantity is sufficient for a single node to be responsible for updating all dense parameters. Therefore, to reduce the communication burden between nodes, each node can calculate the gradient of all dense parameters and update each dense parameter based on the gradients calculated by each node. To realize the calculation of the gradient of dense parameters and the updating of parameters based on the gradient, the nodes only need to interact for the purpose of passing the gradient, without needing to interact to pass the updated dense parameters. In this way, the number of communication sessions between nodes and the amount of data sent can be reduced, thereby improving the communication efficiency in the distributed training process.

[0048] Because the number of sparse parameters in the model to be trained is enormous, requiring more memory and computational resources for updates, sparse parameters can be stored in shards across nodes to improve computational efficiency. Different nodes store different shards. That is, each node stores a portion of the sparse parameters and is responsible for updating its own corresponding sparse parameter set. Only one complete set of sparse parameters for the model to be trained exists globally; each node does not store a copy of the global sparse parameters, but rather a portion, i.e., a shard. The sparse parameters corresponding to shards stored on different nodes do not have to be completely identical (including partial similarities and complete differences). That is, the sparse parameters corresponding to shards stored on different nodes can be partially the same. Alternatively, for clear division of labor, the sparse parameters corresponding to shards stored on different nodes can be completely different. The following explanation uses the example of shards stored on different nodes having completely different sparse parameters.

[0049] Since different nodes are responsible for updating slightly different sparse parameters, each node needs to interact twice during a training iteration of the model: once for passing parameters and once for passing gradients.

[0050] Furthermore, since each sparse parameter may not be updated in every training round, nor does its gradient need to be calculated in every round, each node only needs to transmit the gradients and / or updated parameters of a portion of the sparse parameters during each training round. Therefore, although the number of sparse parameters in the model to be trained is enormous, storing the sparse parameters in chunks and allowing different nodes to update different sparse parameters does not impose a communication burden on the nodes; on the contrary, it improves computational efficiency, thereby enhancing training efficiency.

[0051] As can be seen, this specification can classify parameters into sparse parameters and dense parameters according to their characteristics, and adopt different methods for parameter updates accordingly. This can reduce the number of communication sessions and the amount of communication data between nodes while ensuring computational efficiency.

[0052] Furthermore, since the nodes in this specification are used for both calculating gradients and updating parameters, the utilization rate of the nodes can be improved, achieving "making the best use of the nodes".

[0053] In this specification, for sparse parameters, nodes can perform parameter passing, gradient calculation, and parameter updates through an all-to-all approach. For dense parameters, nodes can perform gradient calculation and parameter updates through an all-reduce approach. In the all-to-all approach, the data sent by the same node to different nodes differs, and different nodes receive different data from the same node. The purpose of the all-reduce approach is to integrate the data from all nodes, ensuring that each node receives the integrated data.

[0054] Each node can update the parameters of the model to be trained through multiple rounds of training. In each round, each node uses the updated parameters of the model from the previous round as the current parameters for training. That is, based on the previously updated model, each node calculates the gradient using its corresponding training samples, and then updates the parameters of the model based on the gradient, completing the gradient calculation and parameter update for the current round. Then, the next round of training continues based on the updated model until the model converges, resulting in a well-trained model.

[0055] It should be noted that this manual describes the process of distributing the training of the model to be trained between the current node and other nodes from the perspective of one of the nodes participating in the distributed training (hereinafter referred to as the current node). Furthermore, for ease of description only, the nodes participating in the distributed training of the model to be trained are divided into the current node and other nodes; each node participating in the distributed training can be considered the current node.

[0056] Furthermore, this specification does not limit the number of nodes participating in distributed training. Other nodes can include multiple nodes, and the number of other nodes can be set as needed. Moreover, as mentioned above, in this specification, each node can be used both to calculate gradients and to update parameters based on gradients.

[0057] In this specification, the current node and other nodes can be different devices, such as different servers (specifically, chips within different devices). Alternatively, they can be different chips within the same device. For example, the chip can be a Central Processing Unit (CPU). Alternatively, to reduce communication overhead and improve efficiency between the current node and other nodes during training, the chip can also be a Graphics Processing Unit (GPU). Compared to CPUs, high-speed direct connections between GPUs enable faster interaction, improving communication efficiency and thus training efficiency.

[0058] Therefore, in one or more embodiments of this specification, the current node and other nodes may include a graphics processor. The graphics processor is used to calculate gradients and update parameters. Furthermore, the current node and other nodes can transmit data via a high-speed direct connection channel of the graphics processor.

[0059] Furthermore, regarding privacy protection, the model training method provided in this manual differs from existing methods. In this manual, nodes can transmit data without a gateway. Instead, data can be read through a high-speed direct connection. This improves communication efficiency and prevents data leakage.

[0060] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0061] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0062] Figure 1 This is a flowchart illustrating a model training method described in this specification. This model training method is applied to nodes in a distributed training process of the model to be trained. The model to be trained contains dense and sparse parameters, and different nodes require different updates to the sparse parameters. The model training method specifically includes the following steps:

[0063] S100: The current node inputs its own training samples into the model to be trained, and determines the first gradient corresponding to the dense parameters and the second gradient corresponding to the sparse parameters based on the output of the model to be trained and the labeling of the training samples.

[0064] Since the parameters of the model to be trained need to be updated based on the gradient, and the gradient needs to be calculated based on the output of the model to be trained after the training samples are input into the model to be trained.

[0065] Therefore, the current node can obtain its own training samples and input them into the model to be trained. Based on the output of the model and the labels of the training samples, it can determine the first gradient corresponding to the dense parameters and the second gradient corresponding to the sparse parameters. As mentioned above, due to the characteristics of sparse parameters, the second gradient obtained by the current node may not be the second gradient of all sparse parameters.

[0066] Specifically, the current node can input the training sample corresponding to the current node into the model to be trained for forward computation, so as to determine the first gradient corresponding to the dense parameters and the second gradient corresponding to the sparse parameters based on the forward computation result, the label of the training sample and the loss function of the model to be trained.

[0067] The forward computation result can include both the final result and intermediate results. The final result is the output of the model to be trained, and the intermediate results are the outputs of each network in the model, such as the outputs of each fully connected layer. Furthermore, a dense parameter corresponds to a first gradient, and a sparse parameter corresponds to a second gradient.

[0068] The training sample corresponding to the current node can be obtained by partitioning the entire training sample used to train the model, based on the total number of the current node and other nodes. The training sample corresponding to the current node is different from the training samples corresponding to other nodes.

[0069] For example, taking a scenario where there are 2 other nodes, the total number of nodes participating in distributed training is 3. The training samples for the model to be trained can be divided into 3 batches, with each of the 3 nodes corresponding to one batch. For ease of distinction and description, the portion corresponding to the current node is designated as the first batch, and the portions corresponding to each of the other nodes are designated as the second and third batches, respectively. The current node trains the model based on the first batch, while one of the other nodes trains the model based on the second batch, and the other node trains the model based on the third batch.

[0070] Therefore, the training sample obtained by the current node can be the first batch, or it can be a training sample from the first batch. Of course, when the first batch is obtained, each training sample is input into the model to be trained separately. The same applies to other nodes, and the number of training samples obtained is consistent with that of the current node.

[0071] That is, the current node can update its parameters once based on a training sample, or it can update its parameters once based on a batch of training samples, and the same applies to other nodes.

[0072] S102: Send the first gradient and the second gradient to other nodes, and receive the third gradient corresponding to the dense parameters and the fourth gradient corresponding to the sparse parameters sent by the other nodes, wherein the third gradient and the fourth gradient are determined by the other nodes based on their own training samples through the model to be trained.

[0073] In this specification, the current node and the other nodes are all responsible for calculating the gradients of each sparse parameter, and not just the gradients of the sparse parameters that need to be updated.

[0074] However, due to the differences in training samples, the sparse parameters involved in different training samples are different. Therefore, for each node, after each training sample is input into the model to be trained, the sparse parameters corresponding to the second gradient determined based on the output of the model to be trained and the labels of the training samples can be exactly the same, not exactly the same, or even completely different. For example, when the model to be trained is a recommendation model, the training samples can be the products on a page that a user has browsed. Then, one product can correspond to one sparse parameter. Since the products involved in different training samples are different, the sparse parameters that can be updated based on different training samples are also different.

[0075] That is, although the sparse parameters that the current node needs to update are predetermined, the second gradient corresponding to each sparse parameter that it needs to update cannot be calculated every time a training sample is input into the model to be trained. Furthermore, the current node does not only calculate the second gradient of the sparse parameters that it needs to update, but also calculates the second gradient of each sparse parameter corresponding to the training sample.

[0076] Since the sparse parameters that the current node needs to update are not exactly the same as those of the other nodes, the current node can send the second gradients of the sparse parameters to the other nodes after calculating them. This allows the other nodes to filter out the second gradients corresponding to the sparse parameters that the current node needs to update, so that they can update the sparse parameters that the current node needs to update based on the filtered second gradients (when the training samples input to the model by the other nodes do not involve the same sparse parameters and have not calculated the gradients of the same sparse parameters). Alternatively, it allows the other nodes to filter out the second gradients corresponding to the sparse parameters that the current node needs to update, so that they can update the sparse parameters that the current node needs to update based on the filtered second gradients and the gradients of the sparse parameters that the current node needs to update, calculated based on its own training samples (when the training samples input to the model by the other nodes involve the same sparse parameters and have obtained the gradients of the same sparse parameters).

[0077] The sparse parameters that the current node needs to update are not exactly the same as those of the other nodes; that is, they can be completely different or partially the same. However, the sparse parameters that each node needs to update can cover all the sparse parameters of the model to be trained.

[0078] The current node can also receive the fourth gradients of the sparse parameters calculated by other nodes, and filter out the gradients corresponding to the sparse parameters that the current node needs to update. This allows the current node to update its sparse parameters based on the filtered fourth gradients and the second gradients of the sparse parameters calculated from its own training samples (when the training samples input to the model involve the same sparse parameters). Alternatively, the current node can update its sparse parameters based on the filtered fourth gradients (when the training samples input to the model do not involve the same sparse parameters).

[0079] Alternatively, the current node can calculate the second gradient of the sparse parameters based on its own corresponding training samples, determine the gradient of the sparse parameters that other nodes need to update, and send the determined gradient of the sparse parameters that other nodes need to update to the corresponding other nodes, so that the other nodes can update the sparse parameters that other nodes need to update according to the gradient sent by the current node.

[0080] Similarly, after calculating the gradients corresponding to all sparse parameters, the other node can send the gradients of the sparse parameters that the current node needs to update to the current node. The current node can receive only the fourth gradient of the sparse parameters that the current node needs to update.

[0081] In one or more embodiments of this specification, the current node may determine the second gradient corresponding to the sparse parameters that other nodes need to update from the second gradient, use it as the target gradient, and send the first gradient and the target gradient to other nodes.

[0082] Furthermore, the current node can receive the fourth gradient corresponding to the sparse parameters that the current node needs to update, as well as the third gradient corresponding to the dense parameters, sent by other nodes.

[0083] The first gradient sent by the current node is used to enable other nodes to update the dense parameters based on the first gradient and the third gradient, and the target gradient is used to enable other nodes to update the sparse parameters that other nodes need to update based on the target gradient and at least some of the gradients in the fourth gradient.

[0084] It should be noted that since the computation speed of the current node may differ from that of other nodes, and the sending and receiving of gradients have no logical order and will not affect each other, this specification does not restrict the order in which nodes receive and send gradients.

[0085] For dense parameters, since each node is responsible for updating all dense parameters, after the current node calculates the first gradient of each dense parameter based on its own training samples, it can send the first gradient to other nodes. Other nodes can then update each dense parameter based on the first gradient sent by the current node and the third gradient of each dense parameter calculated based on the training samples of those other nodes.

[0086] Similarly, the other node can send the third gradient of each dense parameter calculated based on the training samples of the other node to the current node.

[0087] The current node can then receive the third gradient of each dense parameter sent by the other node, so that it can update each dense parameter according to the first gradient of each dense parameter calculated by itself and the received third gradient.

[0088] Therefore, after determining the first and second gradients, the current node can send the first and second gradients to other nodes, and receive the third gradient corresponding to the dense parameters and the fourth gradient corresponding to the sparse parameters sent by other nodes.

[0089] The third and fourth gradients are determined by other nodes using the model to be trained based on their own training samples.

[0090] It should be noted that since gradient calculation and parameter updates for sparse parameters and for dense parameters are two independent and logically distinct processes, the first and second gradients can be sent separately. The current node does not need to wait for both gradients to be calculated before sending them to other nodes, thus reducing unnecessary waiting time. Alternatively, to reduce the number of communications, the current node can also wait for both gradients to be calculated before sending them to other nodes. The same applies to other nodes sending the third and fourth gradients.

[0091] S104: Update the dense parameters of the model to be trained based on the first gradient and the third gradient.

[0092] After obtaining the third gradient, the current node can update the dense parameters of the model to be trained based on the first and third gradients.

[0093] S106: Update the sparse parameters of the current node according to at least a portion of the gradient in the second gradient and / or at least a portion of the gradient in the fourth gradient.

[0094] As described above, since the fourth gradient sent by other nodes can be any of the calculated fourth gradients, or it can be the fourth gradient corresponding to the partial sparse parameters that the current node needs to update, but the current node can only update the sparse parameters that it needs to update based on the fourth gradient corresponding to the sparse parameters that it needs to update. Therefore, after obtaining the fourth gradient, the current node can update the partial sparse parameters that it needs to update based on at least a portion of the fourth gradient and at least a portion of the second gradient calculated by the current node.

[0095] Additionally, it should be noted that since the second and fourth gradients are calculated based on different training samples, the sparse parameters matched by these different training samples may be the same, partially the same, or even completely different. Therefore, the current node updates the sparse parameters based on at least some gradients in the second gradient and / or at least some gradients in the fourth gradient. That is, for each sparse parameter in the sparse parameters that the current node needs to update, the sparse parameter is updated based on the second gradient and / or the fourth gradient corresponding to that sparse parameter.

[0096] For example, suppose the sparse parameters that the current node needs to update are B, C, and D. The training samples corresponding to the current node itself only match the sparse parameters B and D, and can only be input into the model to be trained, so as to calculate the second gradient of the sparse parameters B and D based on the output of the model to be trained. The training samples corresponding to other nodes only match the sparse parameters C and D, and can only be input into the model to be trained, so as to calculate the fourth gradient of the sparse parameters C and D based on the output of the model to be trained.

[0097] Therefore, for sparse parameter B, the current node is updated only based on the second gradient corresponding to sparse parameter B. For sparse parameter D, the current node can be updated based on both the second and fourth gradients corresponding to sparse parameter D. For sparse parameter C, the current node can be updated only based on the fourth gradient corresponding to sparse parameter C.

[0098] S108: Send the updated sparse parameters to the other nodes, and receive the updated sparse parameters sent by the other nodes.

[0099] After updating the dense parameters, the updated dense parameters of each node are consistent. However, since different nodes need to update different sparse parameters, after updating the sparse parameters, each node can exchange its updated sparse parameters, so that each node can hold the latest parameters of the model to be trained for the next round of training.

[0100] Therefore, the current node can send the updated sparse parameters to other nodes and receive sparse parameters updated by other nodes.

[0101] Alternatively, due to the differences in sparse parameters involved in the training samples, the sparse parameters obtained by the current node in this round of training may not be applied in the next round of training.

[0102] Therefore, in one or more embodiments of this specification, the current node may further redetermine its own training samples and determine the sparse parameters corresponding to the redetermined training samples as target parameters. For each target parameter, the node that updated the target parameter is queried.

[0103] When a target parameter has been updated by another node, the current node can send a parameter retrieval request to that node. This allows other nodes to send the target parameter required by the current node based on the parameter retrieval request.

[0104] Then, the current node can interpret the sparse parameters that other nodes send as target parameters, which are updates from other nodes.

[0105] Furthermore, the current node can also receive parameter retrieval requests from other nodes, and based on the received parameter retrieval requests, send its updated sparse parameters to other nodes.

[0106] Therefore, in one or more embodiments of this specification, when the current node sends the updated sparse parameters to other nodes, it may specifically receive an acquisition request sent by the other nodes. Based on the acquisition request, the updated sparse parameters are then sent to the other nodes. This acquisition request is determined by the other nodes based on the training samples required for them to continue training the model to be trained.

[0107] In one or more embodiments of this specification, when the current node receives sparse parameters updated by other nodes, it may first determine the training samples required to continue training the model to be trained, and use them as input samples.

[0108] Then, based on the pre-stored correspondence between training samples and sparse parameters, the sparse parameters corresponding to the input sample can be determined as the target sparse parameters.

[0109] Then, a request can be sent to other nodes that update the target sparse parameters, and the target sparse parameters updated by those other nodes can be received.

[0110] Alternatively, in one or more embodiments of this specification, the current node may also broadcast the identifier of the sparse parameters involved in the redefined training samples to other nodes. After receiving the identifier, other nodes can determine whether there is an identifier of their own updated sparse parameters. If there is, they will send the corresponding updated sparse parameters to the current node.

[0111] S110: Based on the updated dense and sparse parameters, continue to train the model to be trained in conjunction with the other nodes until the model convergence condition is met.

[0112] Once the current node holds the latest updated dense parameters of the model to be trained and the sparse parameters involved in the current training samples, the current node can continue to train the model to be trained in conjunction with other nodes based on the latest updated dense and sparse parameters until the model convergence condition is met.

[0113] The convergence condition of the model can be set as needed. For example, the loss that is less than a preset first threshold can be used as the ideal loss in the loss determined by the output of the model to be trained and the annotation of the training samples.

[0114] After inputting several training samples into the model to be trained and determining the loss multiple times based on the output of the model to be trained and the labels of the training samples, the model convergence condition is determined to be met when the number of ideal losses is greater than the preset second threshold.

[0115] Of course, other methods can be used to determine whether the model convergence condition is met. For example, after training the model to be trained for a preset number of rounds using training samples from the training set, the model can be tested using test samples from the test set to determine the Area Under Curve (AUC) metric for the model. Based on the determined AUC value, it can be determined whether the model convergence condition is met.

[0116] based on Figure 1 The model training method shown involves the current node inputting its own training samples into the model to be trained. Based on the model output and the annotations of the training samples, the current node determines the first gradient of the dense parameters and the second gradient of the sparse parameters of the model to be trained. It then sends these first and second gradients to other nodes and receives the third gradient of the dense parameters and the fourth gradient of the sparse parameters, determined and sent by other nodes based on their own training samples. The current node then updates its dense parameters based on the first and third gradients, and updates its sparse parameters based on at least some of the gradients in the second and / or the fourth gradients. Other nodes update their dense parameters and their own sparse parameters based on the received gradients. The current node sends the updated sparse parameters to other nodes and receives the updated sparse parameters from them. Based on the update results, the current node can then continue distributed training of the model to be trained with other nodes.

[0117] Using the above method, the current node and other nodes determine the output of the model to be trained based on different training samples. Each node then calculates the gradients of the dense and sparse parameters of the model based on its own output. After exchanging their calculated gradients, each node updates its dense parameters based on the gradients corresponding to the dense parameters, and then updates its own sparse parameters based on the gradients corresponding to the sparse parameters. The sparse parameters that different nodes need to update are not entirely the same. After updating the parameters, the current node continues to collaborate with other nodes to perform distributed training of the model to be trained, based on the updated dense and sparse parameters. This distributed training method allows for the synchronous updating of the sparse and dense parameters of the model to be trained in different ways, improving the training efficiency of the model.

[0118] In addition, in one or more embodiments of this specification, each node may include a chief node and slave nodes.

[0119] During the first round of training of the model, the parameters of the model can be initialized. For dense parameters, the master node can initialize them and broadcast them to all slave nodes. For sparse parameters, since different nodes need to update different sparse parameters, each node can initialize its own required sparse parameters and send them to other nodes. Alternatively, each node can send its corresponding initialized sparse parameters to other nodes based on parameter retrieval requests received from other nodes.

[0120] Therefore, before inputting the current node's own training samples into the model to be trained, each node can also transmit its own initialized parameters.

[0121] In one or more embodiments of this specification, when the current node is the master node, the dense parameters of the model to be trained, as well as the first sparse parameter that the current node needs to update, can be initialized. The initialized dense parameters and the first sparse parameter are then sent to other nodes.

[0122] Furthermore, the current node can receive the second sparse parameter that other nodes need to update from among the sparse parameters of the model to be trained. This second sparse parameter is obtained from the initialization of other nodes.

[0123] Then, the current node can obtain the initialized model to be trained based on the initialized dense parameters, the first sparse parameter, and the second sparse parameter. Other nodes can also obtain the initialized model to be trained based on the initialized dense parameters, the first sparse parameter, and the second sparse parameter.

[0124] Similarly, there is no sequential order in the operations of sending and receiving parameters by the current node. Furthermore, the distinction between the first and second sparse parameters is merely for ease of description and does not imply that there are two nodes. When there are multiple other nodes, each other node corresponds to a set of second sparse parameters.

[0125] In one or more embodiments of this specification, when the current node is a slave node, the current node can receive the dense parameters of the model to be trained obtained from the initialization broadcast by the master node.

[0126] Furthermore, the current node can receive sparse parameters that other slave nodes need to update when initializing, as well as sparse parameters that the master node needs to update when initializing, sent by other slave nodes and the master node respectively.

[0127] Of course, the current node can initialize the sparse parameters that need to be updated for the current node, and send the initialized sparse parameters to other slave nodes and the master node.

[0128] Finally, the current node can obtain the initialized model to be trained based on the initialized dense parameters and the initialized sparse parameters. Similarly, other nodes can obtain the initialized model to be trained based on the initialized dense parameters and the initialized sparse parameters.

[0129] It should be noted that there is no specific order in which the current node receives the dense parameters broadcast by the master node, receives the sparse parameters sent by other slave nodes and the master node respectively, and sends the initialized sparse parameters to other slave nodes and the master node.

[0130] Furthermore, in step S104, when updating the dense parameters of the model to be trained based on the first and third gradients, specifically, the current node can determine the gradients corresponding to the dense parameter in the first and third gradients for each dense parameter. It then calculates the average of the gradients corresponding to the dense parameter and updates the dense parameter based on the obtained average.

[0131] In one or more embodiments of this specification, when updating sparse parameters, the current node can determine the gradients corresponding to the sparse parameter in the second gradient and the fourth gradient for each sparse parameter that the current node needs to update, so as to calculate the average of the gradients corresponding to the sparse parameter, and update the sparse parameter according to the obtained average.

[0132] It should be noted that there are no restrictions on the type of model to be trained. For example, the model to be trained can be a recommendation model, a classification model, or other models.

[0133] Furthermore, as mentioned above, the purpose of full reduction is to integrate the data from all nodes, ensuring that each node receives the integrated data. This can be achieved in various ways. In the embodiments described above, each node sends the gradient of its calculated dense parameters to every other node. Then, each node can calculate the mean of the gradients corresponding to the dense parameters based on the received gradients, and update its data accordingly.

[0134] To conserve resources, one node can receive the gradients corresponding to the dense parameters calculated by each node, calculate the mean, and send the result to the other nodes. This eliminates the need for each node to perform the calculation, thus saving computational and communication resources. For example, each node can implement all reduce using algorithms such as Halving Doubling (HD) or Ring-Reduce.

[0135] Therefore, each node can include representative nodes and regular nodes. Each regular node can send the calculated gradients of the dense parameters to the representative nodes. The representative nodes can receive the gradients from the regular nodes and, for each dense parameter, calculate its mean based on its gradient. Then, the representative nodes can broadcast the means of each dense parameter to all regular nodes. Finally, each node can update its dense parameters based on the corresponding means.

[0136] There are other ways to implement the full specification, which will not be described in this specification.

[0137] In one or more embodiments of this specification, the representative node can be a master node, and the regular node can be a slave node. Alternatively, the representative node can also be a slave node, and this specification does not impose any restrictions on this.

[0138] In addition, for ease of understanding, this manual also provides related information. Figure 1 The corresponding model training system, such as Figure 2 As shown.

[0139] Figure 2 This is a schematic diagram of a model training system provided in this specification. The system includes a current node and other nodes. The current node and the other nodes synchronously update the sparse and dense parameters. The current node and the other nodes update the parameters of the model to be trained based on different training samples. Specifically, the current node updates the parameters based on a first training sample, while the other nodes update the parameters based on a second training sample.

[0140] Among the sparse parameters of the model to be trained, the sparse parameter that the current node needs to update is the first sparse parameter, and the sparse parameter that the other nodes need to update is the second sparse parameter.

[0141] When updating sparse parameters, the current node communicates with other nodes in an all-to-all manner. The current node first sends the first sparse parameter to other nodes and receives the second sparse parameter from them. The other nodes then send the second sparse parameter to the current node and receive the first sparse parameter. Through this first round of interaction, the current node and all other nodes obtain all the latest updated parameters of the model to be trained. Of course, the first sparse parameter sent by the current node to other nodes may be the sparse parameters involved in the other nodes' own training samples, and may not necessarily be all the sparse parameters that the current node needs to update. Similarly, the second sparse parameter sent by other nodes to the current node may be the sparse parameters involved in the current node's own training samples, and may not necessarily be all the sparse parameters that the other nodes need to update.

[0142] Then, the current node can obtain the output corresponding to the first training sample through the model to be trained, based on the first training sample, and determine the second gradient of the sparse parameters based on the output and the annotation of the first training sample. The other nodes can obtain the output of the second training sample through the model to be trained, and determine the fourth gradient of the sparse parameters based on the output and the output annotation of the second training sample.

[0143] The current node can send the second gradient to other nodes and receive the fourth gradient from other nodes. Other nodes, in turn, send the fourth gradient to the current node and receive the second gradient from the current node. This completes the second round of interaction between the current node and other nodes when updating sparse parameters.

[0144] The current node and other nodes can update at least some of their respective sparse parameters based on the second gradient and / or the fourth gradient.

[0145] When updating dense parameters, since the current node and other nodes update all dense parameters separately, there is no need to pass dense parameters. The current node and other nodes only need to interact with the gradient in one round through full reduction.

[0146] The current node can determine the output corresponding to the first training sample after inputting the first training sample into the model to be trained, and then determine the first gradient of each dense parameter based on the output and annotation of the first training sample. The other nodes can determine the output corresponding to the second training sample after inputting the second training sample into the model to be trained, and then determine the third gradient of each dense parameter based on the output and annotation of the second training sample.

[0147] The current node can send the first gradient to other nodes and receive the third gradient sent by other nodes. Other nodes can then send the third gradient to the current node and receive the first gradient sent by the current node.

[0148] Of course, the output of the first training sample used when updating sparse parameters and dense parameters at the current node is the same, not obtained by inputting the first training sample into the model to be trained separately. The same applies to other nodes.

[0149] In one or more embodiments of this specification, the model training system and the model training method can be applied to the TensorFlow (TF) framework.

[0150] To ensure consistency with checkpoint (CKPT) files for other training modes within the TensorFlow framework, and to allow models previously trained using other TensorFlow models to be applied within the training mode corresponding to the model training method described in this manual, the dense parameters of the chief node and all sparse parameters of each node can be placed in global variables, while the dense parameters of non-chief nodes can be placed in local variables. This method ensures that the saved CKPT files are consistent with existing TensorFlow training modes, facilitating user operation.

[0151] The CKPT file is used to store the trained model or to store the progress of the training model.

[0152] Because the dense variables corresponding to dense parameters are replicated, meaning each node stores the full set of dense parameters, the dense variables of all nodes can be based on the initialization variables of the chief node. Furthermore, the placement strategies for dense and sparse parameters can differ. In the TensorFlow framework, the placement strategy determines how an operation is assigned to a device for execution; in this specification, it refers to the variable placement strategy.

[0153] The sparse parameter placement strategy places a sparse parameter fragment in each node. The dense parameter placement strategy places the full set of dense parameters in each node.

[0154] Furthermore, the model training system in this specification may include higher-level operations (Ops) on key-value variable (KVVariable) files in all-to-all mode. Ops are operations used for computation (e.g., arithmetic operations), defining constant variables, defining placeholders, etc., and are therefore operators. KVVariables are variables used to store parameters and their corresponding gradients.

[0155] To facilitate understanding of how the model training system is implemented, this manual also provides the algorithm modules included in the training mode of the model training system under the TF framework: Application Programming Interface (API) module, kernel module, and communication backend module.

[0156] The API may include: device fn, training mode switching interface, variable creator, gradient rewriter, and all-to-all embedding lookup.

[0157] The device interface is used to place shards of sparse variables onto independent nodes. The training mode switching interface provides an embedding operation for one-click switching from other training modes in the TensorFlow framework to the training modes provided in this manual.

[0158] In the TensorFlow framework, the computation graph is the foundation for model training. Nodes in the graph represent operations (ops), and edges between nodes represent data. The graph is TensorFlow's data flow diagram used to describe the training process.

[0159] This variable creation interface allows users to control the initializer of dense variables during graph construction, as well as the partitioning of function return values ​​collected from dense or sparse variables in the graph, so as to manage dense or sparse parameters independently later.

[0160] The gradient application interface is used to update parameters during backpropagation by applying the obtained gradients in the manner of dense variable all-reduce and sparse variable all-to-all-reduce, and to return the rewritten training ops after the user has rewritten them. It should be noted that the reason for rewriting the ops is to modify the ops of other training modes in the TensorFlow framework to the ops corresponding to the training modes provided in this specification.

[0161] The lookup interface provides an all-to-all embedding query operator for the kvvariable. This query operator is used to retrieve the value for a given key.

[0162] The kernel module is used to implement the custom kernel in the training mode provided in this manual, which can be further divided into two parts: communication and computation.

[0163] The communication kernel can be used to implement the all-to-all, broadcasting, or all-reduce communication primitives. That is, it can include all-to-all, broadcasting, and all-reduce operations. Additionally, it can include custom operations for sparse merge queries. Sparse merge queries combine queries with different sparse parameters. Without sparse merge queries, a corresponding network connection needs to be established for each query with sparse parameters, and each query corresponds to an all-to-all process, resulting in extremely high communication costs. With sparse merge queries, the operation can perform queries for all sparse parameters in a single execution.

[0164] The computation kernel is used to perform forward and backward computations of sparse parameters.

[0165] The communication backend module is used to implement low-level communication. It includes communication libraries based on various communication protocols such as Transmission Control Protocol (TCP), Remote Direct Memory Access (RDMA), Peripheral Component Interconnect Express (PCIE), and nvlink. It serves as the backend for communication and can provide protocol support for the communication kernel.

[0166] This instruction manual also provides Figure 3 The diagram shows a schematic of the algorithm module. As shown, the algorithm module consists of three parts: an interface, a kernel, and a communication backend. The interface includes a device interface, a training mode switching interface, a variable creation interface, a gradient application interface, and a lookup interface.

[0167] The kernel comprises two types: a communication kernel and a computation kernel. The communication kernel includes the all-to-all reduce operation (all-to-all reduce), the encoding operation, the all-reduce operation, the coalescing operation, and the broadcasting operation. The computation kernel includes the optimizer and key-value pair variables (KVvariables). The optimizer computes gradients and updates parameters. Furthermore, the optimizer interacts with the key-value pair variables, retrieves the corresponding parameters from them, updates the key-value pair variables based on the computed gradients, and updates the key-value pair variables again based on the parameters obtained from the gradient updates.

[0168] The communication backend is used to provide protocol support for full switching protocols, full protocols, and broadcasting operations.

[0169] In addition, this specification also provides a schematic diagram of constructing a forward computation graph, such as... Figure 4 As shown, the TensorFlow framework can request the creation of a computation graph through the graph creation interface. Then, graph nodes can be created. These graph nodes are the nodes in the graph, representing operations (ops), not the nodes mentioned above used for gradient calculation. For distinction, nodes used for gradient calculation and parameter updating are designated as computation nodes.

[0170] When creating a graph node, firstly, a graph node can be generated, and a request can be made to the device interface to allocate compute nodes for the graph node. The device interface can then allocate the corresponding compute nodes for the graph node according to the user-predefined mapping between ops and compute nodes, and return the allocated compute nodes. Afterwards, a request can be made to the variable creation interface to adjust the collection and initializers for the graph node. This variable creation interface can allocate initializers for broadcasting dense parameters to master nodes and initializers for receiving dense parameters to slave nodes.

[0171] After the initializer is assigned, the variable creation interface returns the initializer. Furthermore, the variable creation interface can assign different collections for dense and sparse parameters. Specifically, all variables under the master node can be configured as global collections, while slave nodes can only configure global collections for sparse parameters and local collections for dense parameters. Additionally, sparse parameters in all compute nodes can be written to a sparse collection. Afterward, the variable creation interface returns the allocation result. The graph node creation is complete. The process of creating graph nodes can then be repeated.

[0172] After all graph nodes have been created, the creation result can be returned. That is, the forward computation graph can be returned.

[0173] Furthermore, to implement the model training system provided in this specification, enabling nodes within this system to perform distributed training of the model to be trained by executing the model training methods provided in this specification, training operations (train ops) can be defined within the training modes provided in this specification. This allows for the rewriting of the original ops in the TensorFlow framework.

[0174] This instruction manual also provides Figure 5 The diagram illustrates the rewriting of the reverse process. (See diagram for example.) Figure 5The TensorFlow framework can create training operations by overriding interfaces and obtain the compute gradient operations (ops) from the created backpropagation graph, using them as the compute gradient operations in that training operation. It can also obtain the apply gradient operations (ops) and rewrite them through the gradient application interface. The gradient application interface can be rewritten based on the following logic: distinguishing between the gradients of dense and sparse parameters according to the collection during forward graph construction; performing a full exchange update of sparse parameters based on the gradients of sparse parameters; and performing a full reduction update of dense parameters based on the gradients of dense parameters. Furthermore, the full exchange update and full reduction update are implemented through a communication backend based on ops in the communication kernel. Afterward, the gradient application interface can return the rewritten apply gradient operation. Finally, it can return the rewritten training operation.

[0175] This manual also provides information related to Figure 1 The corresponding device. For example... Figure 6 As shown.

[0176] Figure 6 This is a schematic diagram of a model training device provided in this specification. The model training device is applied to nodes performing distributed training on a model to be trained. The model to be trained contains dense parameters and sparse parameters, and different nodes require different updates to the sparse parameters. The device includes:

[0177] The determination module 200 is used to input the training samples of the current node itself into the model to be trained, and determine the first gradient corresponding to the dense parameters and the second gradient corresponding to the sparse parameters based on the output of the model to be trained and the labeling of the training samples.

[0178] The first interaction module 201 is used to send the first gradient and the second gradient to other nodes, and to receive the third gradient corresponding to the dense parameters and the fourth gradient corresponding to the sparse parameters sent by the other nodes, wherein the third gradient and the fourth gradient are determined by the other nodes based on their own training samples through the model to be trained.

[0179] The first update module 202 is used to update the dense parameters of the model to be trained based on the first gradient and the third gradient.

[0180] The second update module 203 is used to update the sparse parameters of the current node according to at least a portion of the gradient in the second gradient and / or at least a portion of the gradient in the fourth gradient.

[0181] The second interaction module 204 is used to send the updated sparse parameters to the other nodes and receive the sparse parameters updated by the other nodes.

[0182] The training module 205 is used to continue training the model to be trained in conjunction with the other nodes based on the updated dense parameters and sparse parameters until the model convergence condition is met.

[0183] Optionally, the first interaction module 201 is further configured to determine, from the second gradient, the second gradient corresponding to the sparse parameters that the other nodes need to update, as the target gradient, send the first gradient and the target gradient to the other nodes, and receive the fourth gradient corresponding to the sparse parameters that the current node needs to update, and the third gradient corresponding to the dense parameters sent by the other nodes.

[0184] Optionally, the first update module 202 is further configured to, for each dense parameter, determine the gradients corresponding to the dense parameter in the first gradient and the third gradient, calculate the average of the gradients corresponding to the dense parameter, and update the dense parameter according to the obtained average.

[0185] Optionally, the second update module 203 is further configured to, for each sparse parameter that needs to be updated for the current node, determine the gradients corresponding to the sparse parameter in the second gradient and the fourth gradient, calculate the average of the gradients corresponding to the sparse parameter, and update the sparse parameter according to the obtained average.

[0186] Optionally, each node includes a master node and a slave node.

[0187] Optionally, the device further includes:

[0188] The first initialization module 206, before inputting the training samples of the current node into the model to be trained, is used to initialize the dense parameters of the model to be trained and the first sparse parameter that the current node needs to update when the current node is the master node. It then sends the initialized dense parameters and the first sparse parameter to the other nodes, receives the second sparse parameter that the other nodes need to update from the sparse parameters of the model to be trained, where the second sparse parameter is obtained by the other nodes during initialization, and obtains the initialized model to be trained based on the initialized dense parameters, the first sparse parameter, and the second sparse parameter.

[0189] Optionally, the device further includes:

[0190] The second initialization module 207, before inputting the training samples of the current node itself into the model to be trained, is used to, when the current node is a slave node, receive the dense parameters of the model to be trained obtained by the master node broadcasting the initialization, receive the sparse parameters to be updated by the other slave nodes and the master node respectively sent by the other slave nodes and the master node, initialize the sparse parameters to be updated by the master node, initialize the sparse parameters to be updated by the current node, and send the initialized sparse parameters to the other slave nodes and the master node, and obtain the initialized model to be trained based on the initialized dense parameters and the initialized sparse parameters.

[0191] Optionally, the second interaction module 204 is further configured to receive an acquisition request sent by the other nodes; and send the updated sparse parameters to the other nodes according to the acquisition request; wherein the acquisition request is determined by the other nodes based on the training samples required for the other nodes to continue training the model to be trained.

[0192] Optionally, the second interaction module 204 is further configured to determine the training samples required to continue training the model to be trained, as input samples; determine the sparse parameters corresponding to the input samples according to the pre-stored correspondence between training samples and sparse parameters, as target sparse parameters; send an acquisition request to other nodes that update the target sparse parameters, and receive the target sparse parameters updated by the other nodes.

[0193] Optionally, the current node and the other nodes include a graphics processor, which is used to calculate gradients and update parameters, and the current node and the other nodes transmit data through a high-speed direct connection channel of the graphics processor.

[0194] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the model training method described above.

[0195] This instruction manual also provides Figure 7 The diagram shows a schematic structural representation of the electronic device. Figure 7At the hardware level, the electronic device includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for the business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the above-mentioned model training method. Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0196] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0197] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, ASICs, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0198] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0199] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0200] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0201] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0202] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0203] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0204] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0205] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0206] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0207] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0208] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0209] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0210] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0211] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A model training method applied to each node for training a to-be-trained model, wherein the to-be-trained model comprises dense parameters and sparse parameters, and different nodes need to update different sparse parameters; the method comprises: inputting, by a current node, a training sample of the current node into the to-be-trained model, determining a first gradient corresponding to the dense parameters and a second gradient corresponding to the sparse parameters according to an output of the to-be-trained model and a label of the training sample; sending the first gradient and the second gradient to other nodes, and receiving a third gradient corresponding to the dense parameters and a fourth gradient corresponding to the sparse parameters sent by the other nodes, wherein the third gradient and the fourth gradient are determined by the other nodes based on training samples of the other nodes through the to-be-trained model; updating the dense parameters of the to-be-trained model according to the first gradient and the third gradient; updating the sparse parameters needed to be updated by the current node according to at least part of the second gradient and / or at least part of the fourth gradient; sending the updated sparse parameters to the other nodes, and receiving the sparse parameters updated by the other nodes; and continuing to train the to-be-trained model in combination with the other nodes according to the updated dense parameters and sparse parameters until a model convergence condition is met. 2.The method of claim 1, wherein sending the first gradient and the second gradient to other nodes, and receiving a third gradient corresponding to the dense parameters and a fourth gradient corresponding to the sparse parameters sent by the other nodes comprises: determining, from the second gradient, a second gradient corresponding to the sparse parameters needed to be updated by the other nodes as a target gradient; sending the first gradient and the target gradient to the other nodes; and receiving the fourth gradient corresponding to the sparse parameters needed to be updated by the current node and the third gradient corresponding to the dense parameters sent by the other nodes. 3.The method of claim 1, wherein updating the dense parameters of the to-be-trained model according to the first gradient and the third gradient comprises: determining, for each dense parameter, the gradients corresponding to the dense parameter in the first gradient and the third gradient; and averaging the gradients corresponding to the dense parameter, and updating the dense parameter according to the obtained average value. 4.The method of claim 1, wherein updating the sparse parameters needed to be updated by the current node according to at least part of the second gradient and / or at least part of the fourth gradient comprises: determining, for each sparse parameter needed to be updated by the current node, the gradients corresponding to the sparse parameter in the second gradient and the fourth gradient; and averaging the gradients corresponding to the sparse parameter, and updating the sparse parameter according to the obtained average value. 5.The method of claim 1, wherein the nodes comprise a master node and slave nodes. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ 6.The method of claim 5, wherein before inputting the training sample of the current node itself into the to-be-trained model, the method further comprises: initializing each dense parameter of the to-be-trained model and a first sparse parameter needed to be updated by the current node when the current node is a master node; sending the initialized each dense parameter and the first sparse parameter to other nodes; receiving a second sparse parameter needed to be updated by the other nodes from each sparse parameter of the to-be-trained model sent by the other nodes, the second sparse parameter being initialized by the other nodes; and obtaining the initialized to-be-trained model according to the initialized each dense parameter, the first sparse parameter and the second sparse parameter. 7.The method of claim 5, wherein before inputting the training sample of the current node itself into the to-be-trained model, the method further comprises: receiving the initialized dense parameter of the to-be-trained model broadcasted by the master node when the current node is a slave node; receiving a sparse parameter initialized by other slave nodes and the master node, the sparse parameter being needed to be updated by the other slave nodes and the master node respectively; initializing a sparse parameter needed to be updated by the current node and sending the initialized sparse parameter to the other slave nodes and the master node; and obtaining the initialized to-be-trained model according to the initialized dense parameter and the initialized sparse parameter. 8.The method of claim 1, wherein sending the updated sparse parameter to the other nodes comprises: receiving an acquisition request sent by the other nodes; and sending the updated sparse parameter to the other nodes according to the acquisition request, wherein the acquisition request is determined by the other nodes according to a training sample needed to be inputted by the other nodes for continuing training the to-be-trained model. 9.The method of claim 1, wherein receiving the sparse parameter updated by the other nodes comprises: determining a training sample needed to be inputted for continuing training the to-be-trained model as a to-be-inputted sample; determining a sparse parameter corresponding to the to-be-inputted sample as a target sparse parameter according to a pre-stored corresponding relationship between training samples and sparse parameters; sending an acquisition request to the other nodes updating the target sparse parameter, and receiving the target sparse parameter updated by the other nodes. 10.The method of any one of claims 1-9, wherein the current node and the other nodes comprise a graphic processing unit (GPU), the GPU being used for calculating a gradient and updating a parameter, and the current node and the other nodes transmit data through a high-speed direct connection channel of the GPU. 11.A model training apparatus applied to nodes for training a to-be-trained model in a distributed manner, the to-be-trained model comprising dense parameters and sparse parameters, and sparse parameters needed to be updated by different nodes being not completely same, the apparatus comprising: ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ determining, by a determining module, a first gradient corresponding to the dense parameters and a second gradient corresponding to the sparse parameters based on an output of the to-be-trained model and a label of the training sample; sending, by a first interaction module, the first gradient and the second gradient to other nodes, and receiving a third gradient corresponding to the dense parameters and a fourth gradient corresponding to the sparse parameters sent by the other nodes, the third gradient and the fourth gradient being determined by the other nodes based on training samples of the other nodes and the to-be-trained model; updating, by a first updating module, the dense parameters of the to-be-trained model based on the first gradient and the third gradient; updating, by a second updating module, the sparse parameters of the current node based on at least part of the second gradient and / or at least part of the fourth gradient; sending, by a second interaction module, the updated sparse parameters to the other nodes, and receiving the sparse parameters updated by the other nodes; continuing, by a training module, training the to-be-trained model based on the updated dense parameters and sparse parameters until a model convergence condition is met.

12. The apparatus of claim 11, wherein the first interaction module is further configured to determine, from the second gradient, a second gradient corresponding to the sparse parameters of the other nodes as a target gradient, send the first gradient and the target gradient to the other nodes, and receive the fourth gradient corresponding to the sparse parameters of the current node and the third gradient corresponding to the dense parameters sent by the other nodes.

13. The apparatus of claim 11, wherein the first updating module is further configured to determine, for each dense parameter, a gradient corresponding to the dense parameter in the first gradient and the third gradient, average the gradients corresponding to the dense parameter, and update the dense parameter based on the average.

14. The apparatus of claim 11, wherein the second updating module is further configured to determine, for each sparse parameter of the current node, a gradient corresponding to the sparse parameter in the second gradient and the fourth gradient, average the gradients corresponding to the sparse parameter, and update the sparse parameter based on the average.

15. The apparatus of claim 11, wherein the nodes include a master node and a slave node.

16. The apparatus of claim 15, further comprising: The first initialization module is configured to, before inputting the training sample of the current node into the to-be-trained model, initialize each dense parameter of the to-be-trained model and a first sparse parameter required to be updated by the current node when the current node is the master node; send the initialized each dense parameter and the first sparse parameter to the other nodes; and receive second sparse parameters required to be updated by the other nodes from each sparse parameter of the to-be-trained model sent by the other nodes, the second sparse parameters being initialized by the other nodes. The initialized to-be-trained model is obtained according to the initialized each dense parameter, the first sparse parameter and the second sparse parameter.

17. The apparatus of claim 15, further comprising: The second initialization module is configured to, before inputting the training sample of the current node into the to-be-trained model, receive the dense parameter of the to-be-trained model initialized by the master node when the current node is the slave node; receive the sparse parameter initialized by the other slave nodes and required to be updated by the other slave nodes and the sparse parameter initialized by the master node and required to be updated by the master node sent by the other slave nodes and the master node respectively; initialize the sparse parameter required to be updated by the current node, send the initialized sparse parameter to the other slave nodes and the master node, and obtain the initialized to-be-trained model according to the initialized each dense parameter and the initialized sparse parameter.

18. The apparatus of claim 11, wherein the second interaction module is further configured to determine a training sample required to be input for continuing to train the to-be-trained model as a to-be-input sample, determine a sparse parameter corresponding to the to-be-input sample as a target sparse parameter according to a pre-stored corresponding relationship between the training sample and the sparse parameter, send an acquisition request to other nodes for updating the target sparse parameter, and receive the target sparse parameter updated by the other nodes sent by the other nodes.

19. A computer readable storage medium, the storage medium storing a computer program, the computer program being executed by a processor to implement the method of any one of claims 1-10.

20. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor implementing the method of any one of claims 1-10 when executing the program.

Images