Accelerator cluster and model training method

Abstract

A model training method is applied to an accelerator cluster in the field of artificial intelligence. According to the model training method, for a model with a data conversion layer, in the process that an accelerator executes model training, after the accelerator calculates the gradient of partial features, corresponding to training data, of the data conversion layer, the accelerator only transmits the gradient of the partial features obtained through calculation to other accelerators; and the gradients of all features in the whole data conversion layer are not transmitted, so that the transmission quantity of the gradients is reduced, and the communication efficiency between accelerators is improved. As the gradients of only part of the features in the data conversion layer are effective gradients calculated in the training process, the accelerator executes gradient aggregation based on the gradients of the features transmitted between the accelerators, and feature updating in the subsequent data conversion layer is not affected, namely normal training of the model is not affected.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to an accelerator cluster and model training method. Background Technology

[0002] With the development of AI technology, neural network models have been widely used in content recommendation, natural language processing, and computer vision. As these applications become more widespread, the scale of neural network models continues to increase. Some neural network models can have hundreds of layers, and the number of parameters can reach tens of millions.

[0003] Training large-scale neural network models often takes a long time. Therefore, distributed training is frequently used to accelerate the training process, which involves using multiple accelerators to perform model training in parallel. In a distributed training scenario, the same neural network model is deployed on multiple accelerators, and each accelerator uses different training data to train the same neural network model to calculate the gradients of the parameters in the neural network model. Furthermore, each accelerator transmits its calculated gradients to other accelerators, ensuring that each accelerator can aggregate the gradients calculated by multiple accelerators, and then update the parameters of the neural network model based on the aggregated gradients.

[0004] Because neural network models are large in scale, transmitting gradients in distributed scenarios often results in significant communication overhead, leading to prolonged communication times between accelerators and consequently impacting the training efficiency of the neural network model. Therefore, improving the communication efficiency between accelerators is a pressing issue that needs to be addressed. Summary of the Invention

[0005] This application provides an accelerator cluster and a model training method that can improve the communication efficiency between accelerators during model training.

[0006] Firstly, an accelerator cluster is provided, including a first accelerator and a second accelerator, both of which are equipped with training models. The training model is a model that needs to be optimized; that is, the training model needs to be collaboratively trained by the accelerators in the cluster to optimize its parameters. The training model includes a data transformation layer, which comprises a feature set. The data transformation layer is used to convert the training data input to the training model into corresponding features. For example, the data transformation layer of the training model includes an embedding table that records multiple features (feature set), with each feature represented as an embedding vector. Based on the feature set recorded in the embedding table, the data transformation layer can find the corresponding features in the feature set for the training data, thereby converting the training data into the corresponding features.

[0007] A first accelerator is used to train a training model using first training data, obtain a first gradient set, and send the first gradient set to a second accelerator. The first gradient set includes the gradients of a first subset of features in the feature set. Since the first training data used by the first accelerator does not correspond to all features in the data transformation layer, the first accelerator can train the training model by obtaining the gradients of the first subset of features in the data transformation layer.

[0008] The second accelerator is used to train the training model using the second training data, obtain a second gradient set, and send the second gradient set to the first accelerator. The second gradient set includes the gradients of the second part of the features in the feature set.

[0009] The first accelerator is also used to receive a second gradient set and update the corresponding features in the data transformation layer based on the first and second gradient sets. Specifically, based on the first and second gradient sets, the first accelerator performs gradient aggregation of the features to obtain aggregated gradients. These aggregated gradients are used to update the features corresponding to the data transformation layer. Furthermore, the gradient aggregation of features can be performed by averaging, weighted averaging, or maximizing.

[0010] In this scheme, for models with a data transformation layer, during model training by the accelerator, since only a subset of features corresponding to the training data in the data transformation layer will have gradients with valid values, the accelerator, after calculating the gradients, only transmits the gradients of a subset of features in the data transformation layer to other accelerators, rather than transmitting the gradients of all features in the entire data transformation layer. This reduces the amount of gradient transmission and improves the communication efficiency between accelerators. Furthermore, since only some features in the data transformation layer have valid gradients while others have invalid gradients, the accelerators can update the features in the data transformation layer based on the gradients of the subset of features transmitted between them, thus completing the update of the entire data transformation layer without affecting the normal training of the model.

[0011] In one possible implementation, the gradients in the first gradient set have a one-to-one correspondence with the corresponding features of the first training data in the data transformation layer; that is, the first gradient set only includes the gradients of the corresponding features of the first training data in the data transformation layer. Similarly, the gradients in the second gradient set have a one-to-one correspondence with the corresponding features of the second training data in the data transformation layer; that is, the second gradient set only includes the gradients of the corresponding features of the second training data in the data transformation layer. In this case, the first and second sets of features are often not the same.

[0012] In other words, the first accelerator only transmits the gradients of the features corresponding to the first training data to the second accelerator. The second accelerator also only transmits the gradients of the features corresponding to the second training data it uses to the first accelerator.

[0013] In this scheme, when the accelerator performs model training, after the accelerator matches the corresponding features in the data transformation layer for the training data and calculates the gradients of these features, it only transmits the gradients of the features corresponding to the training data to other accelerators, instead of transmitting the gradients of the features in the entire data transformation layer, thereby minimizing the amount of gradient transmission and improving the communication efficiency between accelerators.

[0014] In one possible implementation, both the first gradient set and the second gradient set include the gradients of the corresponding features of the first and second training data in the data transformation layer. That is, the first part of the features and the second part of the features mentioned above are the same. The gradients included in the first gradient set and the second gradient set correspond to the same part of the features.

[0015] In this scheme, by setting the accelerators to transmit the gradients of the same set of features (i.e., the features corresponding to all training data) when transmitting gradients, the accelerators only need to align and aggregate the gradients they transmit with the gradients they acquire to complete the gradient aggregation operation. This avoids the need for the accelerators to perform complex logical operations to align the feature gradients when the number of gradients transmitted by each accelerator is different, thereby reducing the burden on accelerators with poor logical computing capabilities and improving the efficiency of gradient aggregation.

[0016] In one possible implementation, the first accelerator is further configured to: receive feature identifiers, which indicate the features corresponding to the first and second training data in the data transformation layer. The first accelerator may, for example, receive the feature identifiers from a processor. That is, the first accelerator obtains the feature identifiers from the processor in advance to determine the features corresponding to all training data, and then determines which feature gradients should be included in the first gradient set.

[0017] In this scheme, the processor obtains the features corresponding to all the training data used by the accelerator and informs the accelerator of these features. This enables the accelerator to obtain the features that need to be transmitted gradients, ensuring that the normal model training of the accelerator is not affected. This effectively leverages the synergistic advantages of the processor and the accelerator, ensuring that the accelerator can smoothly perform gradient aggregation.

[0018] In one possible implementation, the feature identifier is also used to indicate the order of the features corresponding to the training data in the training dataset in the data transformation layer. The first accelerator is specifically used to send gradients from the first gradient set to the second accelerator in the ordered sequence.

[0019] In this scheme, by specifying the order in which the gradients transmitted by the accelerators are arranged, it can be ensured that each accelerator transmits the gradients of the features in the same order. This allows the accelerators to quickly align gradients under the same features by simply aggregating gradients at the same position according to the specified order when performing gradient aggregation. This effectively reduces the burden on accelerators with poor logical operation capabilities and improves the efficiency of gradient aggregation.

[0020] In one possible implementation, the processor that sends the feature identifier to the first accelerator runs a first thread corresponding to the first accelerator, and the first thread is used to interact with a second thread corresponding to the second accelerator to obtain the feature identifier.

[0021] In this scheme, the feature information corresponding to the training data processed by the accelerator is interacted with by the thread running on the processor corresponding to the accelerator. This allows the interaction of feature information to be executed synchronously with the model training process, ensuring that it will not affect the normal execution of model training by the accelerator. This effectively leverages the synergistic advantages of the processor and the accelerator, ensuring that the accelerator can smoothly perform gradient aggregation.

[0022] In one possible implementation, a first thread sends first information to a second thread and retrieves second information from the second thread. The first information indicates the features corresponding to the first training data in the data transformation layer, and the second information indicates the features corresponding to the second training data in the data transformation layer. Thus, based on the first and second information, the first thread can determine the features corresponding to all training data in the data transformation layer. That is, the first thread obtains the aforementioned feature identifiers based on the first and second information.

[0023] In one possible implementation, the first accelerator is specifically used to aggregate gradients corresponding to the same feature in the first gradient set and the second gradient set according to the feature identifier, and to update the corresponding feature in the data transformation layer using the aggregated gradients.

[0024] In one possible implementation, the first accelerator is specifically used to: train a training model using first training data to obtain training results. The training results include the gradients of all features in the data transformation layer. However, in the training results, only the gradients of the features corresponding to the first training data in the data transformation layer have valid values; the gradients of features in the data transformation layer that do not correspond to the first training data are invalid values ​​(e.g., gradient values ​​are 0). Thus, the first accelerator can be used to determine a first gradient set based on feature identifiers and the training results, that is, to select the gradients of a first subset of features in the training results based on the feature identifiers to form the first gradient set.

[0025] In one possible implementation, the accelerator cluster also includes a third accelerator, on which a trained model is deployed. The third accelerator is used to train the deployed model using third training data, obtaining a third gradient set, which is then sent to the first and second accelerators. The third gradient set includes the gradients of the third portion of the features in the feature set. The first accelerator is also used to send a first gradient set to the third accelerator, receive the third gradient set, and update the corresponding features in the data transformation layer based on the first, second, and third gradient sets.

[0026] In other words, besides the first and second accelerators, the accelerator cluster can include other accelerators. The interaction process between the first accelerator and other accelerators is similar to the interaction process between the first accelerator and the second accelerator. Furthermore, when updating the features corresponding to the data transformation layer, the first accelerator updates based on the gradient set obtained by all accelerators in the accelerator cluster.

[0027] Secondly, an accelerator is provided, including a processing unit and an interface; the processing unit is used to: train a training model using first training data to obtain a first gradient set; the training model includes a data transformation layer, the data transformation layer includes a feature set; the first gradient set includes the gradients of a first part of the features in the feature set; send the first gradient set to a second accelerator through the interface; receive a second gradient set through the interface, the second gradient set being obtained by the second accelerator training the training model using second training data, the second gradient set including the gradients of a second part of the features in the feature set; and update the corresponding features in the data transformation layer based on the first gradient set and the second gradient set.

[0028] In one possible implementation, the gradients contained in the first gradient set have a one-to-one correspondence with the corresponding features of the first training data in the data transformation layer; the gradients contained in the second gradient set have a one-to-one correspondence with the corresponding features of the second training data in the data transformation layer.

[0029] In one possible implementation, both the first gradient set and the second gradient set include the gradients of the corresponding features of the first training data and the second training data in the data transformation layer.

[0030] In one possible implementation, the processing unit is further configured to: receive feature identifiers via an interface, the feature identifiers being used to indicate the features corresponding to the first training data and the second training data in the data transformation layer.

[0031] In one possible implementation, the feature identifier is also used to indicate the order of the corresponding features in the data transformation layer of the training data in the training dataset; the processing unit is specifically used to send the gradients in the first gradient set to the second accelerator in the order of arrangement via an interface.

[0032] Thirdly, a computing device is provided, including a processor and an accelerator implemented in either of the second aspects, wherein the processor is used to schedule the accelerator to train a training model.

[0033] In one possible implementation, the processor is further configured to send a feature identifier to the accelerator, the feature identifier indicating the features corresponding to the first training data and the second training data in the data transformation layer; both the first gradient set sent by the accelerator and the second gradient set received by the accelerator include the gradients of the features indicated by the feature identifier.

[0034] Fourthly, a model training method is provided, comprising: a first accelerator training a training model using first training data to obtain a first gradient set; the training model including a data transformation layer, the data transformation layer including a feature set; the first gradient set including the gradients of a first part of the features in the feature set; sending the first gradient set to a second accelerator; the second accelerator training the training model using second training data to obtain a second gradient set; the second gradient set including the gradients of a second part of the features in the feature set; sending the second gradient set to the first accelerator; the first accelerator receiving the second gradient set and updating the corresponding features in the data transformation layer based on the first gradient set and the second gradient set.

[0035] In one possible implementation, the gradients contained in the first gradient set have a one-to-one correspondence with the corresponding features of the first training data in the data transformation layer; the gradients contained in the second gradient set have a one-to-one correspondence with the corresponding features of the second training data in the data transformation layer.

[0036] In one possible implementation, both the first gradient set and the second gradient set include the gradients of the corresponding features of the first training data and the second training data in the data transformation layer.

[0037] In one possible implementation, the model training method further includes: a first accelerator receiving feature identifiers, which are used to indicate the features corresponding to the first training data and the second training data in the data transformation layer.

[0038] In one possible implementation, the feature identifier is also used to indicate the order of the features corresponding to the training data in the training dataset in the data transformation layer; the first accelerator sends gradients from the first gradient set to the second accelerator in the order of arrangement.

[0039] In one possible implementation, the first accelerator aggregates gradients from the first gradient set and the second gradient set that correspond to the same feature based on the feature identifier, and uses the aggregated gradients to update the corresponding features in the data transformation layer.

[0040] In one possible implementation, the first accelerator trains the training model using the first training data to obtain the training result; the first accelerator determines the first gradient set based on the feature identifier and the training result.

[0041] In one possible implementation, the third accelerator trains the deployed training model using third training data to obtain a third gradient set; the training model is deployed on the third accelerator; the third gradient set includes the gradients of the third part of the features in the feature set; the third accelerator sends the third gradient set to the first accelerator and the second accelerator; the first accelerator sends the first gradient set to the third accelerator; the first accelerator receives the third gradient set and updates the corresponding features in the data transformation layer based on the first gradient set, the second gradient set, and the third gradient set.

[0042] Fifthly, a computer-readable storage medium is provided that stores instructions which, when executed on a computer, cause the computer to perform the methods of any of the above aspects.

[0043] Sixthly, a computer program product containing instructions is provided, which, when executed on a computer, enable the computer to perform the methods described above.

[0044] In a seventh aspect, a chip system is provided, the chip system including a processor and a communication interface for communicating with a module other than the chip, the processor for running computer programs or instructions such that an apparatus on which the chip system is mounted can perform the methods of any of the above aspects.

[0045] Eighthly, a computing device is provided, the computing device including a processor, an accelerator cluster and a memory, the memory storing code, the processor being configured to execute the code, and when the code is executed, the computing device being used to implement the operational steps of the method of the fourth aspect described above.

[0046] A ninth aspect provides a computing device cluster, comprising at least one computing device, wherein any one computing device is used to run a computer program or instructions, such that the computing device cluster can perform the method described in the fourth aspect. Alternatively, some or all of the computing devices are used together to run a computer program or instructions, such that the computing device cluster can perform the method described in the fourth aspect.

[0047] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of gradient synchronization for an embedded table in a related technology;

[0049] Figure 2 A schematic diagram of an accelerator cluster provided for this application;

[0050] Figure 3 A flowchart illustrating a model training method provided in this application;

[0051] Figure 4 A schematic diagram of gradient synchronization of an embedded table provided in this application;

[0052] Figure 5 A schematic diagram illustrating gradient aggregation performed by an accelerator, as provided in this application;

[0053] Figure 6 A schematic diagram illustrating the interaction process between an accelerator and a thread running on a processor, provided for the purposes of this application;

[0054] Figure 7 This application provides a schematic diagram illustrating how a thread running on a processor and an accelerator use interactive indexing to indicate features that need to be transmitted gradients.

[0055] Figure 8 This application provides a schematic diagram of the process of a processor and an accelerator working together.

[0056] Figure 9 Another schematic diagram illustrating the collaborative operation of a processor and accelerator provided for this application;

[0057] Figure 10 A schematic diagram illustrating the gradient of an accelerator synchronously embedded in-table feature, as provided in this application;

[0058] Figure 11 A schematic diagram of the structure of a model training system provided in this application;

[0059] Figure 12 A schematic diagram of the structure of a computing device provided in this application;

[0060] Figure 13 This application provides a schematic diagram of the structure of a computing device cluster;

[0061] Figure 14 This application provides a schematic diagram of another computing device cluster structure.

[0062] Figure 15 This application provides a schematic diagram of the structure of a chip;

[0063] Figure 16 This is a schematic diagram of the structure of a computer-readable storage medium provided in this application. Detailed Implementation

[0064] The embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are merely some, and not all, of the embodiments of this application. Those skilled in the art will recognize that, with the emergence of new application scenarios, the technical solutions provided by this application are also applicable to similar technical problems.

[0065] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such descriptions can be used interchangeably where appropriate to allow embodiments to be implemented in a sequence other than that illustrated or described in this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved. The division of units in this application is a logical division. In practical applications, there may be other division methods. For example, multiple units may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the shown or discussed mutual coupling, direct coupling, or communication connection may be through some interface, and the indirect coupling or communication connection between units may be electrical or other similar forms, none of which are limited in this application. Furthermore, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed among multiple circuit units. Some or all of the units can be selected to achieve the purpose of the solution in this application according to actual needs.

[0066] First, some technical terms used in this application will be introduced.

[0067] (1) Neural Network

[0068] Neural networks can be composed of neural units, which can refer to units represented by x. s The arithmetic unit that takes input data as input can output the following:

[0069]

[0070] Where s = 1, 2, ..., n, n is a natural number greater than 1, and Ws is x sThe weights are denoted by b, the bias of the neural unit is denoted by f, and the activation functions are denoted by f, which introduce nonlinear characteristics into the neural network to convert the input signal into the output signal. The output signal of this activation function can be used as the input to the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting multiple individual neural units, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be a region composed of several neural units.

[0071] (2) Transformer network

[0072] The Transformer network is a neural network model based on the self-attention mechanism, used to process sequential data. Compared to traditional recurrent neural network models, the Transformer network has better parallel performance and shorter training time, and therefore has been widely used in the field of natural language processing.

[0073] The Transformer network comprises multiple encoder and decoder layers, with each layer consisting of multiple attention mechanism modules and feedforward neural network modules. The encoder encodes the input sequence into a high-dimensional feature vector representation, while the decoder decodes this vector representation into the target sequence. The Transformer network also utilizes techniques such as residual connections and layer normalization to accelerate model convergence and improve model performance.

[0074] Specifically, the core of the Transformer network is the self-attention mechanism, which assigns a weight to each position in the input sequence and then outputs these weighted position vectors. Generally, the computation process of the self-attention mechanism includes three steps: 1. Calculate the attention weights: calculate the attention weights between each position and other positions, i.e., the importance of each position to the others; 2. Calculate the weighted sum: multiply each position vector by its attention weight and then sum them to obtain a weighted sum vector; 3. Perform a linear transformation: apply a linear transformation to the weighted sum vector to obtain the final output vector.

[0075] (3) Content recommendation model

[0076] Content recommendation models are a type of neural network that uses machine learning algorithms to analyze and learn based on a user's historical browsing history and the user's own characteristics. This determines the probability (i.e., click-through rate) of a user accessing specific content, and then recommends content that the user is likely to be interested in based on the probability of accessing various types of content.

[0077] (4) Embedded Table

[0078] In deep learning, an embedding table is a table in a neural network model used to represent categorical entities (such as products, applications, words, etc.). Essentially, it is used to transform input data into unique dense vectors.

[0079] Taking content recommendation as an example, when performing content recommendations, it's often necessary to predict user-preferred products based on product categories, and then recommend those products to the user. When product categories are used as input to the content recommendation model, the model typically uses an embedding table to find corresponding features for the input product categories, thus transforming the product categories from a sparse vector into a dense vector (i.e., features). The embedding table includes multiple features, each of which can be represented as an embedding vector. Furthermore, different features in the embedding table often correspond to different input data. Therefore, based on the embedding table, different input data can be converted into different features. For example, assuming the input data is product categories, and the input data includes "0" and "1," representing two different product categories respectively, then based on the embedding table, the "0" in the input data can be converted into a corresponding 128-dimensional vector, and the "1" in the input data can be converted into another corresponding 128-dimensional vector.

[0080] Taking natural language processing as an example, the input data of a natural language processing model is text. In order for the natural language processing model to process text effectively, it is usually necessary to convert the words in the text into dense vectors through embedding tables.

[0081] (5) Large Language Model (LLM)

[0082] Large language models are deep learning models trained on massive amounts of text data that can generate natural language text or understand the meaning of language text. Large language models can handle various natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence.

[0083] Specifically, large language models are a technology that has emerged in recent years. Because large language models undergo meticulous data engineering and training processes, their parameters have learned a wealth of existing natural language processing knowledge. This knowledge can now replace humans in many language-related tasks, such as having large language models write code or perform text summarization.

[0084] (6) Token

[0085] A lexical is the smallest semantic unit in natural language processing, typically derived from the decomposition of long texts such as words, sentences, paragraphs, and articles. Generally, lexical units can be composed of letters, words, or symbols; for example, they can be composed of one or more letters or one or more words. Simply put, a lexical unit can be understood as a "fragment" of a word or phrase—the smallest indivisible semantic unit. For example, "waterfall" can be broken down into two lexical units: "water" and "fall." Additionally, punctuation marks can also be decomposed into lexical units because they affect the semantic understanding of the entire text. For example, "I don't know." can be decomposed into five lexical units: "I," "don," "'t," "know," and ".".

[0086] (7) Loss Function

[0087] During neural network training, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value and update the weight vector of each layer based on the difference (usually preceded by initialization, where parameters are pre-configured for each layer). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value, and this process continues until the neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are crucial equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference between the predicted and target values, thus training the neural network becomes a process of minimizing this loss.

[0088] (8) Back propagation (BP) algorithm

[0089] Neural networks can employ backpropagation algorithms to refine the parameters of the initial prediction model during training, thereby minimizing the error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then used to update the parameters of the initial prediction model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process designed to obtain the optimal parameters of the prediction model, such as the weight matrix.

[0090] Specifically, during model training, the backpropagation algorithm is typically used to calculate the gradients of each node in the model. This allows for the adjustment of node weights based on the gradients of each node, thereby minimizing the model's loss function value. The gradient represents the rate of change of a function at a given point. Furthermore, the gradient of each node in the model can be determined by calculating its partial derivatives.

[0091] (9) Gradient descent method

[0092] Gradient descent is a first-order optimization algorithm commonly used in machine learning to recursively approximate a minimum-bias prediction model. To find a local minimum of a function using gradient descent, iterative searches must be performed at points a predetermined step distance away in the opposite direction of the gradient (or approximate gradient) at the current point on the function. Gradient descent is one of the most frequently used methods for solving the prediction model parameters of machine learning algorithms, i.e., unconstrained optimization problems.

[0093] Specifically, when finding the minimum value of the loss function, gradient descent can be used to iteratively solve the problem step by step, obtaining the minimized loss function and the predicted model parameter values. Conversely, if we need to find the maximum value of the loss function, we need to use gradient ascent iteratively.

[0094] The applicant's research revealed that in current distributed training scenarios, multiple accelerators use different training data to train the same neural network model in order to calculate the gradients of the parameters in the neural network model. Furthermore, each accelerator transmits its calculated gradients to other accelerators, ensuring that each accelerator can aggregate the gradients calculated by multiple accelerators, and then update the parameters of the neural network model based on the aggregated gradients.

[0095] For neural network models with embedding tables (such as content recommendation models or large language models in natural language processing scenarios), these models often need to calculate the gradients of features in the embedding table in addition to the gradients of the weight parameters in the model, so as to update the features based on the gradients. Neural network models with embedding tables typically consist of sparse layers and dense layers. The sparse layers transform the input data into corresponding features through embedding table lookup operations. The dense layers receive the features passed from the sparse layers, and through feature interaction and other operations, finally obtain the output results, which are then used to calculate the loss function and, based on the loss function, calculate the gradients of the model parameters.

[0096] Generally, in related technologies, to facilitate the rapid alignment of gradients from various accelerators (i.e., aligning gradients corresponding to the same feature) during gradient aggregation for accelerators with limited logical computation capabilities, accelerators often exchange gradients of features across the entire embedding table. This ensures that the gradient scale of a feature received by one accelerator from any other accelerator is the same. In this way, by aggregating the gradients of the received features with the gradients of the features it has computed, the accelerator can obtain the gradients of all features in the entire embedding table, facilitating feature updates.

[0097] For example, please refer to Figure 1 , Figure 1 This is a schematic diagram of gradient synchronization using an embedded table in a related technology. For example... Figure 1 As shown, assume that accelerator 1 and accelerator 2 are deployed with the same embedding table. During one round of model training iteration, the input data on accelerator 1 finds six corresponding features in the embedding table. Therefore, accelerator 1 will calculate the gradients of these six features during subsequent model training, and these gradient values ​​are valid. For other features in the embedding table that do not correspond to the input data, accelerator 1 will also calculate gradients for these features, but the gradients calculated for other features will be invalid values, such as 0. The input data on accelerator 2 finds five corresponding features in the embedding table, and accelerator 2 will calculate the gradients of these five features during subsequent model training. Similarly, for other features in the embedding table that do not correspond to the input data, accelerator 2 will also calculate gradients for these features, but the gradients calculated for other features will be invalid values, such as 0.

[0098] Although accelerators 1 and 2 only compute gradients with valid values ​​for a subset of features in the embedding table, during embedding table synchronization, accelerators 1 and 2 synchronize the gradients of all features within the embedding table. That is, they transmit the gradient of each feature in the embedding table sequentially according to its order of arrangement. In this way, accelerators 1 and 2 can obtain the gradients of all features in the entire embedding table by aggregating the gradients of all received features with the gradients of all features in their own embedding table, thus facilitating feature updates.

[0099] However, in some scenarios, such as content recommendation, the size of the embedding table in the content recommendation model is quite large due to the large number of categories of content to be recommended (e.g., there may be hundreds of millions of product categories). During one iteration of training for the content recommendation model, the features matched by the input training data in the embedding table are only a small subset of those features. Therefore, only these matching features will have valid gradients calculated. For features that do not match the training data, their gradients are typically set to 0 (i.e., invalid values). Consequently, with a large embedding table and only a small subset of features having valid gradients, the accelerator transmitting the gradients of all features in the entire embedding table to other accelerators results in a large amount of redundant communication, leading to low communication efficiency.

[0100] In view of this, this application provides a data transmission method for models with a data transformation layer. During model training by the accelerator, since only a subset of features corresponding to the training data in the data transformation layer will have gradients with valid values, the accelerator, after completing gradient calculation, only transmits the gradients of a subset of features in the data transformation layer to other accelerators, rather than transmitting the gradients of all features in the entire data transformation layer. This reduces the amount of gradient transmission and improves the communication efficiency between accelerators. Furthermore, since only a subset of features in the data transformation layer have valid gradients while the gradients of other features are invalid, the accelerators can update the features in the data transformation layer based on the gradients of the subset of features transmitted between them, thus completing the update of the entire data transformation layer without affecting the normal training of the model.

[0101] Please see Figure 2 , Figure 2 This is a schematic diagram of an accelerator cluster provided in this application. Figure 2 As shown, this accelerator cluster includes multiple accelerators (e.g., Figure 2 Accelerators 101 and 102 in the example. Multiple accelerators can be integrated with one or more processors (e.g., ...). Figure 2The processor 20 is connected, and one or more processors may be connected to a data storage system 30. Figure 2 Taking an accelerator cluster consisting of two accelerators as an example, in practical applications, an accelerator cluster can include two or more accelerators.

[0102] Accelerators 101 and 102 are used to run the training model to execute the model training tasks issued by processor 20, thereby achieving parallel training of the first model. Processor 20 is used to obtain model training instructions and schedule the corresponding accelerators to execute the model training tasks based on these instructions. It should be noted that... Figure 2 In the original text, both accelerators 101 and 102 are connected to processor 20 to receive data sent by processor 20. However, in practical applications, an accelerator can actually correspond to a thread running on the processor; that is, different accelerators are managed by different threads. The threads corresponding to different accelerators can run on the same processor or on different processors; this application does not make any specific limitations on this.

[0103] In addition, the data storage system 30 is used to store data such as training data or program code. Specifically, when the data storage system 30 stores training data, the processor 20 retrieves the training data from the data storage system 30 and transmits the training data to the corresponding accelerator so that the accelerator can perform model training based on the input training data.

[0104] Optionally, for persistent data storage, the data storage system 30 may be located on a different node from the processor 20, exchanging data with the processor 20 via a network. Alternatively, the data storage system 30 may be located inside the physical host where the processor 20 resides, such as exchanging data with the processor 20 via a bus. In this case, the data storage system 30 behaves as a hard disk.

[0105] The accelerator cluster provided in this application can be applied to distributed training scenarios for neural network models with embedding tables, such as training scenarios for content recommendation models or natural language processing models.

[0106] Please see Figure 3 , Figure 3 This is a flowchart illustrating a model training method provided in this application. Figure 3As shown, the model training method provided in this application is applied to an accelerator cluster, which includes a first accelerator and a second accelerator. Training models are deployed on both the first and second accelerators, and different accelerators are used to train the training models based on different training data. Furthermore, the training model includes a data transformation layer, which includes a feature set. This data transformation layer is used to convert the input data of the training model into features in the feature set. For example, the data transformation layer includes an embedding table that records multiple features, each represented as an embedding vector. Based on the multiple features recorded in the embedding table (i.e., the feature set), the data transformation layer can find the corresponding features among the multiple features for the training data, thereby converting the training data into the corresponding features.

[0107] In a model training system, an accelerator can specifically be hardware used for training or inference of an artificial intelligence model, such as a graphics processing unit (GPU), a tensor processing unit (TPU), or a neural network processing unit (NPU).

[0108] like Figure 3 As shown, the model training method specifically includes the following steps 301-305.

[0109] Step 301: The first accelerator trains the training model using the first training data to obtain a first gradient set; the first gradient set includes the gradients of the first part of the features in the feature set.

[0110] In this application, the first accelerator refers to any one of the accelerators in the accelerator cluster. That is, this application uses the first accelerator as an example to describe the workflow executed by multiple accelerators in the accelerator cluster. For other accelerators besides the first accelerator, they may also execute a similar workflow to that of the first accelerator described below, which will not be elaborated here.

[0111] In this application, the training model is the model that needs to be optimized; that is, the training model needs to be collaboratively trained by accelerators in the accelerator cluster to optimize the parameters in the training model. Optionally, the training model can be applied to content recommendation scenarios or natural language processing scenarios. In content recommendation scenarios, the input data of the training model is usually category information for entities such as users, products, or applications. Therefore, the embedding table in the training model is used to convert entity categories such as user categories, product categories, and application categories into features. The features included in the same embedding table are usually fixed-dimensional vectors (also called embedding vectors), such as 128-dimensional vectors. In natural language processing scenarios, the input data of the training model is usually text data. Therefore, the embedding table in the training model is used to convert words in the text into features. Of course, the training model in this application can also be applied to other scenarios, and no specific limitation is made here. During one round of iterative training of the training model, the first accelerator obtains the first training data as input, where the first training data can be a batch of training data in the current round of iterative training. Based on the initial training data, the first accelerator first performs the forward propagation process of the training model. This involves finding the corresponding features in the embedding table of the sparse layer and performing feature processing on the features in the dense layer, thereby obtaining the output of the training model. Then, based on the output of the training model, the first accelerator calculates the loss function value for the current round and performs backpropagation based on the loss function value to calculate the gradients of the parameters in the dense layer and the gradients of the features in the embedding table of the sparse layer.

[0112] Because the first training data contains limited content, while the embedding table of the data transformation layer of the training model records a large number of features, the first training data often only corresponds to a subset of those features in the embedding table. When the first training data only matches a subset of features in the embedding table, the first accelerator, during the training process of the training model, will actually only calculate gradients with valid values ​​for a subset of those features. That is, only features in the embedding table that match the first training data can have gradients with valid values ​​calculated. Therefore, based on the first training data, the first accelerator calculates the gradients of a subset of features in the embedding table. For features in the embedding table that do not correspond to the first training data, the gradients of these features are invalid values, for example, the gradient value is set to 0.

[0113] Since the first training data used by the first accelerator does not correspond to all features in the data transformation layer, the first accelerator trains the training model to obtain a first gradient set, which includes the gradients of the first part of the features in the data transformation layer.

[0114] Step 302: The first accelerator sends the first gradient set to the second accelerator.

[0115] After calculating the first set of gradients, the first accelerator transmits the calculated first set of gradients to the other accelerators in the accelerator set, so that the other accelerators can obtain the gradients calculated on all accelerators. Furthermore, during the gradient transmission process, the first accelerator no longer transmits the gradients of all features in the embedding table, but only the gradients of some features in the embedding table, thereby reducing the amount of gradient transmission.

[0116] Step 303: The second accelerator uses the second training data to train the training model and obtains the second gradient set.

[0117] The second gradient set includes the gradients of the second part of the features in the feature set. The first training data and the second training data are different training data. The way the second accelerator trains the training model using the second training data is similar to the way the first accelerator trains the training model using the first training data. For details, please refer to step 301 above, which will not be repeated here.

[0118] Step 304: The second accelerator sends the second gradient set to the first accelerator.

[0119] Similarly, while the first accelerator is training the model, the second accelerator is also training the model based on the second training data and calculating the gradients of the features in the embedding table corresponding to the second training data. Therefore, the second accelerator also transmits the calculated gradients, allowing the first accelerator to obtain the second gradient set. This second gradient set also includes only the gradients of a subset of the features in the data transformation layer (i.e., the gradients of the second subset of features mentioned above), rather than the gradients of all features in the data transformation layer.

[0120] For example, please refer to Figure 4 , Figure 4 This is a schematic diagram of gradient synchronization for an embedded table provided in this application. Figure 4 As shown, assume that accelerator 1 and accelerator 2 are deployed with the same embedding table. During one round of model iteration training, the training data on accelerator 1 finds the corresponding 6 features in the embedding table and calculates the gradients of these 6 features. The training data on accelerator 2 finds the corresponding 5 features in the embedding table and calculates the gradients of these 5 features. Thus, during embedding table synchronization, accelerator 1 and accelerator 2 synchronize the gradients of the features corresponding to the training data in the embedding table. Therefore, assuming that the embedding table includes a total of 24 features, accelerator 1 can actually transmit only the gradients of 6 features, and accelerator 2 can transmit only the gradients of 5 features, effectively reducing the amount of gradient data transmitted.

[0121] Step 305: The first accelerator updates the corresponding features in the data transformation layer based on the first gradient set and the second gradient set.

[0122] After obtaining the gradients of the features calculated by the second accelerator, the first accelerator can perform gradient aggregation of the features based on the gradients of the first part of the features it calculated and the gradients of the second part of the features obtained from the second accelerator. During the gradient aggregation process, the first accelerator aggregates the gradients calculated by different accelerators that correspond to the same feature, thereby obtaining the aggregated gradient, and uses this aggregated gradient to update the corresponding features in the data transformation layer.

[0123] It should be noted that since the features corresponding to the training data input to different accelerators are often not exactly the same, it is possible that some accelerators compute valid gradients for a certain feature (i.e., the gradients computed for a certain feature are valid values), while other accelerators do not compute valid gradients for that feature (i.e., the gradients computed for that feature are invalid values). In this case, when performing gradient aggregation, if some accelerators do not compute valid gradients for a certain feature, the gradients of those accelerators for that feature can be set to 0, so that the gradients computed by each accelerator for that feature can be obtained.

[0124] For example, please refer to Figure 5 , Figure 5 This is a schematic diagram illustrating gradient aggregation performed by an accelerator, as provided in this application. Figure 5 As shown, accelerator 1 calculates the gradients of features 1-5 and feature 8, and transmits these gradients to accelerator 2. Accelerator 2 calculates the gradients of features 6-10 and transmits them to accelerator 1. Clearly, both accelerator 1 and accelerator 2 actually calculate the gradient of feature 8. For feature 1, accelerator 1 calculates the gradients of features 1-5, while accelerator 2 does not calculate any valid gradients for features 1-5. Therefore, when performing gradient aggregation of features 1-5, the gradients actually calculated by accelerator 2 for features 1-5 can be set to 0. Similarly, when accelerator 2 performs gradient aggregation of features 6-7 and features 9-10, the gradients actually calculated by accelerator 1 for features 6-7 and features 9-10 can both be set to 0.

[0125] The method for aggregating the gradients of features can be to calculate the average, the weighted average, or the maximum value. The specific method can be determined by the training method of the training model, and no specific limitation is made here.

[0126] In this application, the specific contents of the first gradient set and the second gradient set that are interacted between accelerators can be implemented in two ways.

[0127] In implementation method 1, the first gradient set transmitted by the first accelerator includes only the gradients of the corresponding features of the first training data in the data transformation layer. The second gradient set includes only the gradients of the corresponding features of the second training data in the data transformation layer.

[0128] In other words, the first accelerator only transmits the gradients of the features corresponding to the first training data to the second accelerator. The second accelerator also only transmits the gradients of the features corresponding to the second training data.

[0129] In implementation method 1, when the accelerator performs model training, after the accelerator matches the corresponding features in the data transformation layer for the training data and calculates the gradients of these features, it only transmits the gradients of the features corresponding to the training data to other accelerators, instead of transmitting the gradients of the features in the entire data transformation layer, thereby minimizing the amount of gradient transmission and improving the communication efficiency between accelerators.

[0130] In implementation method 2, both the first gradient set and the second gradient set include the gradients of the corresponding features of the first and second training data in the data transformation layer. That is, the first part of the features and the second part of the features are the same. The gradients included in the first gradient set and the second gradient set correspond to the same set of features.

[0131] In other words, the first accelerator transmits the gradients of the features corresponding to the first and second training data to the second accelerator. The second accelerator also transmits the gradients of the features corresponding to the first and second training data to the outside world.

[0132] In implementation method 2, by setting the accelerators to transmit the gradients of the same batch of features (i.e., the features corresponding to all training data) when transmitting gradients, the accelerators only need to align and aggregate the gradients they transmit with the gradients they acquire to complete the gradient aggregation operation. This avoids the need for the accelerators to perform complicated logical operations to align the gradients of features when the number of gradients transmitted by each accelerator is different, thereby reducing the burden on accelerators with poor logical computing capabilities and improving the efficiency of gradient aggregation.

[0133] Optionally, the accelerator cluster also includes a third accelerator, on which the trained model is deployed. The third accelerator is used to train the deployed model using third training data, obtaining a third gradient set, which is then sent to the first and second accelerators. The third gradient set includes the gradients of the third part of the features in the feature set. The first accelerator is also used to send a first gradient set to the third accelerator, receive the third gradient set, and update the corresponding features in the data transformation layer based on the first, second, and third gradient sets.

[0134] In other words, besides the first and second accelerators, the accelerator cluster can include other accelerators. The interaction process between the first accelerator and other accelerators is similar to the interaction process between the first accelerator and the second accelerator. Furthermore, when updating the features corresponding to the data transformation layer, the first accelerator updates based on the gradient set obtained by all accelerators in the accelerator cluster.

[0135] It should be noted that, in addition to the first, second, and third accelerators mentioned above, an accelerator cluster may include more other accelerators. That is, an accelerator cluster may contain two or more accelerators, and the accelerators in the cluster are used to collaboratively train the training model based on different training data.

[0136] In this application, the accelerator cluster can specifically be a decentralized system, meaning that multiple accelerators in the cluster are located on distributed nodes, and there is no central node coordinating the multiple accelerators. Therefore, when the first accelerator transmits the gradient of a partial feature to other accelerators in the cluster, it can transmit the gradient of that partial feature to each accelerator in the cluster. In the case where the multiple accelerators in the cluster form a ring structure, each accelerator is only directly connected to the two accelerators before and after it. Therefore, the first accelerator can send the gradient of a partial feature to adjacent accelerators, ultimately enabling the first gradient set to be transmitted along the ring structure.

[0137] In general, the way the first accelerator transmits gradients to other accelerators is related to the topology of the multiple accelerators. This application does not limit the way the first accelerator transmits gradients to other accelerators, as long as it ensures that each accelerator in the accelerator cluster can aggregate the gradients calculated by multiple accelerators.

[0138] Understandably, since each accelerator in the accelerator cluster transmits the gradients of some calculated features to other accelerators, in order to ensure that the first accelerator can know which features the gradients transmitted by other accelerators correspond to, this application provides multiple implementation methods that enable the first accelerator to determine the features corresponding to the acquired gradients.

[0139] In the first implementation method, the threads corresponding to the accelerator will exchange feature information so that the accelerator can learn from the corresponding thread the features corresponding to the gradient it has obtained.

[0140] For example, the aforementioned accelerator cluster also includes one or more processors for running threads corresponding to the accelerators. In the accelerator cluster, each accelerator has a corresponding thread, and different accelerators correspond to different threads. Threads corresponding to different accelerators can run on the same processor or on different processors. The multiple processors in the accelerator cluster can be, for example, hardware with computing capabilities such as a Central Processing Unit (CPU).

[0141] In this application, the thread corresponding to the first accelerator is specifically the first thread, and the thread corresponding to the second accelerator is specifically the second thread.

[0142] Please see Figure 6 , Figure 6 This application provides a schematic diagram illustrating the interaction process between an accelerator and a thread running on a processor. For example... Figure 6 As shown, during the training process of the training model on the first accelerator, the first thread sends first information to the second thread. This first information indicates the features corresponding to the first training data. Since the first training data is usually passed to the first accelerator by the first thread when issuing the model training task, the first thread can actually obtain the first training data and determine the features in the embedding table that match the first training data based on it. Therefore, the first thread can send the first information to the second thread to inform it of the features corresponding to the first training data on the first accelerator. In this way, the second thread can determine which features the gradients subsequently calculated by the first accelerator correspond to based on the first information.

[0143] Furthermore, the first thread can obtain second information from the second thread, which indicates the features corresponding to the second training data. Similarly, the second thread can also learn the features corresponding to the training data processed by its own second accelerator, and then interact with these features among multiple threads.

[0144] It should be noted that when the accelerator cluster includes other accelerators besides the first and second accelerators, each accelerator in the cluster corresponds to a thread, and the threads corresponding to the other accelerators in the cluster constitute a thread set. Thus, the first thread can obtain feature-related information from each thread in the thread set, and this information constitutes the aforementioned second information. That is, the aforementioned second information can be used to indicate the actual features corresponding to the training data processed by each accelerator in the cluster other than the first accelerator. For example, in the case where the accelerator cluster includes a first accelerator, a second accelerator, and a third accelerator, the aforementioned second information can indicate the features corresponding to the training data processed by the second accelerator and the features corresponding to the training data processed by the third accelerator.

[0145] Then, based on the first and second information, the first thread sends feature identifiers to the first accelerator. These feature identifiers indicate the features corresponding to the training data in the training dataset at the data transformation layer. In other words, the first accelerator obtains the feature identifiers from the processor in advance to determine the features corresponding to all training data, and then determines which feature gradients should be included in the first gradient set.

[0146] In this way, the first accelerator can determine which features the gradients obtained from other accelerators correspond to based on the acquired feature identifiers. That is, when performing gradient aggregation of features, the first accelerator can aggregate gradients corresponding to the same feature based on the feature identifiers.

[0147] In this scheme, the feature information corresponding to the training data processed by the accelerator is interacted with by the thread running on the processor corresponding to the accelerator. This allows the interaction of feature information to be executed synchronously with the model training process, ensuring that it will not affect the normal execution of model training by the accelerator. This effectively leverages the synergistic advantages of the processor and the accelerator, ensuring that the accelerator can smoothly perform gradient aggregation.

[0148] Optionally, in order to facilitate the rapid alignment of gradients transmitted by each accelerator, the feature identifier sent by the first processor is also used to indicate the order of the corresponding features in the data transformation layer of the training data in the training dataset.

[0149] Furthermore, when transmitting the first set of gradients, the first accelerator transmits the gradients in the first set of gradients sequentially according to their arrangement order. Similarly, when the accelerators in the accelerator set transmit the gradients of features outward, they also perform the gradient transmission in the same arrangement order, thereby ensuring that the order of gradients received by the first accelerator from each accelerator in the accelerator set is the same as the order of gradients transmitted outward by the first accelerator.

[0150] In this way, the first accelerator can align the gradients of features according to their order of arrangement when performing gradient aggregation. That is, the first accelerator aggregates the gradients corresponding to the same feature in the first and second gradient sets based on the feature identifier, and uses the aggregated gradients to update the corresponding features in the data transformation layer.

[0151] For example, assuming the first part of the features includes features 1-5 and feature 8, and the second part includes features 6-10, then the arrangement order indicated in the feature identifiers could mean that the gradients of the features are arranged sequentially according to the order of features 1-10. Furthermore, when the first accelerator receives the 10 gradients transmitted by the second accelerator, the first accelerator can perform gradient aggregation according to the sequential arrangement of these 10 gradients corresponding to features 1-10, thereby completing the gradient aggregation of the features.

[0152] In other words, by having the threads running on the processor specify the order in which the gradients transmitted by the accelerators are arranged, it can be ensured that each accelerator transmits the gradients of the features in the same order. This allows the accelerators to quickly align gradients for the same features by simply aggregating gradients at the same location in the specified order when performing gradient aggregation. This effectively reduces the burden on accelerators with poor logic processing capabilities and improves the efficiency of gradient aggregation.

[0153] For example, when the first information includes the index of the feature corresponding to the first training data and the second information includes the index of the feature corresponding to the second training data, the first thread can perform feature deduplication based on the first and second information to obtain the aforementioned feature identifier. The feature identifier includes multiple indices arranged sequentially, each index including the index of the feature corresponding to all training data in the training set. Therefore, based on the multiple indices arranged sequentially, the accelerator can determine the order of gradients of the features, and thus can implement gradient transfer according to the order of the indices.

[0154] In other words, when each feature is uniquely identified by an index in the embedding table, the first thread carries multiple indexes arranged in sequence in the feature identifier sent to the first accelerator, thereby indicating which feature gradients the first accelerator needs to send to other accelerators. This makes it easier for the first accelerator to quickly and accurately determine the features in the embedding table that need to transmit gradients, and it is beneficial for the accelerators to quickly unify and align the feature gradients.

[0155] For example, please refer to Figure 7 , Figure 7This application provides a schematic diagram illustrating how a thread running on a processor and an accelerator use interactive indexing to indicate features that need to be transferred gradients. For example... Figure 7 As shown, during the interaction between the first thread and the second thread, the first information sent by the first thread to the second thread includes indices 1-5 and 8, indicating that the features corresponding to the first training data input to the first accelerator are the features corresponding to indices 1-5 and 8. Furthermore, the second information received by the first thread from the second thread includes indices 6-10, indicating that the features corresponding to the second training data input to the second accelerator are the features corresponding to indices 6-10. Based on the first and second information, the first thread can determine that the features corresponding to all training data processed by the first and second accelerators are features 1-10 (i.e., the features corresponding to indices 1-10). Therefore, the first thread sends feature identifiers to the first accelerator, and the feature identifiers include sequentially arranged indices 1-10, indicating that the first accelerator sends the gradients of features 1-10 to the second accelerator.

[0156] Since the first accelerator actually calculates the effective gradients for features 1-5 and feature 8, when the first accelerator transmits the gradients for features 1-10 to the second accelerator, the gradients for features 6-7 and features 9-10 that the first accelerator actually transmits are all 0. Similarly, the first accelerator receives the gradients for features 1-10 from the second accelerator, and the gradients for features 1-5 are all 0.

[0157] The second implementation method involves accelerators exchanging information about the features corresponding to the gradients they have calculated.

[0158] Unlike implementation method one, in implementation method two, accelerators directly exchange feature information without the need for a coprocessor to obtain feature information.

[0159] For example, in the method described above, in addition to transmitting the gradient of the feature corresponding to the first training data to the second accelerator, the first accelerator also transmits the index of the feature corresponding to the first training data to the accelerators in the accelerator set, so as to indicate the feature corresponding to the gradient transmitted by the first accelerator. That is, the first accelerator transmits the gradient of the feature and the index of this feature to the second accelerator at the same time, so that the second accelerator can determine which features the gradient sent by the first accelerator corresponds to.

[0160] Furthermore, in addition to acquiring the second set of gradients transmitted by the second accelerator, the first accelerator also acquires the index of the feature corresponding to the second training data transmitted by the second accelerator. The index of the feature corresponding to the second training data is used to indicate the feature corresponding to the gradient transmitted by the accelerator in the accelerator set.

[0161] In this way, when performing gradient aggregation of features, the first accelerator can aggregate gradients corresponding to the same feature based on the obtained index. That is, the first accelerator can determine which features the gradient transmitted by the second accelerator corresponds to based on the index transmitted by the second accelerator, and then align the gradients of each feature with the features corresponding to the gradients it has calculated, thereby achieving the aggregation of gradients corresponding to the same feature.

[0162] In this scheme, by using the feature information corresponding to the gradients calculated through direct interaction between accelerators, it can be ensured that each accelerator can determine the features corresponding to the received gradients, thereby enabling the accelerators to successfully aggregate gradients and ensuring the feasibility of the scheme.

[0163] The execution process of the model training method provided in this application has been described above. The following section will detail the specific process of how the model training method is executed collaboratively by the processor and accelerator, using concrete examples.

[0164] For example, please refer to Figure 8 and Figure 9 , Figure 8 This application provides a schematic diagram of the process of a processor and an accelerator working together. Figure 9 This application provides another schematic diagram of the process by which a processor and an accelerator work together.

[0165] like Figure 8 As shown, the processor performs the following steps 811-815 in collaboration with the accelerator to perform data preprocessing.

[0166] Step 811: The processor determines the features corresponding to the training data.

[0167] Since each accelerator corresponds to a thread running on the processor, and the thread can obtain the training data input to the accelerator during a round of iterative training, the processor can determine the features corresponding to the training data based on the training data input to the accelerator by running the thread. Specifically, to facilitate the processor in determining the features corresponding to the training data, the processor may also have an embedding table deployed on it, or a mapping relationship between the indexes of features in the embedding table and the input data recorded.

[0168] like Figure 9As shown, assume the accelerator cluster includes accelerator 1 and accelerator 2. Processor 1 runs a thread corresponding to accelerator 1, therefore processor 1 can determine the features corresponding to the training data as features 1-5 and feature 8 based on the training data input to accelerator 1. Processor 2 runs a thread corresponding to accelerator 2, therefore processor 2 can determine the features corresponding to the training data as features 6-10 based on the training data input to accelerator 2.

[0169] Step 812: The threads running on the processor perform a global interaction feature index.

[0170] Once all the threads running on the processor have determined the features corresponding to the training data, the threads can exchange the indexes of the features they have determined, so that each thread can obtain the features corresponding to all the training data in the current iteration.

[0171] like Figure 9 As shown, the indices of the features sent by processor 1 to processor 2 are indices 1-5 and 8, and the indices of the features sent by processor 2 to processor 1 are indices 6-10.

[0172] Step 813: Based on the feature index of global interaction, the thread running the processor performs feature index deduplication.

[0173] For any given thread, after obtaining the feature index sent by another thread, since there may be overlap between feature indices, the thread can perform a deduplication operation on the feature indexes to remove the overlapping indices.

[0174] like Figure 9 As shown, by combining the feature index determined by the thread it runs and the feature index received from other threads, both processor 1 and processor 2 will remove the overlapping index 8.

[0175] Step 814: The execution characteristics of the threads run by the processor are arranged in sequence.

[0176] To ensure that subsequent accelerators can align feature gradients, the threads running the processor arrange the order of features corresponding to all training data so that the accelerators can transmit feature gradients in the arranged order, thus ensuring that the gradients transmitted by each accelerator are arranged in the same order.

[0177] Specifically, the threads on the processor can arrange the order of features according to the size of the index in the embedded table; or, the threads on the processor can arrange the order of features sent by each processor according to the processor number. This application does not impose specific restrictions on how the threads on the processor arrange the order of features, as long as it ensures that the final arrangement order of each thread is the same.

[0178] like Figure 9 As shown, based on the running threads, after processor 1 and processor 2 arrange the order of features, the specific arrangement order of the features can be determined as feature 1 - embedding 10.

[0179] Step 815: The threads running the processor pass the order of gradients to be transmitted to the accelerator.

[0180] After determining the feature order, the threads running the processor can pass this feature order to the accelerator, instructing the accelerator to arrange the gradients according to this order when transmitting feature gradients. Figure 9 In the example shown, the thread running the processor can pass sequentially ordered indices 1-10 to the accelerator to instruct the accelerator to arrange the gradients of the features in the order of feature 1-feature 10.

[0181] like Figure 8 As shown, the accelerator performs the following steps 821-825 in the process of working with the processor to achieve joint training of the model by multiple accelerators.

[0182] Step 821: The accelerator performs forward propagation on the model.

[0183] During an iterative training run of the model by the accelerator, the accelerator first performs a forward propagation process on the model based on the input training data, that is, it uses the model to process the training data and obtain the model's output. Then, the accelerator calculates the model's loss function value based on the model's output.

[0184] Step 822: The accelerator performs backpropagation gradient calculation on the model.

[0185] Based on the calculated loss function value, the accelerator can use the backpropagation algorithm to calculate the weight parameters in the model and the gradients of the features in the embedding table, thereby obtaining the gradients of the features in the embedding table that correspond to the training data.

[0186] It should be noted that in this application, steps 821 and 822 can be executed in parallel with steps 811-815 above. That is, during the model training process of the accelerator, the processor performs data preprocessing synchronously, thereby ensuring that the training progress of the model is not affected in any way.

[0187] like Figure 9 As shown, accelerator 1 calculates the gradients of features 1-5 and feature 8. Accelerator 2 calculates the gradients of features 6-10.

[0188] Step 823: The accelerator arranges the gradients to be transmitted in the order specified by the processor.

[0189] Before the accelerator calculates the gradients of the features corresponding to the training data, it often receives the permutation order passed by the processor. Therefore, the accelerator can arrange the gradients to be transmitted based on the permutation order indicated by the processor. If the permutation order indicated by the processor contains gradients of some features that the accelerator has not actually calculated, the accelerator can set the gradients of these features to 0.

[0190] like Figure 9 As shown, since processor 1 instructs the accelerator to transmit the gradients in the order of features 1-10, and accelerator 1 actually only calculates the gradients of features 1-5 and features 8, accelerator 1 can set the gradients of features 6-7 and features 9-10 to 0, thus arranging the gradients to be transmitted in the order of features 1-10. Similarly, accelerator 2 sets the gradients of features 1-5 to 0, thus arranging the gradients to be transmitted in the order of features 1-10.

[0191] Step 824: Perform global interactive gradients between accelerators.

[0192] After the gradients to be transmitted are arranged, the gradients between accelerators can be interactively arranged, so that each accelerator can obtain gradients corresponding to the same set of features from other accelerators.

[0193] In step 825, the accelerator performs gradient aggregation and updates the embedding table based on the aggregated gradients.

[0194] After acquiring gradients transmitted from other accelerators, the accelerator can aggregate its own pre-arranged gradients with the acquired gradients. This involves performing operations such as averaging, weighted averaging, or maximizing gradients corresponding to the same feature, thus achieving gradient aggregation. Furthermore, the accelerator needs to update the corresponding features in the embedding table based on the aggregated gradients, thereby updating the features in the embedding table. At this point, the accelerator only needs to update the features in the embedding table that correspond to all training data, rather than updating all features in the embedding table.

[0195] like Figure 9As shown, both accelerator 1 and accelerator 2 perform gradient aggregation of features 1 to features 10, and update the features in the embedding table based on the aggregated gradient corresponding to each feature, thereby realizing the update of features 1 to features 10.

[0196] Please see Figure 10 , Figure 10 This is a schematic diagram illustrating the gradient of an accelerator synchronously embedding in-table features, as provided in this application. Figure 10 As shown, assuming there are 24 features in the embedding table, the training data processed by accelerator 1 corresponds to 6 features in the embedding table, and the training data processed by accelerator 2 corresponds to 5 features in the embedding table. Furthermore, one of the 6 features matched by accelerator 1 overlaps with the feature matched by accelerator 2. Ultimately, when transmitting the gradients of features, accelerator 1 transmits the gradients of 10 features to accelerator 2; accelerator 2 also transmits the gradients of 10 features to accelerator 1. Clearly, compared to transmitting the gradients of all features in the embedding table in existing technologies, this application can significantly reduce the number of gradients that need to be transmitted, thereby improving the communication efficiency between accelerators. For example, in content recommendation scenarios, for the embedding table in the content recommendation model, the amount of gradient data transmitted between accelerators can be reduced from the gigabyte (GB) level to the megabit (MB) level.

[0197] The method provided in this application has been described in detail above. Next, the device provided in this application for performing the above method will be described.

[0198] Please see Figure 11 , Figure 11 This is a schematic diagram of an accelerator cluster structure provided in this application. Figure 11 As shown, the accelerator cluster includes a first accelerator 1101 and a second accelerator 1102, both of which are equipped with training models. Each training model includes a data transformation layer, which in turn includes a feature set. The first accelerator 1101 is used to train the training model using first training data to obtain a first gradient set. The first gradient set includes the gradients of a first portion of the features in the feature set. The first gradient set is then sent to the second accelerator 1102. The second accelerator 1102 is used to train the training model using second training data to obtain a second gradient set. The second gradient set includes the gradients of a second portion of the features in the feature set. The second gradient set is then sent to the first accelerator 1101. The first accelerator 1101 is also used to receive the second gradient set and, based on the first and second gradient sets, update the corresponding features in the data transformation layer.

[0199] In one possible implementation, the gradients contained in the first gradient set have a one-to-one correspondence with the corresponding features of the first training data in the data transformation layer; the gradients contained in the second gradient set have a one-to-one correspondence with the corresponding features of the second training data in the data transformation layer.

[0200] In one possible implementation, both the first gradient set and the second gradient set include the gradients of the corresponding features of the first training data and the second training data in the data transformation layer.

[0201] In one possible implementation, the first accelerator 1101 is further configured to: receive feature identifiers, which indicate the features corresponding to the first training data and the second training data in the data transformation layer.

[0202] In one possible implementation, the feature identifier is also used to indicate the order of the features corresponding to the training data in the training data set in the data transformation layer; the first accelerator 1101 is specifically used to send the gradients in the first gradient set to the second accelerator 1102 in the order of arrangement.

[0203] In one possible implementation, the first accelerator 1101 is specifically used to aggregate gradients corresponding to the same feature in the first gradient set and the second gradient set according to the feature identifier, and use the aggregated gradients to update the corresponding features in the data transformation layer.

[0204] In one possible implementation, the first accelerator 1101 is specifically used to: train the training model using the first training data to obtain the training result; and determine the first gradient set based on the feature identifier and the training result.

[0205] In one possible implementation, a third accelerator 1103 is also included, on which a training model is deployed; the third accelerator 1103 is used to train the deployed training model using third training data to obtain a third gradient set; the third gradient set includes the gradients of the third part of the features in the feature set; the third gradient set is sent to the first accelerator 1101 and the second accelerator 1102; the first accelerator 1101 is also used to send a first gradient set to the third accelerator 1103; receive the third gradient set; and update the corresponding features in the data transformation layer based on the first gradient set, the second gradient set and the third gradient set.

[0206] Please see Figure 12 , Figure 12 A schematic diagram of the structure of a computing device provided in this application. Figure 12 The computing device 1200 shown can be used to execute the model training method provided in this application. For example... Figure 12As shown, the computing device 1200 includes: a bus 1202, a processor 1204, multiple accelerators 1205, a memory 1206, and a communication interface 1208. The processor 1204, the memory 1206, and the communication interface 1208 communicate with each other via the bus 1202. The computing device 1200 can be a server or a terminal device. It should be understood that this application does not limit the number of processors, accelerators, and memory in the computing device 1200.

[0207] Bus 1202 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 12 The bus 1202 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1202 may include a path for transmitting information between various components of the computing device 1200 (e.g., memory 1206, processor 1204, communication interface 1208).

[0208] Processor 1204 may include any one or more processors such as a central processing unit (CPU), a microprocessor (MP), or a digital signal processor (DSP). Processor 1204 includes one or more processor cores.

[0209] Multiple accelerators 1205 may be, for example, neural network processing units (NPUs), graphics processing units (GPUs), or tensor processing units (TPUs).

[0210] The memory 1206 may include volatile memory, such as random access memory (RAM). The processor 1204 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0211] The memory 1206 stores executable program code, which the processor 1204 executes to schedule multiple accelerators 1205 to implement the model training method described above. That is, the memory 1206 stores instructions for executing the model training method.

[0212] The communication interface 1208 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1200 and other devices or communication networks.

[0213] It should be understood that the computing device 1200 according to this application is used to perform, for example... Figures 3 to 10 The model training method can correspond to the subject that performs the method according to this application, and for the sake of brevity, it will not be described in detail here.

[0214] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0215] Please see Figure 13 , Figure 13 This is a schematic diagram of a computing device cluster provided in this application. Figure 13 As shown, the computing device cluster includes at least one computing device 1200. The memory 1206 of one or more computing devices 1200 in the computing device cluster may store the same instructions for executing model training methods.

[0216] In some possible implementations, the memory 1206 of one or more computing devices 1200 in the computing device cluster may also store partial instructions for executing the model training method. In other words, a combination of one or more computing devices 1200 can jointly execute instructions for executing the model training method.

[0217] It should be noted that the memory 1206 in different computing devices 1200 in the computing device cluster can store different instructions, which are used to implement some functions of the accelerator cluster.

[0218] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN), a local area network (LAN), or similar. Figure 14 One possible implementation method is shown. Figure 14 This is a schematic diagram of another computing device cluster structure provided in this application. Figure 14As shown, in the computing device cluster 1400, two computing devices 1200A and 1200B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device.

[0219] It should be understood that Figure 14 The functions of the computing device 1200A shown can also be performed by multiple computing devices 1200. Similarly, the functions of the computing device 1200B can also be performed by multiple computing devices 1200.

[0220] It should be understood that the computing device 1200 or computing device cluster 1400 in this application may correspond to this application. Figure 11 The accelerator cluster in the application can correspond to the execution of this application. Figure 3 The corresponding main body of the model training method. Furthermore, the above and other operations and / or functions of each module in the computing device 1200 or computing device cluster 1400 are respectively for implementing... Figure 3 For the sake of brevity, the corresponding process of the model training method will not be described in detail here.

[0221] For details, please refer to Figure 15 , Figure 15 This application provides a schematic diagram of a chip structure. The chip can be represented as a neural processing unit (NPU), a graphics processing unit (GPU), or a tensor processing unit (TPU), meaning it can be used to implement the aforementioned accelerators. The following description uses an NPU 1500 as an example. The NPU 1500 is mounted as a coprocessor on the host CPU, which allocates tasks. The core of the NPU is the arithmetic circuit 1503, which is controlled by a controller 1504 to retrieve matrix data from memory and perform multiplication operations.

[0222] In some implementations, the arithmetic circuit 1503 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1503 is a two-dimensional pulsating array. The arithmetic circuit 1503 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1503 is a general-purpose matrix processor.

[0223] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1502 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1501 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1508.

[0224] Unified memory 1506 is used to store input and output data. Weight data is directly transferred to weight memory 1502 via Direct Memory Access Controller (DMAC) 1505. Input data is also transferred to unified memory 1506 via DMAC.

[0225] The Bus Interface Unit (BIU) 1510 is used by the instruction fetch memory 1509 to fetch instructions from external memory, and also by the memory access controller 1505 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0226] The DMAC is mainly used to move input data from external memory DDR to unified memory 1506, or to weight data to weight memory 1502, or to input data to input memory 1501.

[0227] The vector computation unit 1507 includes multiple processing units that further process the output of the computation circuit 1503 when needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0228] In some implementations, the vector computation unit 1507 can store the processed output vector in the unified memory 1506. For example, the vector computation unit 1507 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1503, such as linear interpolation of feature planes extracted by a convolutional layer, or, for example, a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 1507 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to the computation circuit 1503, for example, for use in subsequent layers of the neural network.

[0229] The instruction fetch buffer 1509 connected to the controller 1504 is used to store the instructions used by the controller 1504;

[0230] Unified memory 1506, input memory 1501, weighted memory 1502, and instruction fetch memory 1509 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0231] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0232] It should be understood that this application Figure 15 The chip in this application may correspond to the accelerator mentioned herein, or it may be deployed on the computing device 1200 or computing device cluster 1400 of this application. Furthermore, this application... Figure 15 The chip in the application may correspond to the one that executes this application. Figure 3 The corresponding main body of the model training method, and the above and other operations and / or functions of each module in the chip are respectively implemented to achieve the model training method. Figure 3 The corresponding process of the model training method is omitted here for brevity. Furthermore, in this application, the chip structure is not subject to... Figure 15 The chip structure shown may be limited to more or fewer hardware structures to achieve the aforementioned. Figure 3 The function of the method shown.

[0233] Please refer to Figure 16 , Figure 16 This is a schematic diagram of a computer-readable storage medium provided in this application. This application also provides a computer-readable storage medium in some embodiments, wherein the above-described... Figure 3 The disclosed method can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0234] Figure 16 A conceptual partial view of an example computer-readable storage medium arranged according to at least some of the embodiments shown herein is illustrated schematically. The example computer-readable storage medium includes a computer program for executing computer processes on a computing device.

[0235] In one embodiment, the computer-readable storage medium 1600 is provided using a signal bearer medium 1601. The signal bearer medium 1601 may include one or more program instructions 1602, which, when executed by one or more processors, can provide the above-mentioned... Figure 3 The described function or part of the function.

[0236] In some examples, the signal carrying medium 1601 may include a computer-readable medium 1603, such as, but not limited to, a hard disk drive, a compact disc (CD), a digital video disc (DVD), a digital magnetic tape, a memory, ROM, or RAM, etc.

[0237] In some embodiments, the signal-bearing medium 1601 may comprise a computer-recordable medium 1604, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, and so on. In some embodiments, the signal-bearing medium 1601 may comprise a communication medium 1605, such as, but not limited to, digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, and so on). Therefore, for example, the signal-bearing medium 1601 may be transmitted by a wireless communication medium 1605 (e.g., a wireless communication medium conforming to the IEEE 1202.X standard or other transmission protocols).

[0238] One or more program instructions 1602 may be, for example, computer-executable instructions or logical implementation instructions. In some examples, the computing device may be configured to provide various operations, functions, or actions in response to one or more program instructions 1602 conveyed to the computing device via a computer-readable medium 1603, a computer-recordable medium 1604, and / or a communication medium 1605.

[0239] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0240] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods of the various embodiments of this application.

[0241] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0242] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, computer instructions can be transferred from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0243] The above are merely specific embodiments of this application. Variations or substitutions conceived by those skilled in the art based on the specific embodiments provided in this application should all be covered within the protection scope of this application.

Claims

1. An accelerator cluster, characterized in that: It includes a first accelerator and a second accelerator, on which a training model is deployed; the training model includes a data transformation layer, and the data transformation layer includes a feature set; The first accelerator is used to train the training model using the first training data to obtain a first gradient set; the first gradient set includes the gradients of a first portion of the features in the feature set; Send the first set of gradients to the second accelerator; The second accelerator is used to train the training model using the second training data to obtain a second gradient set; the second gradient set includes the gradients of the second part of the features in the feature set; and sends the second gradient set to the first accelerator. The first accelerator is further configured to receive the second gradient set and update the corresponding features in the data transformation layer based on the first gradient set and the second gradient set.

2. The accelerator cluster according to claim 1, characterized in that: The gradients contained in the first gradient set have a one-to-one correspondence with the corresponding features of the first training data in the data transformation layer; The gradients contained in the second gradient set have a one-to-one correspondence with the corresponding features of the second training data in the data transformation layer.

3. The accelerator cluster according to claim 1, characterized in that: Both the first gradient set and the second gradient set include the gradients of the corresponding features of the first training data and the second training data in the data transformation layer.

4. The accelerator cluster according to claim 3, characterized in that, The first accelerator is also used for: Receive feature identifiers, which are used to indicate the features corresponding to the first training data and the second training data in the data transformation layer.

5. The accelerator cluster according to claim 4, characterized in that, The feature identifier is also used to indicate the order of the corresponding features of the training data in the training dataset in the data transformation layer; The first accelerator is specifically used to send gradients from the first gradient set to the second accelerator in the order of arrangement.

6. The accelerator cluster according to claim 4 or 5, characterized in that, The first accelerator is specifically used to aggregate gradients corresponding to the same feature in the first gradient set and the second gradient set according to the feature identifier, and use the aggregated gradients to update the corresponding features in the data transformation layer.

7. The accelerator cluster according to claim 4, characterized in that, The first accelerator is specifically used for: The training model is trained using the first training data to obtain the training result; The first gradient set is determined based on the feature identifier and the training results.

8. The accelerator cluster according to claim 1, characterized in that: It also includes a third accelerator on which the training model is deployed; The third accelerator is used to train the deployed training model using third training data to obtain a third gradient set; the third gradient set includes the gradients of the third part of the features in the feature set; and the third gradient set is sent to the first accelerator and the second accelerator. The first accelerator is further configured to send the first gradient set to the third accelerator; receive the third gradient set; and update the corresponding features in the data transformation layer based on the first gradient set, the second gradient set, and the third gradient set.

9. An accelerator, characterized in that, It includes a processing unit and an interface; the processing unit is used for: The training model is trained using the first training data to obtain a first gradient set; the training model includes a data transformation layer, and the data transformation layer includes a feature set; the first gradient set includes the gradients of a first portion of the features in the feature set; The first gradient set is sent to the second accelerator through the interface; The interface receives a second gradient set, which is obtained by the second accelerator training the training model using the second training data. The second gradient set includes the gradients of the second part of the features in the feature set. Based on the first gradient set and the second gradient set, update the corresponding features in the data transformation layer.

10. The accelerator according to claim 9, characterized in that: The gradients contained in the first gradient set have a one-to-one correspondence with the corresponding features of the first training data in the data transformation layer; The gradients contained in the second gradient set have a one-to-one correspondence with the corresponding features of the second training data in the data transformation layer.

11. The accelerator according to claim 9, characterized in that: Both the first gradient set and the second gradient set include the gradients of the corresponding features of the first training data and the second training data in the data transformation layer.

12. The accelerator according to claim 11, characterized in that, The processing unit is also used for: The interface receives feature identifiers, which are used to indicate the features corresponding to the first training data and the second training data in the data transformation layer.

13. The accelerator according to claim 12, characterized in that, The feature identifier is also used to indicate the order of the corresponding features of the training data in the training dataset in the data transformation layer; The processing unit is specifically used to send the gradients in the first gradient set to the second accelerator through the interface in the order of arrangement.

14. A computing device, characterized in that: It includes a processor and an accelerator as described in any one of claims 9 to 13, wherein the processor is configured to schedule the accelerator to train a training model.

15. The computing device according to claim 14, characterized in that: The processor is further configured to send feature identifiers to the accelerator, the feature identifiers being used to indicate the features corresponding to the first training data and the second training data in the data transformation layer; Both the first set of gradients sent by the accelerator and the second set of gradients received by the accelerator include the gradients of the features indicated by the feature identifier.

16. A model training method, characterized in that, include: The first accelerator uses the first training data to train the training model and obtains the first gradient set; The training model includes a data transformation layer, and the data transformation layer includes a feature set; The first gradient set includes the gradients of a first portion of the features in the feature set; the first gradient set is sent to the second accelerator; The second accelerator trains the training model using the second training data to obtain a second gradient set; the second gradient set includes the gradients of the second part of the features in the feature set; and sends the second gradient set to the first accelerator. The first accelerator receives the second gradient set and updates the corresponding features in the data transformation layer based on the first gradient set and the second gradient set.

17. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 16.

Images