A model training method and device, system, and electronic device
By employing a hybrid training mode in deep learning, combining ring connections and the ring-allreduce algorithm, the problems of gradient failure and low communication efficiency in the distributed parameter server architecture are solved, achieving efficient model training and improved prediction accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING WODONG TIANJUN INFORMATION TECH CO LTD
- Filing Date
- 2022-03-01
- Publication Date
- 2026-05-19
AI Technical Summary
In deep learning, distributed parameter server architecture suffers from gradient failure and low network communication efficiency during model training, resulting in slow training speed and low prediction accuracy.
A hybrid training mode is adopted, which uses a logical ring topology structure with ring connections between worker servers, and combines the ring-allreduce algorithm to perform synchronous updates of dense parameters and bounded asynchronous updates of sparse parameters. Parameter synchronization and updates are performed in batches to avoid communication blockage, and gradients are sent using asynchronous RPC communication.
It improves model training speed and prediction accuracy, supports distributed training of ultra-large-scale models, ensures model convergence speed and final accuracy, and improves the linear speedup ratio of the cluster.
Smart Images
Figure CN116739078B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of deep learning technology, and more specifically, to a model training method, a model training device, a distributed model training system, a computer-readable storage medium, and an electronic device. Background Technology
[0002] Deep learning, as an important branch of machine learning research, uses a hierarchical approach to abstract data at multiple levels to obtain more effective feature information. However, this complex algorithmic structure increases the computational complexity of the model, posing certain challenges to the model training task.
[0003] In related technologies, deep learning is used to model click-through rate (CTR) prediction models, which are trained using a large-scale sparse model based on a distributed parameter server (PS) architecture. This training process has two significant characteristics: first, a large number of training samples; and second, a large model scale, with the number of high-dimensional, sparse feature parameters reaching tens or hundreds of billions.
[0004] In related technologies, on the one hand, the asynchronous training architecture of PS suffers from gradient invalidation during asynchronous parameter updates. That is, when the trainer completes gradient calculation, the model parameters in the parameter server have already been updated, causing the current gradient to become invalid and significantly impacting the model's prediction accuracy. On the other hand, the trainer and parameter server in the cluster need to perform large-scale communication and transmission of model parameters and gradients, which greatly reduces network communication efficiency, resulting in a long model training process and affecting the training speed of the cluster.
[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention
[0006] The purpose of this disclosure is to provide a model training method, a model training device, a distributed model training system, a computer-readable storage medium, and an electronic device, thereby solving to some extent the problems of low prediction accuracy of trained models and long model training time caused by gradient failure in related technologies.
[0007] According to a first aspect of this disclosure, a model training method is provided, applied to a worker server in a distributed system, comprising:
[0008] Retrieve the parameters of the current first model corresponding to the training samples of the current batch from the parameter server;
[0009] Obtain the current second model parameters corresponding to the training samples of the current batch from the working server;
[0010] Based on the training samples of the current batch, the current first model parameters and the current second model parameters are iteratively trained to obtain the gradients of the first model parameters and the second model parameters;
[0011] Send the gradient of the first model parameters to the corresponding parameter server so that the parameter server updates the current first model parameters;
[0012] The gradient of the second model parameters is synchronized to other working servers besides the working server corresponding to the training samples of the current batch, so that each working server updates the current second model parameters based on the gradient of the second model parameters.
[0013] In one exemplary embodiment of this disclosure, based on the foregoing scheme, the method further includes:
[0014] Obtain the training samples of the current batch and parse them to obtain the feature identifiers of the training samples of the current batch;
[0015] Based on the feature identifier, the parameter server corresponding to the feature identifier is determined.
[0016] In an exemplary embodiment of this disclosure, based on the foregoing scheme, the current first model parameters are pre-stored in a parameter server corresponding to the feature identifier, and obtaining the current first model parameters corresponding to the training samples of the current batch includes:
[0017] Based on the feature identifier, obtain the current first model parameters corresponding to the feature identifier from the parameter server corresponding to the feature identifier;
[0018] Each working server pre-stores the current second model parameters corresponding to all training samples; the step of retrieving the current second model parameters corresponding to the current batch of training samples from the working servers includes:
[0019] Based on the feature identifier, the current second model parameters corresponding to the feature identifier are obtained from the working server.
[0020] In an exemplary embodiment of this disclosure, based on the foregoing scheme, the working servers are connected in a ring; synchronizing the gradient of the second model parameters to other working servers besides the working server corresponding to the training samples of the current batch includes:
[0021] The system receives a slice of the second model parameter gradient from the predecessor working server and sends the slice of the second model parameter gradient from the working server to the successor working server, so that after multiple rounds of communication, each working server has a complete second model parameter gradient; the slice of the second model parameter gradient of each working server contains a portion of the second model parameter gradient.
[0022] In an exemplary embodiment of this disclosure, based on the foregoing scheme, each working server updates the current second model parameters based on the gradient of the second model parameters, including:
[0023] Each working server locally updates its own current second model parameters based on the gradient of the second model parameters in the current training epoch.
[0024] In an exemplary embodiment of this disclosure, based on the foregoing scheme, synchronizing the gradient of the second model parameters to other work servers besides the work server corresponding to the training samples of the current batch further includes:
[0025] When a worker server enters the next training epoch, and the current training epoch of its successor worker server has not yet ended, the worker server waits for a period of time before synchronizing the gradients of the second model parameters, so that the update time of the first model parameters between the worker server and its successor worker server differs by at most one training epoch.
[0026] According to a second aspect of this disclosure, a model training apparatus is provided, comprising: an acquisition module, a training module, and a synchronization update module;
[0027] The acquisition module is used to acquire the current first model parameters corresponding to the training samples of the current batch from the parameter server;
[0028] The acquisition module is also used to acquire the current second model parameters corresponding to the training samples of the current batch from the working server;
[0029] The training module is used to iteratively train the current first model parameters and the current second model parameters based on the training samples of the current batch, so as to obtain the gradient of the first model parameters and the gradient of the second model parameters.
[0030] The synchronization update module is used to send the gradient of the first model parameters to the corresponding parameter server so that the parameter server updates the current first model parameters.
[0031] The synchronization update module is also used to synchronize the gradient of the second model parameters to other working servers besides the working server corresponding to the training samples of the current batch, so that each working server updates the current second model parameters based on the gradient of the second model parameters.
[0032] According to a third aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the above embodiments.
[0033] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: one or more processors; and a storage device for one or more programs, which, when executed by the one or more processors, cause the one or more processors to perform the method described in any of the above embodiments.
[0034] According to a fifth aspect of this disclosure, a distributed model training system is provided, including multiple parameter servers and multiple worker servers, any one of the worker servers being used to perform the method described in any of the above embodiments.
[0035] The exemplary embodiments disclosed herein may have some or all of the following beneficial effects:
[0036] In the model training method provided by the exemplary embodiments of this disclosure, on the one hand, the gradient of the second model parameters can be synchronized to other working servers besides the working server corresponding to the training samples of the current batch during each batch of training, so that each working server updates the current second model parameters based on the gradient of the second model parameters, avoiding communication between the working server and the parameter server caused by the second model parameters during training, and alleviating the communication blocking problem during training; on the other hand, the first model parameters can be obtained from the parameter server, and the gradient of the first model parameters can be sent to the corresponding parameter server during training, so that the parameter server updates the current first model parameters; that is, the synchronization and updating of the first model parameters are performed batch by batch, which solves the problem that the training speed is affected by the large size of the first model parameters.
[0037] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0038] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.
[0039] Figure 1 A schematic diagram of an exemplary system architecture for a model training method and apparatus to which embodiments of the present disclosure can be applied is shown;
[0040] Figure 2 A flowchart illustrating a model training method according to an embodiment of the present disclosure is shown schematically.
[0041] Figure 3 This schematically illustrates a parameter update flowchart of a training process according to one embodiment of the present disclosure;
[0042] Figure 4 A timing diagram illustrating parameter updates during training is shown schematically in one embodiment of the present disclosure.
[0043] Figure 5 This schematically illustrates a flowchart of a model training process implemented according to one embodiment of the present disclosure;
[0044] Figure 6 A schematic diagram illustrating the structure of a model training apparatus according to one embodiment of the present disclosure is shown.
[0045] Figure 7 A block diagram of an electronic device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation
[0046] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.
[0047] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.
[0048] Figure 1A schematic diagram of a system architecture 100 for an exemplary application environment to which an embodiment of the model training method and system of this disclosure can be applied is shown. Figure 1 As shown, system architecture 100 is used to train a prediction model. System architecture 100 includes multiple parameter servers 101 and multiple worker servers 102. For example, Figure 1 The diagram shows an n1-parameter server 101. In practical applications, the value of n1 can be determined based on the sparse parameter size of the training samples, and can be several or dozens. The number of worker servers can be determined based on the number of training samples. For example, Figure 1 The diagram shows n² working servers 102. The value of n² can be determined based on the size of the training samples. In practical applications, the value of n² can be determined based on the batch size of the training samples. The parameter server 101 and working servers 102 in the system architecture 100 adopt a distributed architecture. Each working server is connected to multiple parameter servers, and the multiple working servers can form a logical ring topology through a ring connection. The parameter server 101 and working servers 102 can be connected via wired or wireless means; this embodiment does not limit the connection in this way.
[0049] The model training method provided in this embodiment can be executed on the working server 102. Accordingly, the model training device is generally set in the working server 102.
[0050] In practice, click-through rate (CTR) prediction models in search, advertising, and recommendation scenarios exhibit two significant characteristics: First, a large number of training samples; second, large model scale, with high-dimensional, sparse ID feature parameters reaching tens or hundreds of billions in scale. Furthermore, ID feature parameter updates are sparse, with each step updating only a subset of parameters relevant to the current batch. For example, in e-commerce, product IDs are a sparse feature, and the model uses sparse parameters to represent all products. These sparse parameters are enormous, but each training round only involves a portion of the products. Therefore, based on these business and model characteristics, distributed model training is the optimal solution for improving training efficiency. However, for CTR prediction models in search, advertising, and recommendation scenarios, existing training methods have several issues, as detailed below:
[0051] In an asynchronous training architecture using a parameter server (PS), the asynchronous updating of model parameters can lead to gradient invalidation. When a trainer completes gradient calculation, it may find that the model parameters on the parameter server have already been updated. Therefore, the current gradient is based on previously calculated parameters and is no longer valid. Moreover, this problem worsens with the increase in training nodes, significantly impacting the model's final prediction accuracy. Furthermore, the fully connected communication between worker servers and the parameter server in the training architecture cluster greatly reduces network communication efficiency, severely limiting model training speed and preventing the cluster from achieving a good linear speedup.
[0052] For the allreduce synchronous training architecture, the ring-allreduce algorithm, which synchronizes parameter and gradient updates, can solve the network communication problem. However, it requires each node (worker) in the training cluster to store the completed model and parameters, and it has strict limitations on the model size, making it unsuitable for training ultra-large-scale models. To address these issues, this disclosure presents a model training method.
[0053] The technical solutions of the embodiments of this disclosure are described in detail below:
[0054] refer to Figure 2 As shown, a model training method according to an example implementation of this disclosure may include the following steps:
[0055] Step S210: Obtain the current first model parameters corresponding to the training samples of the current batch from the parameter server.
[0056] In this example implementation, the first model parameter can be a sample feature ID from different business scenarios. For example, when a user browses a client or webpage, the video or product on the page is a sample, and the video ID or product ID can be used as the first model parameter. Since video IDs or product IDs have sparsity characteristics, the first model parameter is a sparse parameter. In e-commerce scenarios, sparse parameters can be used to represent all products, demonstrating the enormous scale of sparse parameters. In this example, the first model parameter can be pre-stored in a parameter server. The parameter server corresponding to the first model parameter for the current batch of training samples can be one or more. When the first model parameter corresponding to each batch of training samples is large, it can be divided into multiple parts, each stored in a different parameter server, solving the problem that a single parameter server cannot support the complete model.
[0057] In this example implementation, the model is trained on training samples in batches, with each batch involving only a subset of training samples; that is, each training batch involves a subset of sparse parameters. During each training round, the corresponding first model parameters are retrieved from the parameter server (which can be multiple servers) corresponding to the training samples of the current batch. The current first model parameters are the first model parameters corresponding to the training samples of the current batch.
[0058] In this example implementation, the first model parameters can be stored in a sharded manner. For example, the full set of first model parameters can be distributed and stored across multiple parameter servers, with each parameter server storing a portion of the full set of first model parameters, facilitating parallel training of the first model parameters.
[0059] Step S220: Obtain the current second model parameters corresponding to the training samples of the current batch from the working server.
[0060] In this example implementation, the current second model parameter is the second model parameter corresponding to the current batch. The second model parameter can be a network parameter in the model (such as a click-through rate prediction model), for example, it can be a network parameter such as weights and biases in each network unit. The second model parameter is a dense parameter, and its capacity is relatively small, only in the MB or GB range. Each working server (trainer) can store the complete dense parameter locally.
[0061] In this example implementation, any worker server can be selected to locally read the current second model parameters corresponding to the training samples of the current batch from the GPU's graphics card area.
[0062] Step S230: Based on the training samples of the current batch, iteratively train the current first model parameters and the current second model parameters to obtain the gradients of the first model parameters and the second model parameters.
[0063] In this example implementation, the working server can use the training samples of the current batch, the current first model parameters, and the current second model parameters as input data to perform forward and backward propagation calculations on the sparse model (click rate prediction model) to obtain the parameter gradients of the current first model parameters and the parameter gradients of the second model parameters.
[0064] For details, please refer to the worker training process in the existing PS architecture; it will not be elaborated here.
[0065] Step S240: Send the gradient of the first model parameters to the corresponding parameter server so that the parameter server updates the current first model parameters.
[0066] In this example implementation, the worker server sends the gradient of the first model parameters obtained in each training epoch to the corresponding parameter server, causing the parameter server to update the corresponding first model parameters. In this example, each worker server can send the calculated gradient of the first model parameters to the remote parameter server via asynchronous RPC (Remote Procedure Call) communication, without waiting for the processing result from the parameter server.
[0067] Step S250: Synchronize the gradient of the second model parameters to other working servers besides the working server corresponding to the training samples of the current batch, so that each working server updates the current second model parameters based on the gradient of the second model parameters.
[0068] In this example implementation, the gradient of the second model parameters can be synchronized and updated between different working servers using a ring-all-reduce parameter synchronization algorithm. Each working server can locally store the complete second model parameters, which can reduce the parameter transfer process during training.
[0069] In the model training method provided in this example embodiment, on the one hand, the gradient of the second model parameters can be synchronized to other working servers besides the working server corresponding to the training samples of the current batch during each batch of training, so that each working server updates the current second model parameters based on the gradient of the second model parameters, avoiding communication between the working server and the parameter server caused by the second model parameters during training, and alleviating the communication blocking problem during training; on the other hand, the first model parameters can be obtained from the parameter server, and the gradient of the first model parameters can be sent to the corresponding parameter server during training, so that the parameter server updates the current first model parameters; that is, the synchronization and updating of the first model parameters are performed batch by batch, which solves the problem that the training speed is affected by the large size of the first model parameters.
[0070] The steps of this disclosure are described in more detail below.
[0071] In some embodiments, the method further includes:
[0072] First, obtain the training samples of the current batch and parse them to obtain the feature identifiers of the training samples of the current batch.
[0073] In this example implementation, the working server can obtain the training samples for the current batch from an external source. The training samples can be product IDs or video IDs from search, advertising, or recommendation scenarios. The corresponding feature identifiers (feature IDs) are then parsed from the product IDs or video IDs.
[0074] Then, based on the feature identifier, the parameter server corresponding to the feature identifier is determined.
[0075] In this example implementation, the parameter server containing the parsed feature identifier is located based on the feature identifier. The correspondence between feature identifiers and parameter servers can be stored in advance in the working server. Alternatively, the feature identifier can be placed in the parameter server for matching and searching.
[0076] In some embodiments, obtaining the current first model parameters corresponding to the current batch of training samples includes:
[0077] Based on the feature identifier, the current first model parameters corresponding to the feature identifier are obtained from the parameter server corresponding to the feature identifier.
[0078] In this example implementation, a pull operation can be performed from a determined parameter server to obtain the current first model parameters (partially sparse parameters) corresponding to the feature identifier.
[0079] The step of obtaining the current second model parameters corresponding to the current batch of training samples from the working server includes:
[0080] Based on the feature identifier, the current second model parameters corresponding to the feature identifier are obtained from the working server.
[0081] In this example implementation, the correspondence between feature identifiers and second model parameters can be pre-set in each working server, so that the current second model parameters can be directly read from the local working server based on the feature identifiers.
[0082] In some embodiments, reference Figure 3 The step of synchronizing the gradient of the second model parameters to other work servers besides the work server corresponding to the training samples of the current batch includes:
[0083] The system receives a slice of the second model parameter gradient from the predecessor working server and sends the slice of the second model parameter gradient from the working server to the successor working server, so that after multiple rounds of communication, each working server has a complete second model parameter gradient; the slice of the second model parameter gradient of each working server contains a portion of the second model parameter gradient.
[0084] In this example implementation, the ring connections between different worker servers (trainers) form a logical ring topology. In the logical ring, each worker server node receives data (a slice of the second model parameter gradient from the predecessor node) from the predecessor node and sends data (a slice of its own second model parameter gradient) to the successor node.
[0085] For example, refer to Figure 3 This example illustrates the data synchronization process between three worker servers. The three worker servers are Worker_0, Worker_1, and Worker_2. ai, bi, and ci represent the second model parameters, and ei and fi represent the first model parameters, with i = 0, 1, and 2. Each worker server in the cluster divides its full set of second model parameters or their gradients into three parts (e.g., a0, a1, a2), each part serving as a parameter slice. During data synchronization, in each communication round (each step), 1 / 3 of the parameters are received from the preceding worker server node, and 1 / 3 are sent to the subsequent worker server node. After the first two rounds of communication, each worker server node can calculate the gradient sum of a parameter slice. After two more rounds of communication, each gradient slice or second model parameter slice is broadcast to all other worker server nodes. Thus, after four rounds of communication, each worker server obtains the parameter gradient sum of all worker servers for this round of model training, thereby achieving parameter synchronization between worker servers. In this example, the parameters can be the second model parameters or their gradients.
[0086] In some embodiments, each working server locally updates the current second model parameters based on the gradient of the second model parameters, including:
[0087] Each working server locally updates its own current second model parameters based on the gradient of the second model parameters in the current training epoch.
[0088] In this example implementation, after synchronizing the parameter gradients among the worker servers, each worker server can obtain all gradient slices of the second model parameters for the current training epoch, and then use the second model parameter gradients concatenated from these gradient slices to update its own current second model parameters. For example, the average gradient of the second model parameters can be calculated, and then the average gradient can be used to update the local second model parameters.
[0089] In some embodiments, synchronizing the gradient of the second model parameters to other work servers besides the work server corresponding to the training samples of the current batch further includes:
[0090] When a worker server enters the next training epoch, and the current training epoch of its successor worker server has not yet ended, the worker server waits for a period of time before synchronizing the gradients of the second model parameters, so that the update time of the first model parameters between the worker server and its successor worker server differs by at most one training epoch.
[0091] In this example implementation, to avoid the gradient failure problem in related technologies, this example limits the training progress between two adjacent working servers to no more than one training round.
[0092] For example, such as Figure 4 As shown, the example includes two workers and two parameter server threads (thread_0 and thread_1). The two parameter server threads are each responsible for handling requests from one of the two workers. To simulate gradient failure during asynchronous model training, worker_1 is assumed to train slower than worker_0. Figure 4 As shown, when worker_0 enters step 2, worker_1's step 1 has not yet finished. However, worker_0's synchronization operation will wait for worker_1 to complete the first model parameter update task in step 1 before continuing to the synchronization operation in step 2. When worker_0 enters step 3, it can be guaranteed that worker_1's first model parameter update in step 1 has been completed. In other words, the update rhythm of the first model parameters between workers differs by at most one step. This setting can solve the gradient failure problem.
[0093] Although the first model parameter of this disclosure adopts an asynchronous update method, since the calculation of the entire model requires the simultaneous participation of the first model parameter and the second model parameter, and the second model parameter adopts a synchronous update method, the synchronous update method of the second model parameter ensures that the working server will not use the first model parameter from more than one round ago. Therefore, the asynchronous update of the first model parameter is a bounded asynchronous update method.
[0094] For example, the model training process disclosed herein is as follows: Figure 5 As shown, Figure 5 The diagram illustrates the processing flow of one round of training in this disclosure, which may include the following steps:
[0095] Step 501: The working server obtains a batch of training samples from the training sample set according to the batch size (batch_size) set in the training process, and parses the feature ID corresponding to the current training sample.
[0096] In this example, the training sample set can be first sharded, with each training node independently consuming different training sample shards, facilitating parallel processing of the training samples. During each training round, each worker server obtains its corresponding training sample shard and parses the feature ID corresponding to the current training sample. The training sample can be a product ID or video ID from search, advertising, or recommendation services.
[0097] Step 502: The working server accesses the parameter server based on the feature ID to obtain the first model parameter corresponding to the feature ID.
[0098] In this example, the sample feature ID and the first model parameters can be mapped to different parameter server nodes in advance according to certain rules. During the training process, the corresponding parameter server is accessed based on the feature ID to obtain the first model parameters corresponding to that feature ID.
[0099] Step 503: The working server reads the second model parameters stored in its own memory based on the feature ID.
[0100] In this example, the feature ID and the second model parameters can be stored in the working server in advance. During training, the working server reads the corresponding second model parameters from its own cache.
[0101] Step 504: The working server uses the first model parameters and the second model parameters to complete the forward calculation of the model.
[0102] In this example, each worker server pre-stores an identical copy of the model computation graph and the full set of second model parameters. During training, the second model parameters are read directly from its own cache. The first and second model parameters obtained from the parameter server are then used to perform corresponding forward computations according to the model computation graph to obtain the model's forward computation results.
[0103] Step 505: The working server calculates the gradients of the first model parameters and the second model parameters according to the backpropagation algorithm.
[0104] In this example, each worker server calculates the gradients of the first and second model parameters for the current training epoch based on the forward computation results and the expected output using the backpropagation algorithm.
[0105] Step 506: The working server synchronizes the calculated gradient of the second model parameters to other working servers besides itself.
[0106] In this example, the worker servers use the ring-allreduce algorithm to synchronize the gradients of the second model parameters across worker servers.
[0107] Step 507: The working server asynchronously sends the gradient of the first model parameters to the corresponding remote parameter server.
[0108] In this example, the training speeds of different working servers may differ, so the time at which they send the gradients of the first model parameters also differs. Therefore, different working servers do not send the gradients of the first model parameters to the remote parameter server simultaneously, but communicate the gradients of the first model parameters through a purely asynchronous RPC, without waiting for the processing results (i.e., updating feedback information) of the parameter server.
[0109] Step 508: Each working server locally updates the second model parameters based on the gradient of the second model parameters.
[0110] Step 509: The parameter server updates the first model parameters asynchronously and in a bounded manner based on the gradient of the first model parameters.
[0111] In this example, different worker servers can train at different speeds, so the parameter servers receive the gradients of the first model parameters at different times. The parameter servers update the first model parameters immediately after receiving the gradients, hence the update is asynchronous. Since the entire model computation requires the simultaneous participation of both the first and second model parameters, and the second model parameters are updated synchronously, worker servers cannot use first model parameters older than one epoch. Therefore, the update time of the first model parameters between different worker servers differs by at most one epoch. In other words, worker servers with faster training progress will wait for worker servers with slower training progress before synchronizing parameter gradients, ensuring that the first model parameters of different worker servers are from the same training epoch, thereby avoiding gradient failure and guaranteeing the prediction accuracy of the trained model.
[0112] This disclosure presents a click-through rate (CTR) prediction model for search, advertising, and recommendation services. The model parameters mainly fall into two categories: dense parameters (secondary model parameters) and sparse parameters (primary model parameters). Dense parameters are relatively small, only in the MB or GB range, and each worker server can easily store the complete dense parameters locally. However, sparse parameters constitute the vast majority of the model size, reaching hundreds of GB or even TB levels, far exceeding the capacity of a single server. Therefore, different storage strategies are employed for dense and sparse parameters. For dense parameters, an allreduce synchronous training architecture is used, with each trainer storing the complete dense parameters locally. For sparse parameters, an asynchronous parameter server architecture is used, employing parallel model training. This means that sparse parameters can be distributed across multiple parameter servers using a sharding approach, with each parameter server storing a portion of the total sparse parameters, thus addressing the issue of a single server being unable to handle the entire model.
[0113] Furthermore, to ensure the model's convergence speed and final accuracy during training, different update strategies are used for dense and sparse parameters during the model parameter update process. For example... Figure 3 As shown, for dense parameters, the ring-all-reduce parameter synchronization algorithm is used between worker servers to synchronize and update dense parameters in this round of training. Meanwhile, for sparse parameters, each trainer sends its gradient to the remote parameter server via purely asynchronous RPC communication, without waiting for the parameter server's processing results.
[0114] Based on the aforementioned model parameter update strategy, in a complete training process, the synchronous update of dense parameters and the bounded asynchronous update of sparse parameters jointly ensure the model's convergence speed and final model accuracy. When updating sparse parameters, the asynchronous communication between the worker server and the parameter server prevents the entire training process from being blocked by parameter update operations on the parameter server, thus guaranteeing the model's training speed.
[0115] In summary, the model training method disclosed herein is a hybrid training mode that enables synchronous updates of dense parameters and bounded asynchronous updates of sparse parameters. Simultaneously, based on the logical ring topology between worker servers, the ring-allreduce algorithm improves communication efficiency between worker servers. During sparse parameter updates, the purely asynchronous RPC communication method alleviates the problem of network communication between worker servers and parameter servers blocking the entire training process. Based on this disclosed scheme, the hybrid training mode can support distributed training of ultra-large models, and while ensuring model performance, the training framework can achieve the training efficiency of asynchronous training, while also guaranteeing a good linear speedup for the cluster.
[0116] Furthermore, this example embodiment also provides a model training apparatus 600. This model training apparatus 600 can be applied to a worker server in a distributed system. (See reference...) Figure 6 As shown, the model training device 600 may include: an acquisition module 610, a training module 620, and a synchronization update module 630.
[0117] The acquisition module 610 is used to acquire the current first model parameters corresponding to the training samples of the current batch from the parameter server;
[0118] The acquisition module 610 is further configured to acquire the current second model parameters corresponding to the training samples of the current batch from the working server;
[0119] The training module 620 is used to iteratively train the current first model parameters and the current second model parameters based on the training samples of the current batch, so as to obtain the gradient of the first model parameters and the gradient of the second model parameters.
[0120] The synchronization update module 630 is used to send the gradient of the first model parameters to the corresponding parameter server so that the parameter server updates the current first model parameters.
[0121] The synchronization update module 630 is further configured to synchronize the gradient of the second model parameters to other working servers besides the working server corresponding to the training samples of the current batch, so that each working server updates the current second model parameters based on the gradient of the second model parameters.
[0122] In one exemplary embodiment of this disclosure, the device 600 further includes:
[0123] The parsing module is used to obtain the training samples of the current batch and parse them to obtain the feature identifiers of the training samples of the current batch.
[0124] A model is determined for using the feature identifier to determine the parameter server corresponding to the feature identifier.
[0125] In one exemplary embodiment of this disclosure, the acquisition module 610 is further configured to:
[0126] Based on the feature identifier, the current first model parameter corresponding to the feature identifier is obtained from the parameter server corresponding to the feature identifier; based on the feature identifier, the current second model parameter corresponding to the feature identifier is obtained from the working server.
[0127] In one exemplary embodiment of this disclosure, the synchronization update module 630 is further configured to:
[0128] The system receives a slice of the second model parameter gradient from the predecessor working server and sends the slice of the second model parameter gradient from the working server to the successor working server, so that after multiple rounds of communication, each working server has a complete second model parameter gradient; the slice of the second model parameter gradient of each working server contains a portion of the second model parameter gradient.
[0129] In one exemplary embodiment of this disclosure, the synchronization update module 630 is further configured to:
[0130] Each working server locally updates its own current second model parameters based on the gradient of the second model parameters in the current training epoch.
[0131] In one exemplary embodiment of this disclosure, the synchronization update module 630 is further configured to:
[0132] When a worker server enters the next training epoch, and the current training epoch of its successor worker server has not yet ended, the worker server waits for a period of time before synchronizing the gradients of the second model parameters, so that the update time of the first model parameters between the worker server and its successor worker server differs by at most one training epoch.
[0133] The specific details of each module or unit in the above-mentioned model training device have been described in detail in the corresponding model training methods, so they will not be repeated here.
[0134] This disclosure also provides a distributed model training system, including multiple parameter servers and multiple worker servers, wherein any one of the worker servers is used to execute the model training method described in any of the above embodiments.
[0135] On the other hand, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to perform the methods described in the following embodiments. For example, the electronic device may perform... Figures 2-5 The various steps shown are as follows.
[0136] It should be noted that the computer-readable medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.
[0137] The following reference Figure 7 To describe an electronic device 700 according to such an embodiment of the present disclosure. Figure 7 The electronic device 700 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.
[0138] like Figure 7 As shown, the electronic device 700 is manifested in the form of a general-purpose computing device. The components of the electronic device 700 may include, but are not limited to: at least one processing unit 710, at least one storage unit 720, a bus 730 connecting different system components (including storage unit 720 and processing unit 710), and a display unit 740.
[0139] The storage unit stores program code that can be executed by the processing unit 710, causing the processing unit 710 to perform the steps described in the exemplary method section above according to various exemplary embodiments of this disclosure.
[0140] Storage unit 720 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 7201 and / or cache memory 7202, and may further include a read-only memory (ROM) 7203.
[0141] The storage unit 720 may also include a program / utility 7204 having a set (at least one) program module 7205, such program module 7205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.
[0142] Bus 730 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.
[0143] Electronic device 700 can also communicate with one or more external devices 800 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 700, and / or with any device that enables electronic device 700 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 750. Furthermore, electronic device 700 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 760. As shown, network adapter 760 communicates with other modules of electronic device 700 via bus 730. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RA identification systems, tape drives, and data backup storage systems.
[0144] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause an electronic device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.
[0145] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.
[0146] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps, such as omitting certain steps, combining multiple steps into one step, and / or breaking down one step into multiple steps, should all be considered part of this disclosure.
[0147] It should be understood that this disclosure, as disclosed and defined herein, extends to all alternative combinations of two or more individual features mentioned or apparent in the text and / or figures. All these different combinations constitute multiple alternative aspects of this disclosure. The embodiments described in this specification illustrate the best known mode for implementing this disclosure and will enable those skilled in the art to utilize it.
Claims
1. A model training method, characterized in that, Working servers used in distributed systems include: Retrieve the parameters of the current first model corresponding to the training samples of the current batch from the parameter server; Obtain the current second model parameters corresponding to the training samples of the current batch from the working server; Based on the training samples of the current batch, the current first model parameters and the current second model parameters are iteratively trained to obtain the gradients of the first model parameters and the second model parameters; Send the gradient of the first model parameters to the corresponding parameter server so that the parameter server updates the current first model parameters; The gradient of the second model parameters is synchronized to other working servers besides the working server corresponding to the training samples of the current batch, so that each working server updates the current second model parameters based on the gradient of the second model parameters; The working servers are connected in a ring; synchronizing the gradient of the second model parameters to other working servers besides the working server corresponding to the training samples of the current batch includes: The system receives a slice of the second model parameter gradient from the predecessor working server and sends the slice of the second model parameter gradient from the working server to the successor working server, so that after multiple rounds of communication, each working server has a complete second model parameter gradient; the second model parameter gradient slice of each working server contains a portion of the second model parameter gradient. When a worker server enters the next training epoch, and the current training epoch of its successor worker server has not yet ended, the worker server waits for a period of time before synchronizing the gradients of the second model parameters, so that the update time of the first model parameters between the worker server and its successor worker server differs by at most one training epoch.
2. The model training method according to claim 1, characterized in that, The method further includes: Obtain the training samples of the current batch and parse them to obtain the feature identifiers of the training samples of the current batch; Based on the feature identifier, the parameter server corresponding to the feature identifier is determined.
3. The model training method according to claim 2, characterized in that, The current first model parameters are pre-stored in a parameter server corresponding to the feature identifier. Obtaining the current first model parameters corresponding to the current batch of training samples includes: Based on the feature identifier, obtain the current first model parameters corresponding to the feature identifier from the parameter server corresponding to the feature identifier; Each working server pre-stores the current second model parameters corresponding to all training samples; the step of retrieving the current second model parameters corresponding to the current batch of training samples from the working servers includes: Based on the feature identifier, the current second model parameters corresponding to the feature identifier are obtained from the working server.
4. The model training method according to claim 1, characterized in that, Each working server updates the current second model parameters based on the gradient of the second model parameters, including: Each working server locally updates its own current second model parameters based on the gradient of the second model parameters in the current training epoch.
5. A model training device, characterized in that, include: Acquisition module, training module, and synchronization update module; The acquisition module is used to acquire the current first model parameters corresponding to the training samples of the current batch from the parameter server; The acquisition module is also used to acquire the current second model parameters corresponding to the training samples of the current batch from the working server; The training module is used to iteratively train the current first model parameters and the current second model parameters based on the training samples of the current batch, so as to obtain the gradient of the first model parameters and the gradient of the second model parameters. The synchronization update module is used to send the gradient of the first model parameters to the corresponding parameter server so that the parameter server updates the current first model parameters. The synchronization update module is also used to synchronize the gradient of the second model parameters to other working servers besides the working server corresponding to the training samples of the current batch, so that each working server updates the current second model parameters based on the gradient of the second model parameters; The working servers are connected in a ring. The synchronization update module is further configured to receive the second model parameter gradient slices from the predecessor working server and send the second model parameter gradient slices from the working server to the successor working server, so that after multiple rounds of communication, each working server has complete second model parameter gradients. The second model parameter gradient slices of each working server contain partial second model parameter gradients. When a working server enters the next training round, and the current training round of the successor working server has not yet ended, the working server waits for a period of time before synchronizing the second model parameter gradients, so that the update time of the first model parameters between the working server and its successor working server differs by at most one training round.
6. A distributed model training system, characterized in that, It includes multiple parameter servers and multiple working servers, any one of which is used to execute the method of any one of claims 1-4.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-4.
8. An electronic device, characterized in that, include: One or more processors; as well as A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the method as described in any one of claims 1-4.