Scheduling method and device for model training requests

Abstract

The present application provides a scheduling method and device for model training requests, for a model training system for training a model, the system is composed of multiple parameter nodes and multiple computing nodes, the parameter nodes are used to update the parameters of the model based on gradients, and the computing nodes are used to calculate gradients based on the parameters of the model; the method includes: obtaining pending requests sent by multiple computing nodes; sorting each pending request according to the node identifier in the pending request, wherein the node identifier is used to identify the computing node corresponding to the pending request; according to the sorting result, sending each pending request to the parameter node in turn to obtain the processing result of each pending request. By sorting the requests based on the identifier of the computing node in the request, each request is responded to in an orderly manner, the time for the parameter node to respond to a single computing node request is shortened, and the efficiency of model training is improved.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a scheduling method and device for model training requests. Background Art

[0002] In the field of deep learning, such as computer vision, natural language processing, and personalized recommendation, in order to improve the quality or effect of the model, larger model parameters or larger training data are usually used during model training, such as click-through rate prediction models with trillion-level parameters and language models with hundreds of billions of parameters. Distributed training has become a necessary means for efficient training of ultra-large-scale models.

[0003] The commonly used distributed training architecture is the ps-worker architecture, which divides nodes into two roles: ps (parameter server) and worker (work server or computing server). In asynchronous mode, each worker independently initiates pull or push requests to multiple ps, performs gradient calculations based on the model parameters fed back by multiple ps, and updates the model parameters stored in the ps based on the gradients.

[0004] In related technologies, PS often responds to each request in sequence based on the time sequence of the received requests. Since different workers are asynchronous, the time when the request arrives at PS is random. The way of processing each request in time sequence is inefficient in request response, which reduces the speed of asynchronous training. Summary of the invention

[0005] The present application provides a scheduling method and device for model training requests, which sorts the requests based on the node identifiers in the requests, thereby controlling the parameter nodes to respond to the requests of each computing node in an orderly manner based on the computing node, shortening the response time of the parameter nodes to each computing node and improving the overall training speed.

[0006] In a first aspect, the present application provides a scheduling method for a model training request, the method being applied to a model training system, the model training system being used to train a model based on a training set, the model training system comprising a plurality of parameter nodes and a plurality of computing nodes, the parameter nodes being used to update parameters of the model based on gradients, the computing nodes being used to calculate gradients based on the parameters of the model; the method comprising:

[0007] Obtain pending requests sent by multiple computing nodes; sort each pending request according to the node identifier in the pending request; send each pending request to the parameter node in turn according to the sorting result to obtain the processing result of each pending request; wherein the node identifier is used to identify the computing node corresponding to the pending request.

[0008] In a second aspect, the present application provides a scheduling device for model training requests, which is applied to a model training system, wherein the model training system is used to train a model based on a training set, and the model training system includes multiple parameter nodes and multiple computing nodes, wherein the parameter nodes are used to update the parameters of the model based on gradients, and the computing nodes are used to calculate gradients based on the parameters of the model; the scheduling device for model training requests includes a network transceiver module and a scheduling module; wherein the network transceiver module is used to receive pending requests sent by each computing node; the scheduling module is used to sort the received pending requests according to the node identifier in the pending request to obtain a sorting result, and according to the sorting result, send each pending request to the parameter node in turn to obtain a processing result corresponding to each pending request, wherein the node identifier is used to identify the computing node corresponding to the pending request.

[0009] In a third aspect, the present application provides an electronic device, including:

[0010] A processor, and a memory communicatively connected to the processor;

[0011] The memory stores computer-executable instructions;

[0012] The processor executes the computer-executable instructions stored in the memory to implement the scheduling method for model training requests provided in the first aspect of the present application.

[0013] In a fourth aspect, the present application provides a computer-readable storage medium, which stores computer execution instructions. When the computer execution instructions are executed by a processor, they are used to implement the scheduling method for model training requests provided in the first aspect of the present application.

[0014] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the scheduling method for model training requests provided in the first aspect of the present application.

[0015] The scheduling method and device for model training requests provided in the present application are aimed at the scenario of using a model training system for model training. The model training system adopts a ps-worker architecture, which is composed of multiple parameter nodes and multiple computing nodes. For the parameter node, after receiving the pending requests sent by multiple computing nodes, based on the node identifier in the pending request, that is, the identifier used to identify the computing node, the pending requests received from the multiple computing nodes are sorted, and each pending request is sent to the parameter node in turn based on the sorting result, so that the parameter node responds to each pending request in turn according to the sorting result to obtain its processing result, thereby realizing request sorting with computing node as the granularity, so that the order of response requests in multiple parameter nodes remains consistent, so that multiple pending requests of the same computing node can be responded to at the same time, shortening the request response time of the computing node surface and improving the efficiency of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0017] Figure 1A A schematic diagram of a model training system provided for one embodiment of the present application;

[0018] Figure 1B A schematic diagram of a computing node training process in a synchronous mode provided by an embodiment of the present application;

[0019] Figure 1C A schematic diagram of a computing node training process in an asynchronous mode provided by an embodiment of the present application;

[0020] Figure 2 A flowchart of a method for scheduling model training requests provided in an embodiment of the present application;

[0021] Figure 3 For this application Figure 2 A flow chart of a possible implementation of step S202 in the illustrated embodiment;

[0022] Figure 4 For this application Figure 2 A flowchart of another possible implementation of step S202 in the illustrated embodiment;

[0023] Figure 5 For this application Figure 2 A flowchart of another possible implementation of step S202 in the illustrated embodiment;

[0024] Figure 6 A flowchart of a method for scheduling model training requests provided in another embodiment of the present application;

[0025] Figure 7 A schematic diagram of the structure of a scheduling device for model training requests provided in an embodiment of the present application;

[0026] Figure 8 A schematic diagram of the structure of a scheduling device for model training requests provided in another embodiment of the present application;

[0027] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0028] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0029] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0030] First, the terms used in this application are explained:

[0031] Parameter node: During the deep learning training process, a node is used to store, distribute, summarize, and update the parameters of the model. Each parameter node is responsible for part of the model's parameters.

[0032] Compute node: During deep learning training, a node is used to perform training-related tasks, including inference calculations and gradient calculations. There is no communication between different compute nodes.

[0033] Pull: The computing node pulls the latest model parameters from the corresponding parameter node based on the topological structure of the model.

[0034] push: The computing node uploads the calculated gradient to the corresponding parameter node to update the model parameters.

[0035] Figure 1A A schematic diagram of a model training system provided in one embodiment of the present application, such as Figure 1AAs shown, the model training system adopts a ps-worker architecture for training the model based on a training set. The model training system includes multiple parameter nodes and multiple computing nodes. Each of the computing nodes and parameter nodes is composed of a GPU (Graphics Processing Unit).

[0036] The main process of model training based on the ps-worker architecture can be divided into four processes: parameter acquisition (pull), forward calculation, backward calculation, and gradient update (push). The pull and push processes are the processes of interaction between computing nodes and parameter nodes. In each round of training, the above four processes are executed in sequence, that is, after the computing nodes successfully acquire the parameters, they perform forward calculation and backward calculation to obtain the gradient, and update the model parameters based on the gradient.

[0037] During the parameter acquisition process, the computing node sends a pull request or a parameter acquisition request to multiple parameter nodes storing model parameters. The parameter node feeds back the model parameters stored thereon to the computing node. After obtaining all the parameters of the model, the computing node performs forward and backward calculations based on the parameters of the model to obtain the gradient. During the gradient update process, the computing node sends the gradient to the corresponding multiple parameter nodes, that is, sends a push request or a gradient update request to the multiple parameter nodes. The multiple parameter nodes update the model parameters stored thereon based on the gradient calculated by the computing node, thereby completing a round of model training.

[0038] Figure 1B A schematic diagram of a computing node training process in a synchronous mode provided by an embodiment of the present application, such as Figure 1B As shown, in synchronous mode, only after the push process of the previous round of training (such as the nth round) is completed, the computing node will initiate a pull request for the next round of training (such as the n+1th round), that is, enter the pull process of the next round of training.

[0039] Figure 1C A schematic diagram of a computing node training process in an asynchronous mode provided by an embodiment of the present application is shown in FIG. Figure 1C As shown in the figure, in the sparse training scenario, since the timeliness requirement for gradients is relatively low, the push process can be asynchronous. In asynchronous mode, the next round of training does not need to wait for the push process of the previous round of training to be completed, but can enter the pull process of the next round of training after the backward calculation of the previous round is completed, that is, initiate a pull request for the next round of training.

[0040] This application is aimed at the scenario of model training in asynchronous mode, that is, the model training system trains the model based on the training set in asynchronous mode. In each round of model training, each computing node independently initiates a pull request to multiple parameter nodes, waits for the multiple parameter nodes to feedback the latest parameters of the stored model, calculates the gradient based on the parameters, and generates a push request based on the calculated gradient, so that the parameter node updates the stored model parameters based on the gradient.

[0041] There is no need for synchronization between computing nodes. In each round of training, computing nodes will initiate requests to multiple parameter nodes. These requests together constitute a transaction of the computing node. After the processing results of all requests are returned, the computing node can perform subsequent calculations. As a result, the requests of computing nodes have a barrel effect. The slowest response request determines the processing time of the transaction corresponding to the request initiated by the computing node.

[0042] In the related art, for the requests received by the parameter node side, sequential execution or out-of-order execution is often adopted. Sequential execution means executing the requests of each calculation in the order in which the requests arrive at the parameter node; out-of-order execution means executing the requests of each computing node in a random order. In both of the above methods, it is impossible to ensure that the requests of the same computing node are processed at the same time in different parameter nodes, resulting in a long tail phenomenon in the request processing of the computing node, which leads to a long request response time and reduces the overall speed of model training.

[0043] In order to improve the request processing efficiency from the perspective of computing nodes and the overall efficiency of model training, the present application provides a scheduling method for model training requests, which is applied to multiple parameter nodes. The requests received by the parameter nodes are sorted based on the node identifiers, and the parameter nodes are controlled to respond to the requests in sequence based on the sorting results. The same sorting method is used in multiple parameter nodes, so that requests from the same computing node are responded to by the parameter node at the same time, shortening the request response time on the computing node side, thereby improving the overall efficiency of model training.

[0044] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0045] Figure 2A flow chart of a scheduling method for model training requests provided in an embodiment of the present application, wherein the scheduling method for model training requests is applied to a model training system, wherein the model training system is composed of multiple parameter nodes and multiple computing nodes, and can be specifically executed by any device with data processing function in the model training system, which can be in the form of a computer, a server, etc., such as a scheduling device for model training requests, the above-mentioned parameter nodes, or other devices connected to the parameter nodes, and the scheduling device for model training requests can be set in each parameter node or independently.

[0046] In one embodiment, the model training system may be a system for training a click-through rate prediction model.

[0047] In one embodiment, the model training system uses an asynchronous mode to perform model training.

[0048] In one embodiment, each parameter node is provided with a device or apparatus for executing the scheduling method of the model training request provided in the present application, so that the order of responding to the requests in each parameter node is the same.

[0049] like Figure 2 As shown, the scheduling method for the model training request includes the following steps:

[0050] Step S201, obtaining pending requests sent by multiple computing nodes.

[0051] Among them, the pending request is a request sent by the computing node to the parameter node during the model training process. The request is a request related to model parameters. It can be a request for obtaining model parameters, such as a pull request, or a request for updating model parameters, such as a push request. The pending request can include the node identifier of the corresponding computing node, such as worker_id. The node identifier is used to identify the computing node corresponding to the pending request, and can also include the current training round number of the corresponding computing node, such as local_step_id.

[0052] Specifically, during the model training process, the computing node or worker node sends a pull request to multiple parameter nodes storing corresponding model parameters according to the topological structure of the model to pull the latest model parameters from the multiple parameter nodes. After the computing node calculates the gradient based on the current batch of training data and model parameters, the computing node needs to upload the calculated gradient to the corresponding multiple parameter nodes, that is, send a push request to the corresponding multiple parameter nodes.

[0053] In one embodiment, the parameters of the model stored in different parameter nodes are different from each other, that is, each parameter node stores part of the parameters of the model.

[0054] In one embodiment, the computing node needs to obtain all parameters of the model during each round of training, that is, send a pull request to all parameter nodes storing the model parameters.

[0055] For example, if the model includes 1,000 parameters, the number of parameter nodes is 10, and each parameter node stores 100 model parameters. Each computing node can initiate a pull request to the 10 parameter nodes in each round of training. After the computing node obtains all the parameters of the model, it performs forward calculations on the model based on the current batch of training data to obtain the error of the model output, and obtains the gradient through the back propagation of the error; after the gradient calculation is completed, a push request is initiated to the 10 parameter nodes to update the model parameters stored in the 10 parameter nodes.

[0056] In one embodiment, the computing node may only obtain part of the parameters of the model during each round of training, so that the computing node only needs to send a pull request to the corresponding multiple parameter nodes. The parameter node corresponding to the computing node is the parameter node that stores the model parameters required by the computing node.

[0057] For example, the model training system consists of 1,000 parameter nodes, and the current scenario only involves model parameters stored in 9 of the parameter nodes. In each round of training, the computing node can only initiate pending requests to the 9 parameter nodes.

[0058] Specifically, the pending requests received from the computing nodes may be stored in a request pool or a task queue. When the pending requests are sent to the threads of the corresponding parameter nodes, the pending requests are deleted from the request pool or the task queue.

[0059] In one embodiment, in order to balance task allocation, the number of model parameters stored in each parameter node may be as similar as possible.

[0060] Step S202: sorting the requests to be processed according to the node identifiers in the requests to be processed.

[0061] In order to ensure that requests sent from the same computing node to different parameter nodes are processed simultaneously, each parameter node can be controlled to sort the pending requests in the request pool in the same sorting method, that is, each pending request in the request pool is sorted according to the node identifier in the pending request.

[0062] Specifically, the node identifiers in each pending request in the request pool may be extracted, and the pending requests may be sorted based on the node identifiers.

[0063] Specifically, the node identifier of the computing node may be a number, which remains unchanged once set, and different computing nodes have different numbers.

[0064] Exemplarily, the computing nodes may be numbered in sequence based on the order in which the computing nodes initiate model training tasks, such as by using an auto-incrementing method.

[0065] Specifically, the pending requests in the task queue or request pool may be sorted based on the node identifiers in the pending requests. The pending requests may be sorted from large to small or from small to large according to the values ​​corresponding to the node identifiers, or in a specified order.

[0066] For example, taking 10 computing nodes as an example, the nodes are identified as w_01 to w_10, then the requests to be processed may be sorted in the following order: w_05, w_06, w_01, w_03, w_04, w_08, w_02, w_07, w_10 and w_09.

[0067] Furthermore, after obtaining the sorting result, if there are multiple pending requests for the same computing node in the request pool of the parameter node, such as a pull request and a push request, then for the multiple pending requests of the computing node, the multiple pending requests of the computing node can be sorted locally based on the types of the pending requests to update the sorting result. The order of the pull request can be set to precede the push request.

[0068] Step S203: according to the sorting result, each request to be processed is sent to the parameter node in sequence to obtain the processing result of each request to be processed.

[0069] After obtaining the sorting result of the pending requests, based on the sorting result, each pending request is taken out from the task queue or request pool in turn and sent to the thread of the parameter node, so that the thread of the parameter node responds to the pending request and obtains the processing result of the pending request.

[0070] In one embodiment, the parameter node may include two thread pools, each thread pool includes multiple threads, and different thread pools are used to process different types of pending requests. Such as a pull task thread pool and a push task thread pool, wherein the threads in the pull task thread pool are used to process pull requests, and the threads in the push task thread pool are used to process push requests. When the pending requests are sorted, the pull requests and push requests in the request pool or task queue can be sorted respectively based on the node identifier in the pending request, so as to obtain the sorting result of the pull request, such as the sorted pull request queue, and the sorting result of the push request, such as the sorted push request queue. Based on the sorting result of the pull request, each pull request is sent to the pull task thread pool of the parameter node in sequence, and based on the sorting result of the push request, each push request is sent to the push task thread pool of the parameter node in sequence.

[0071] When the requests in one of the request queues have been sent, and there are still multiple pending requests in the other request queue, the pending requests in the other request queue can be sent to the thread pool corresponding to one of the requests to speed up the request processing. That is, when the sorted pull request queue is empty, the push requests in the sorted push request queue can be sent to the pull task thread pool for processing. Similarly, when the sorted push request queue is empty, the pull requests in the sorted pull request queue can be sent to the push task thread pool for processing.

[0072] During the training process, the computing node will send pending requests to multiple parameter nodes at the same time, so that the set of parameter nodes corresponding to the pending requests in the request pool or task queue of each parameter node is the same, and the same scheduling method for model training requests is adopted in each parameter node, so that the pending requests of the same computing node are processed in the same order in their corresponding parameter nodes, which greatly reduces the overall response time of multiple pending requests sent by the computing node.

[0073] The scheduling method for model training requests provided in this embodiment is aimed at the scenario of using a model training system for model training. The model training system adopts a ps-worker architecture, which is composed of multiple parameter nodes and multiple computing nodes. For the parameter node, after receiving the pending requests sent by multiple computing nodes, based on the node identifier in the pending request, that is, the identifier used to identify the computing node, the received pending requests from the multiple computing nodes are sorted, and each pending request is sent to the parameter node in turn based on the sorting result, so that the parameter node responds to each pending request in turn according to the sorting result to obtain its processing result, and realizes request sorting with computing node as the granularity, so that the order of response requests in multiple parameter nodes remains consistent, so that multiple pending requests of the same computing node can be responded to at the same time, shortening the request response time of the computing node surface and improving the efficiency of model training.

[0074] In asynchronous mode, since the pull request for the next round of training is issued after the backward calculation of the previous round of training is completed, the request pool or task queue of the parameter node will contain push requests for the previous round of training and pull requests for the next round of training for the same computing node.

[0075] In order to further improve the efficiency of pending requests, the above step S202 can be replaced by: sorting the pending requests according to the node identifiers in the pending requests, and at least one of the types of the pending requests and the number of training rounds in the pending requests.

[0076] The number of training rounds is used to characterize the number of rounds of model training performed by the computing node corresponding to the pending request. The types of pending requests may include parameter acquisition type and gradient update type. The pending request of parameter acquisition type is the above-mentioned pull request or parameter acquisition request, and the pending request of gradient update type is the above-mentioned push request or gradient update request.

[0077] Specifically, after receiving each pending request, the pending request may be parsed to extract the node identifier, the number of training rounds, and the type of the pending request in the pending request.

[0078] Specifically, each pending request may be sorted based on the node identifier, and then based on the type and / or number of training rounds of the pending request, the pending requests with the same node identifier or the same order in the sorting result may be sorted again to obtain a sorting result.

[0079] In one embodiment, in the sorting result, for pending requests corresponding to the same computing node, the smaller the number of training rounds, the higher the order in the sorting result.

[0080] In one embodiment, in the sorting result, for pending requests corresponding to the same computing node, the order of the pull-type pending requests in the sorting result is earlier than the push-type pending requests.

[0081] In one embodiment, the pending requests of the parameter nodes may be sorted in sequence according to the order of the type of pending request, node identifier and number of training rounds, or according to the order of number of training rounds, node identifier and type of processing request, or according to other orders of the above three parameters to obtain a sorting result.

[0082] Optional, Figure 3 For this application Figure 2 A flow chart of a possible implementation of step S202 in the embodiment shown is as follows: Figure 3 As shown, step S202 may include the following steps:

[0083] Step S301: sort the requests to be processed according to their types to obtain a first sorting result.

[0084] Specifically, the response or processing order of various types of pending requests can be preset, so that the pending requests are sorted based on the type of each pending request in the request pool or task queue and the preset order to obtain a first sorting result.

[0085] Specifically, in the first sorting result, the order of parameter acquisition request is prior to gradient update request, that is, the order of pull request is prior to push request, that is, pull request is responded to first, thereby reducing the blockage of computing node during training and improving the overall efficiency of each round of training.

[0086] Specifically, each pending request in the request pool of the parameter node can be traversed. If the type of the pending request is a gradient update type or a push type, the pending request is placed at the end of the queue corresponding to the request pool, and so on, so that the order of the pull type pending requests in the queue is before the push type pending requests.

[0087] Specifically, it is possible to first determine whether the types of pending requests in the request pool of the parameter node are all the same. If so, step S301 can be omitted, and the queue corresponding to the pending requests in the request pool can be directly determined as the first sorting result; if not, the order of the pending requests in the first sorting result is determined according to the types of the pending requests, and the queues in which the pending requests in the request pool of the parameter node are located are sorted based on the order of the pending requests in the first sorting result, thereby obtaining the first sorting result.

[0088] Exemplarily, the order of pending requests of the push type may be set to 2, and the order of pending requests of the pull type may be set to 1.

[0089] Table 1 List of pending requests in the request pool of the Nth parameter node

[0090] Request sequence number Type of request Node ID Number of training rounds Pending Request 1 pull worker3 6 Pending Request 2 push Worker 3 5 Pending Request 3 pull Worker 4 7 Pending Request 4 pull Worker 5 6 Pending Request 5 pull Worker 6 6

[0091] Exemplarily, Table 1 is a list of pending requests in the request pool of the Nth parameter node provided by an embodiment of the present application. In Table 1, the pending requests currently received or currently stored in the request pool of the Nth parameter node include 5 pending requests, i.e., pending request 1 to pending request 2, wherein the request type of pending request 2 is push type, and the types of the remaining pending requests are all pull types. Then the order of the pending requests in the first sorting result is: pending request 1, pending request 3, pending request 4, pending request 5 and pending request 2.

[0092] Step S302, sorting the pending requests with the same order in the first sorting result according to the node identifiers in the pending requests to obtain a second sorting result, and sending the pending requests to the thread of the parameter node in sequence based on the second sorting result.

[0093] Among them, the pending requests with the same order in the first sorting result are pending requests with the same type, such as pending request 1, pending request 3, pending request 4 and pending request 5 in Table 1.

[0094] After obtaining the first sorting result, in order to ensure that the same type of pending requests corresponding to the same computing node in different parameter nodes are processed simultaneously, the first sorting result can be sorted again, specifically: if there are multiple pending requests in the same order in the first sorting result, the node identifier in each of the multiple pending requests is extracted, and based on the node identifier, the multiple pending requests are sorted for the second time to obtain a second sorting result, such as sorting in descending order according to the node identifier. For example, in the embodiment corresponding to Table 1, the order of the pending requests in the second sorting result can be: pending request 5, pending request 4, pending request 3, pending request 1 and pending request 2.

[0095] By first sorting the pending requests in the request pool based on the request type, and then sorting the pending requests of the same type based on the node identifier, pull-type requests can be processed first, thereby reducing congestion during training. At the same time, it ensures that requests sent from the same computing node to multiple parameter nodes can be processed at the same time, greatly reducing the response time of transactions initiated by computing nodes and improving the efficiency of model training.

[0096] Optional, Figure 4 For this application Figure 2 A flow chart of another possible implementation of step S202 in the embodiment shown is as follows: Figure 4 As shown, step S202 may include the following steps:

[0097] Step S401: sort the pending requests according to the number of training rounds in the pending requests to obtain a third sorting result.

[0098] Specifically, in the third sorting result, the higher the number of training rounds corresponding to the pending request is, the later the order of the pending request is.

[0099] Optionally, the pending requests are sorted according to the number of training rounds in the pending requests to obtain a third sorting result, including:

[0100] According to the number of training rounds in the pending requests, each pending request is sorted in ascending order to obtain a third sorting result.

[0101] Specifically, the number of training rounds in each pending request in the request pool of the parameter node can be extracted, and each pending request can be sorted based on the number of training rounds, such as in ascending order, so as to give priority to pending requests with smaller numbers of training rounds, so as to avoid large differences in the number of training rounds executed between different computing nodes, thereby reducing the parallelism of model training.

[0102] Specifically, the number of training rounds in each pending request that needs to be scheduled in the request pool of the parameter node can be counted, and the order of each pending request in the third sorting result can be set based on the counted number of training rounds. As shown in Table 1, the order of each pending request in the third sorting result corresponding to the pending request can be: pending request 2, pending request 1, pending request 4, pending request 5 and pending request 3, where the order of pending request 1, pending request 4 and pending request 5 is the same.

[0103] Further, it can be determined first whether there is a target number of training rounds in the counted number of training rounds, wherein the target number of training rounds is the number of training rounds whose difference with the corresponding maximum number of rounds is less than a preset difference; if so, then for each pending request whose corresponding number of training rounds is the target number of rounds, sort each pending request in descending order of the number of training rounds, that is, the larger the number of training rounds, the earlier the order; for the remaining pending requests, sort each pending request in ascending order of the number of training rounds, that is, the larger the number of training rounds, the later the order, to obtain a third sorting result, and in the third sorting result, the order of the pending request whose number of training rounds is the target number of training rounds precedes the order of other pending requests; if not, sort each pending request in ascending order of the number of training rounds in the pending requests to obtain a third sorting result.

[0104] The preset difference may be 10, 5, 3 or other values, and the maximum number of training rounds corresponding to different computing nodes, that is, the maximum number of rounds, may be different.

[0105] The advantage of this setting is that the requests of the computing nodes that are about to finish training can be processed first, so that the computing nodes can train the next model faster.

[0106] Step S402, sorting the pending requests with the same order in the third sorting result according to the node identifiers in the pending requests to obtain a fourth sorting result, and sending the pending requests to the thread of the parameter node in sequence based on the fourth sorting result.

[0107] Among them, the pending requests with the same order in the third sorting result are the pending requests with the same number of training rounds, such as pending request 1, pending request 4 and pending request 5 in Table 1.

[0108] After obtaining the third sorting result, in order to ensure that the pending requests with the same number of training rounds corresponding to the same computing node in different parameter nodes are processed simultaneously, the third sorting result can be sorted again, specifically: if there are multiple pending requests with the same order in the third sorting result, the node identifier in each of the multiple pending requests is extracted, and based on the node identifier, the multiple pending requests are sorted for the second time to obtain a fourth sorting result, such as sorting in ascending order according to the node identifier (the larger the number corresponding to the node identifier, the later the order). For example, in the embodiment corresponding to Table 1, the order of the pending requests in the fourth sorting result can be: pending request 2, pending request 1, pending request 4, pending request 5 and pending request 3.

[0109] By first sorting the pending requests in the request pool based on the number of training rounds, and then sorting the pending requests of the same type based on the node identifier, requests with fewer training rounds can be processed first, thereby shortening the overall time required for model training. At the same time, it ensures that requests sent from the same computing node to multiple parameter nodes can be processed at the same time, greatly reducing the response time of transactions initiated by computing nodes and improving the efficiency of model training.

[0110] Optional, Figure 5 For this application Figure 2 A flow chart of another possible implementation of step S202 in the embodiment shown is as follows: Figure 5 As shown, step S202 may include the following steps:

[0111] Step S501, determining the ranking score of the request to be processed according to the number of training rounds in the request to be processed and the type of the request to be processed.

[0112] Specifically, the ranking score of the pending request can be determined based on a pre-established preset corresponding relationship, the number of training rounds in the pending request and the type of the pending request, wherein the preset corresponding relationship is used to describe the corresponding relationship between the ranking score, the number of training rounds and the type of the pending request.

[0113] Specifically, the types of pending requests can be numerically represented, and the numerical value corresponding to the pull type is higher than the numerical value corresponding to the push type, so as to give priority to responding to the pending requests of the pull type. Weights are set for the number of training rounds and the types of pending requests, and then the ranking scores of the pending requests are determined based on the weighted average of the number of training rounds of the pending requests and the types of the pending requests.

[0114] Exemplarily, the value corresponding to the type of the pending request of the pull type may be 1, and the value corresponding to the type of the pending request of the push type may be 0.

[0115] Specifically, the ranking score of the pending request with a lower training round number may be set to be higher.

[0116] For example, the expression of the ranking score can be:

[0117] S i =w i ×N i +w2×T i

[0118] Among them, S i Indicates the ranking score of the i-th pending request in the request pool; N i represents the number of training rounds in the i-th pending request; T iIndicates the type of the i-th pending request; w1 and w2 are weight coefficients respectively.

[0119] Step S502: sort the requests to be processed according to their sorting scores and the node identifiers in the requests to be processed to obtain a sorting result.

[0120] Specifically, the sorting may be performed based on the sorting score first, and then for multiple pending requests with the same sorting score in the sorting result, the multiple pending requests may be sorted again based on the node identifiers in the multiple pending requests to obtain the sorting result.

[0121] Specifically, each pending request can be sorted based on the node identifier in the pending request, such as in ascending, descending or random order according to the node identifier, and then multiple pending requests sent by the same computing node can be sorted again based on the sorting score to obtain a sorting result.

[0122] By comprehensively considering the node identification, type and number of training rounds of the pending requests, the pending requests are sorted, so that the parameter nodes are controlled to respond to each pending request in sequence according to the sorting results, shortening the response time of transactions initiated by the computing nodes, and avoiding the phenomenon that the computing nodes cannot obtain the processing results for a long time, thereby improving the parallelism of model training.

[0123] Figure 6 A flowchart of a method for scheduling model training requests provided in another embodiment of the present application. Figure 2 On the basis of the embodiment shown in the figure, step S202 and step S203 are further refined, and a step of determining the number of threads corresponding to various types is added before step S201, and a step of sending the processing result is added after step S203, such as Figure 6 As shown, the scheduling method for model training requests provided in this embodiment may include the following steps:

[0124] Step S601, determining the number of threads corresponding to various types of pending requests in the parameter node according to the structure of the trained model.

[0125] Among them, the threads corresponding to various types of pending requests include threads corresponding to pull requests, namely pull task threads, and threads corresponding to push requests, namely push task threads.

[0126] Specifically, the number of threads corresponding to various types of pending requests in the parameter node may be determined according to the number of parameters of the trained model.

[0127] Specifically, the thread ratios corresponding to various commonly used models can be established in advance, and then based on the results of the currently trained model, a common model matching the model can be determined from multiple commonly used models, and based on the thread ratio of the matching common model and the total number of threads of the parameter node, the number of threads corresponding to various types in the parameter node can be determined.

[0128] Step S602, obtaining pending requests sent by multiple computing nodes.

[0129] Step S603: for each type of pending requests, sort the pending requests of the type according to the node identifiers in the pending requests of the type, and obtain a sorting result corresponding to the type.

[0130] Specifically, each type of pending requests can be sorted separately according to the type of pending requests. That is, for each type of pending requests in the request pool of the parameter node, the pending requests of this type are sorted according to the node identifier in the pending requests of this type. The specific sorting method can refer to the sorting method provided in any of the above embodiments, such as step S202, or Figures 3 to 5 The sorting method provided by any of the provided embodiments only replaces all pending requests in the request pool with a certain type of pending requests in the request pool.

[0131] Exemplarily, for each type of pending requests, the pending requests of this type may be sorted based on the node identifiers and the number of training rounds in the pending requests of this type.

[0132] Step S604: for each type, according to the sorting result corresponding to the type, each to-be-processed request of the type is sent in sequence to the thread corresponding to the type of the parameter node.

[0133] Specifically, the pending requests in the request pool are sorted separately according to their types to obtain sorting results corresponding to various types, such as the sorting results corresponding to pull requests (or called pull task queues) and the sorting results corresponding to push requests (or called push task queues), and tasks (pending requests) are taken out from the pull task queue and the push task queue in turn, and sent to threads in the pull task thread pool and the push task thread pool, respectively, to process different types of pending tasks in parallel.

[0134] Specifically, when the pull task queue is empty, the tasks in the push task queue can also be executed by threads in the pull thread pool. Similarly, when the push task queue is empty, the tasks in the pull task queue can also be executed by threads in the push thread pool to improve the response speed of pending requests.

[0135] Step S605: Send the processing result of each request to be processed to the corresponding computing node.

[0136] After processing the corresponding pending request, the thread of the parameter node feeds back the processing result to the scheduling device of the model training request, and the scheduling device of the model training request sends the processing result to the corresponding computing node, thereby completing the response to the pending request.

[0137] In this embodiment, based on the type of requests to be processed, each type of request is sorted separately based on the node identifier, and a corresponding thread pool is set for each type in the parameter node, so that the thread pool responds to the request of the corresponding type according to the sorting result, thereby achieving an orderly response to the request based on the type and node identifier, so that the parameter nodes in the model training system process the requests of the same computing node as synchronously as possible, greatly shortening the response time of the transaction corresponding to the request initiated by the computing node; through the dynamic scheduling of the thread pool, the utilization rate of the parameter node thread pool is improved, and the response speed of the requests to be processed is further improved, thereby improving the efficiency of model training.

[0138] Figure 7 A structural schematic diagram of a scheduling device for a model training request provided in an embodiment of the present application, wherein the device is applied to a model training system, wherein the model training system is used to train a model based on a training set, wherein the model training system includes multiple parameter nodes and multiple computing nodes, wherein the parameter nodes are used to update the parameters of the model based on gradients, and the computing nodes are used to calculate gradients based on the parameters of the model.

[0139] like Figure 7 As shown, the device includes: a network transceiver module 710 and a scheduling module 720.

[0140] Among them, the network transceiver module 710 is used to receive the pending requests sent by each computing node; the scheduling module 720 is used to sort the received pending requests according to the node identifier in the pending requests, obtain the sorting result, and according to the sorting result, send each pending request to the parameter node in turn to obtain the processing result corresponding to each pending request, wherein the node identifier is used to identify the computing node corresponding to the pending request.

[0141] Optionally, the scheduling module 720 includes:

[0142] The sorting unit is used to sort the received pending requests according to the node identifier in the pending requests to obtain the sorting result; the request sending unit is used to send each pending request to the thread of the parameter node in turn according to the sorting result to obtain the processing result corresponding to each pending request.

[0143] Optional, sorting unit, specifically used for:

[0144] The pending requests are sorted according to the node identifier in the pending requests, and at least one of the type of the pending requests and the number of training rounds in the pending requests; wherein the number of training rounds is used to represent the number of rounds of model training performed by the computing node corresponding to the pending request.

[0145] Optional, sorting unit, including:

[0146] The first sorting subunit is used to sort each pending request according to the type of the pending request to obtain a first sorting result; the second sorting subunit is used to sort each pending request with the same order in the first sorting result according to the node identifier in the pending request to obtain a second sorting result, so as to send each pending request to the thread of the parameter node in sequence based on the second sorting result.

[0147] Optionally, the requests to be processed include two types: parameter acquisition requests and gradient update requests. In the first sorting result, the parameter acquisition requests are ordered before the gradient update requests.

[0148] Optional, sorting unit, including:

[0149] The third sorting subunit is used to sort each pending request according to the number of training rounds in the pending request to obtain a third sorting result; the fourth sorting subunit is used to sort each pending request with the same order in the third sorting result according to the node identifier in the pending request to obtain a fourth sorting result, so as to send each pending request to the thread of the parameter node in sequence based on the fourth sorting result.

[0150] Optionally, the third sorting subunit is specifically used for:

[0151] According to the number of training rounds in the pending requests, each pending request is sorted in ascending order to obtain a third sorting result.

[0152] Optional, sorting unit, specifically used for:

[0153] According to the number of training rounds in the pending requests and the types of the pending requests, the ranking scores of the pending requests are determined; and according to the ranking scores of the pending requests and the node identifiers in the pending requests, the pending requests are sorted.

[0154] Optional, sorting unit, specifically used for:

[0155] For each type of pending requests, the pending requests of the type are sorted according to the node identifiers and the number of training rounds in the pending requests of the type to obtain a sorting result corresponding to the type.

[0156] Accordingly, the request sending unit is specifically used for:

[0157] For each type, according to the sorting result corresponding to the type, each pending request of the type is sent in sequence to the thread corresponding to the type of the parameter node.

[0158] Optionally, the device further comprises:

[0159] The thread partitioning module is used to determine the number of threads corresponding to various types of pending requests in the parameter node according to the structure of the trained model.

[0160] The scheduling device for model training requests provided in the embodiment of the present application can be used to execute the above Figures 2 to 6 The technical solutions provided in any corresponding embodiment have similar implementation principles and technical effects, which will not be described in detail in this embodiment.

[0161] Optional, Figure 8 A schematic diagram of a scheduling device for model training requests provided in another embodiment of the present application, combined with Figure 7 and Figure 8 It can be seen that the scheduling device for model training requests provided in this embodiment also includes: a task queue module 730 and a thread pool module 740.

[0162] Among them, the task queue module 730 is used to store each pending request in the task queue after receiving the pending request sent by each computing node. The scheduling module 720 is used to sort the pending requests in the task queue based on the node identifier in the pending request to obtain a sorted task queue; and send each pending request in the sorted task queue to the thread of the parameter node in turn to obtain the processing result of each pending request. The thread pool module 740 is composed of multiple threads and is used to respond to each pending request.

[0163] Optionally, the task queue includes a pull task queue and a push task queue. The pull task queue is used to store pull requests, and the push task queue is used to store push requests.

[0164] Optionally, the thread pool module 740 includes a pull task thread pool and a push task thread pool, the threads in the pull task thread pool are used to process or respond to pull requests sent by computing nodes, and the threads in the push task thread pool are used to process or respond to push requests sent by computing nodes.

[0165] Optionally, the network transceiver module 710 is also used to send the processing results to the corresponding computing nodes.

[0166] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the electronic device provided in this embodiment includes:

[0167] At least one processor 910; and a memory 920 communicatively connected to the at least one processor; wherein the memory 920 stores computer-executable instructions; and the at least one processor 910 executes the computer-executable instructions stored in the memory so that the electronic device performs a method as provided in any of the foregoing embodiments.

[0168] Optionally, the memory 920 may be independent or integrated with the processor 910 .

[0169] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the aforementioned embodiments and will not be described in detail here.

[0170] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the method provided in any of the aforementioned embodiments can be implemented.

[0171] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method provided in any of the above embodiments when executed by a processor.

[0172] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0173] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application.

[0174] It should be understood that the above-mentioned processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and can also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.

[0175] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0176] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0177] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0178] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0179] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods provided in each embodiment of the present application.

[0180] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0181] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A scheduling method for model training requests, characterized in that: The method is applied to a model training system, the model training system is used to train a model based on a training set, the model training system includes a plurality of parameter nodes and a plurality of computing nodes, the parameter nodes are used to update the parameters of the model based on the gradient, and the computing nodes are used to calculate the gradient based on the parameters of the model; The method comprises: Get pending requests sent by multiple computing nodes; Sort the pending requests according to at least one of the type of the pending requests and the number of training rounds in the pending requests, and the node identifiers in the pending requests; wherein the number of training rounds is used to represent the number of rounds of model training performed by the computing node corresponding to the pending request; According to the sorting results, each pending request is sent to the parameter node in turn to obtain the processing result of each pending request; The node identifier is used to identify the computing node corresponding to the request to be processed.

2. The method according to claim 1, characterized in that The pending requests are sorted according to at least one of the type of the pending requests and the number of training rounds in the pending requests, and the node identifier in the pending requests, including: Sort the pending requests according to the types of the pending requests to obtain a first sorting result; According to the node identifier in the pending request, the pending requests with the same order in the first sorting result are sorted to obtain a second sorting result, so as to send the pending requests to the thread of the parameter node in sequence based on the second sorting result.

3. The method according to claim 2, characterized in that The requests to be processed include two types: parameter acquisition requests and gradient update requests. In the first sorting result, the order of the parameter acquisition requests is prior to that of the gradient update requests.

4. The method according to claim 1, characterized in that The pending requests are sorted according to at least one of the type of the pending requests and the number of training rounds in the pending requests, and the node identifier in the pending requests, including: Sorting the pending requests according to the number of training rounds in the pending requests to obtain a third sorting result; According to the node identifier in the pending request, the pending requests with the same order in the third sorting result are sorted to obtain a fourth sorting result, so as to send the pending requests to the thread of the parameter node in sequence based on the fourth sorting result.

5. The method according to claim 4, characterized in that According to the number of training rounds in the pending requests, the pending requests are sorted to obtain a third sorting result, including: According to the number of training rounds in the pending requests, each pending request is sorted in ascending order to obtain a third sorting result.

6. The method according to claim 1, characterized in that The pending requests are sorted according to at least one of the type of the pending requests and the number of training rounds in the pending requests, and the node identifier in the pending requests, including: Determine a ranking score of the pending requests according to the number of training rounds in the pending requests and the types of the pending requests; The pending requests are sorted according to the sorting scores of the pending requests and the node identifiers in the pending requests.

7. The method according to claim 1, characterized in that The pending requests are sorted according to at least one of the type of the pending requests and the number of training rounds in the pending requests, and the node identifier in the pending requests, including: For each type of pending requests, sort the pending requests of the type according to the node identifiers and the number of training rounds in the pending requests of the type, and obtain a sorting result corresponding to the type; According to the sorting results, each pending request is sent to the thread of the parameter node in turn, including: For each type, according to the sorting result corresponding to the type, each pending request of the type is sent in sequence to the thread corresponding to the type of the parameter node.

8. The method according to claim 7, characterized in that The method further comprises: According to the structure of the trained model, the number of threads corresponding to various types of pending requests in the parameter node is determined.

9. A scheduling device for model training requests, characterized in that: Applied to a model training system, the model training system is used to train a model based on a training set, the model training system includes a plurality of parameter nodes and a plurality of computing nodes, the parameter nodes are used to update the parameters of the model based on the gradient, and the computing nodes are used to calculate the gradient based on the parameters of the model; The device includes a network transceiver module and a scheduling module; Among them, the network transceiver module is used to receive the pending requests sent by each computing node; the scheduling module is used to sort the pending requests according to the type of the pending request and at least two of the number of training rounds in the pending request, and the node identifier in the pending request, to obtain the sorting result; wherein the number of training rounds is used to characterize the number of rounds of model training performed by the computing node corresponding to the pending request, and according to the sorting result, each pending request is sent to the parameter node in turn to obtain the processing result corresponding to each pending request, wherein the node identifier is used to identify the computing node corresponding to the pending request.

Images