Task scheduling method, system, device and storage medium for asynchronous training

By scheduling the execution order of parameter processing tasks based on task identifiers and time information at the parameter node, the problem of untimely processing of worker server requests is solved, improving computational efficiency and fault tolerance.

CN114265675BActive Publication Date: 2024-11-08ALIBABA (CHINA) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111493619.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-08
Publication Date
2024-11-08
Estimated Expiration
2041-12-08

AI Technical Summary

Technical Problem

In the asynchronous training process of deep learning, the requests of the worker server cannot be processed in a timely manner in the existing technology, resulting in low computational efficiency.

Method used

By scheduling the received parameter processing tasks at the parameter node, the execution order is determined based on the task identifier and time information, prioritizing the processing of tasks on the same working node, and reducing waiting time.

Benefits of technology

It improves the computational efficiency of worker nodes, reduces the waiting time of worker nodes, and enhances the fault tolerance of the asynchronous training process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114265675B_ABST
    Figure CN114265675B_ABST
Patent Text Reader

Abstract

The embodiment of the present application provides a task scheduling method, system, device and storage medium for asynchronous training. In the scheduling method, after the parameter node receives multiple parameter processing tasks corresponding to at least one working node, the execution order of the multiple parameter processing tasks can be scheduled according to the task identifiers corresponding to the source working nodes of the multiple parameter processing tasks and the time information of the multiple parameter processing tasks, and the multiple parameter processing tasks are executed in sequence according to the scheduling results. In this embodiment, when scheduling the execution order of multiple parameter processing tasks, the source working nodes of the parameter processing tasks and the time information of the parameter processing tasks are considered at the same time, so that the parameter nodes can be scheduled to process the parameter processing tasks of the same working node in a timely manner as much as possible within the same or similar time range, thereby reducing the waiting time of the working node and improving the computing efficiency of the working node.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a task scheduling method, system, device and storage medium for asynchronous training. Background Art

[0002] The PS-Worker architecture (parameter server-worker architecture) is a commonly used architecture for asynchronous training of deep learning. In the existing model training process, multiple worker servers independently send parameter processing tasks to the parameter server, and the parameter server returns parameters in the order in which the requests arrive. This method makes it impossible to process requests from the same worker server in a timely manner, which greatly affects the computing efficiency of the worker server. Therefore, a solution needs to be proposed. Summary of the invention

[0003] Various aspects of the embodiments of the present application provide a task scheduling method, system, device and storage medium for asynchronous training, which are used to schedule requests received by parameter nodes from working nodes, thereby improving the computing efficiency of the working nodes.

[0004] An embodiment of the present application provides a task scheduling method for asynchronous training, including: determining multiple parameter processing tasks in a task queue; the multiple parameter processing tasks are sent by at least one working node; wherein the parameter processing tasks sent by any working node carry a task identifier corresponding to the working node; scheduling the execution order of the multiple parameter processing tasks according to the task identifiers of the multiple parameter processing tasks and the time information of the multiple parameter processing tasks to obtain the execution order of the multiple parameter processing tasks; and executing the multiple parameter processing tasks in sequence according to the execution order of the multiple parameter processing tasks.

[0005] An embodiment of the present application also provides a task scheduling system for asynchronous training, comprising: at least one working node and a parameter node; the at least one working node is used to: send a parameter processing task to the parameter node; the parameter processing task sent by any working node carries a task identifier corresponding to the working node; the parameter node is used to: determine multiple parameter processing tasks in a task queue; the multiple parameter processing tasks are sent by at least one working node; the execution order of the multiple parameter processing tasks is scheduled according to the task identifiers of each of the multiple parameter processing tasks and the time information of each of the multiple parameter processing tasks to obtain the execution order of the multiple parameter processing tasks; according to the execution order of the multiple parameter processing tasks, the multiple parameter processing tasks are executed in sequence.

[0006] An embodiment of the present application also provides a server device, comprising: a memory, a processor, and a communication component; the memory is used to store one or more computer instructions; the processor is used to execute one or more computer instructions to execute the steps in the method provided in the embodiment of the present application.

[0007] The embodiment of the present application also provides a computer-readable storage medium storing a computer program, and when the computer program is executed, the steps in the method provided in the embodiment of the present application can be implemented.

[0008] In an embodiment of the present application, after a parameter node receives multiple parameter processing tasks corresponding to at least one working node, the execution order of the multiple parameter processing tasks can be scheduled according to the task identifiers of the multiple parameter processing tasks and the time information of the multiple parameter processing tasks, and the multiple parameter processing tasks can be executed in sequence according to the scheduling results. Among them, the task identifier of the parameter processing task corresponds to the source working node of the parameter processing task. In this embodiment, when scheduling the execution order of multiple parameter processing tasks, the source working node of the parameter processing task and the time information of the parameter processing task are taken into consideration at the same time, so that the parameter node can be scheduled to process the parameter processing tasks of the same working node in a timely manner within the same or similar time range as much as possible, thereby reducing the waiting time of the working node and improving the computing efficiency of the working node. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0010] Figure 1 A schematic diagram of the structure of a task scheduling system for asynchronous training provided by an exemplary embodiment of the present application;

[0011] Figure 2 A schematic diagram of a task queue at a parameter node provided by an exemplary embodiment of the present application;

[0012] Figure 3 A schematic diagram of a time window obtained by dividing at a parameter node provided by an exemplary embodiment of the present application;

[0013] Figure 4 A schematic diagram of a function module of a parameter node provided by an exemplary embodiment of the present application;

[0014] Figure 5 A flowchart of a task scheduling method for asynchronous training provided by an exemplary embodiment of the present application;

[0015] Figure 6A schematic diagram of the structure of a server device provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0017] The PS-Worker architecture (parameter server-worker architecture) is a commonly used architecture for asynchronous training of deep learning. In the existing model training process, multiple worker servers independently send parameter processing tasks to the parameter server, and the parameter server returns the parameters in the order of arrival of the requests. Each worker server needs to wait for the parameter server to return the model parameters before starting to calculate the gradient. This method makes it impossible to process requests from the same worker server in a timely manner, which greatly affects the computing efficiency of the worker server.

[0018] In view of the above technical problems, some exemplary embodiments of the present application provide a solution, which will be described in detail in the following part with reference to the accompanying drawings.

[0019] Figure 1 A schematic diagram of the structure of a task scheduling system for asynchronous training provided by an exemplary embodiment of the present application is shown in FIG. Figure 1 As shown, the task scheduling system 100 for asynchronous training mainly includes: a working node 10 and a parameter node 20. The number of the working node 10 can be one or more, which is not limited in this embodiment.

[0020] The task scheduling system 100 for asynchronous training is implemented based on the PS-Worker architecture. The work node 10 is implemented as a work server (Worker) in the PS architecture, and the parameter node is implemented as a parameter server (ParameterServer, PS) in the PS-Worker architecture. Among them, the work server and the parameter server can be implemented as a conventional server device, or can be implemented as a cloud host, a cloud server, a network server, a distributed server, a virtual host, an elastic computing instance on the cloud, etc., and also includes a plurality of computing units that perform computing tasks according to a distributed cloud computing method, and the computing unit includes a device with computing capability related to this application, which is not limited in this embodiment.

[0021] During the asynchronous training of the deep learning model, multiple working nodes can independently obtain model parameters from the parameter node 20, perform gradient calculation based on the obtained model parameters, and return the calculated gradient to the parameter node 20. The parameter node 20 can update the model parameters based on the received gradient and send the updated parameters to the working node 10.

[0022] When the working node 10 obtains the model parameters from the parameter node 20, a parameter processing task may be sent to the parameter node 20. The working node 10 may send one or more parameter processing tasks to the parameter node 20 according to the actual parameter requirements. When the task scheduling system 100 for asynchronous training includes multiple working nodes, the multiple working nodes may independently send multiple parameter processing tasks to the parameter node 20. The parameter processing task refers to a task for causing the parameter node 20 to return the required parameters to the working node 10. Generally, when the working node 10 sends a parameter acquisition request to the parameter node 20, it can be regarded as sending a parameter processing task to the parameter node 20.

[0023] For the parameter node 20, when it receives multiple parameter processing tasks, in the case of limited hardware resources, it can give priority to adding the multiple parameter processing tasks to the task queue to wait for scheduling planning. Figure 2 As shown, there are working nodes W_a, W_b, and parameter nodes PS_1, PS_n; working node W_a can send task a1 and task a2 to parameter nodes PS_1 and PS_n respectively, and working node W_b can send task b1 and task b2 to parameter nodes PS_1 and PS_n respectively. On the PS_1 and PS_n side, when the tasks cannot be executed in time, tasks a1, a2, b1 and b2 can be added to their respective task queues.

[0024] The parameter node 20 is mainly used for: after determining a plurality of parameter processing tasks in the task queue, scheduling the execution order of the plurality of parameter processing tasks according to the task identifiers of the plurality of parameter processing tasks and the arrival times of the plurality of parameter processing tasks, and obtaining the execution order of the plurality of parameter processing tasks; and executing the plurality of parameter processing tasks in sequence according to the execution order of the plurality of parameter processing tasks.

[0025] For the parameter node 20, when executing any parameter processing task, the parameter processing task can be parsed by performing network protocol parsing, serialization / deserialization, score judgment and other processing operations; according to the parsing results, the corresponding parameters can be obtained and the obtained parameters are returned to the working node.

[0026] Among them, the execution order of multiple parameter processing tasks is scheduled to plan the execution order of multiple parameter processing tasks while they are waiting to be executed, so as to meet the computing needs of the request initiator (i.e., the working node) under the premise of reasonable use of limited resources.

[0027] In this embodiment, the parameter processing task sent by any working node carries the task identifier corresponding to the working node. The task identifiers corresponding to different working nodes are different, so that the parameter node 20 can distinguish the parameter processing tasks from different working nodes according to the task identifiers carried by the parameter processing tasks. Among them, the task identifier can be the name, address or ID (Identity document) of the working node, which is not limited in this embodiment.

[0028] Among them, the time information of any parameter processing task may include the sending time or arrival time of the parameter processing task. Among them, the sending time refers to the time when the working node sends the parameter processing task, and the parameter processing task may carry a sending timestamp for the parameter node 20 to view. Among them, the arrival time refers to the time when the parameter processing task arrives at the parameter node 20. For some parameter processing tasks, there is a certain delay in their arrival time relative to their triggering time. In some embodiments, the parameter node 20 may schedule the execution order according to the sending time and task identification of each of the multiple parameter processing tasks. In other embodiments, the parameter node 20 may schedule the execution order according to the arrival time and task identification of each of the multiple parameters, which is not limited in this embodiment.

[0029] When the parameter node 20 schedules the execution order of multiple parameter processing tasks, it can combine the task identifiers and time information of each of the multiple parameters at the same time, that is, in the process of scheduling, the parameter node 20 simultaneously considers the time sequence factors and source factors of the parameter processing tasks, which is conducive to timely synchronous execution of different parameter processing tasks of the same working node. For example, in some optional embodiments, when the parameter node 20 schedules the execution order of the multiple parameter processing tasks according to the task identifiers of the multiple parameter processing tasks received and the arrival time information of the multiple parameter processing tasks, the parameter processing tasks with the same task identifier can be sorted according to the time information to obtain the time sorting results corresponding to the multiple task identifiers. Afterwards, the multiple task identifiers are sorted according to the difference in the time information of the first-ranked parameter processing tasks corresponding to the multiple task identifiers. Based on the sorting results on the task identifier dimension and the internal sorting results of the same task identifier, the execution order of the multiple parameter processing tasks can be determined.

[0030] In some other optional embodiments, the parameter node 20 sorts the parameter processing tasks with the same task identifier according to the time information, and after obtaining the time sorting results corresponding to the multiple task identifiers, the multiple task identifiers can be further sorted according to the number difference of the parameter processing tasks corresponding to the multiple task identifiers. Based on the sorting results on the task identifier dimension and the internal sorting results of the same task identifier, the execution order of the multiple parameter processing tasks can be determined.

[0031] In some other embodiments, the parameter node 20 may segment the time, and sort the parameter processing tasks whose time information falls within each time segment according to the task identifiers to obtain the execution order of the multiple parameter processing tasks.

[0032] The above embodiments are all optional implementation methods for the parameter node 20 of the present application to schedule the execution order of multiple parameter processing tasks. Of course, the optional implementation method for the parameter node 20 provided in the embodiments of the present application to schedule according to the task identifiers and time information of multiple parameter processing tasks is not limited to the above-mentioned methods.

[0033] In the task scheduling system 100 for asynchronous training, the working node 10 and the parameter node 20 can be connected in communication based on wired or wireless communication. When wireless communication is used for communication connection, the wireless communication method can be implemented as a long-distance wireless communication method such as LORA. The working node 10 can also be connected in communication with the parameter node 20 through a mobile network, and the network standard of the mobile network can be any one of 2G (GSM), 2.5G (GPRS), 3G (WCDMA, TD-SCDMA, CDMA2000, UTMS), 4G (LTE), 4G+ (LTE+), 5G, 5.5G, WiMax, etc.

[0034] In this embodiment, after the parameter node receives multiple parameter processing tasks corresponding to at least one working node, the execution order of the multiple parameter processing tasks can be scheduled according to the task identifiers of the multiple parameter processing tasks and the time information of the multiple parameter processing tasks, and the multiple parameter processing tasks can be executed in sequence according to the scheduling results. Among them, the task identifier of the parameter processing task corresponds to the source working node of the parameter processing task. In this embodiment, when scheduling the execution order of multiple parameter processing tasks, the source working node of the parameter processing task and the time information of the parameter processing task are considered at the same time, so that the parameter node can be scheduled to process the parameter processing tasks of the same working node in a timely manner within the same or similar time range as much as possible, thereby reducing the waiting time of the working node and improving the computing efficiency of the working node.

[0035] In addition, in this asynchronous training implementation, multiple working nodes independently request model parameters from the parameter node, and the mutual influence between different working nodes is low, which further improves the fault tolerance of the asynchronous training process.

[0036] Typically, the asynchronous training process of the model needs to undergo multiple rounds of iterative training so that the model continues to converge. In the above and following embodiments of the present application, in order to facilitate the parameter node 20 to distinguish different training rounds of the same working node, the task identifiers of the parameter processing tasks sent by any working node in different training rounds are different, and the task identifiers of the parameter processing tasks sent in the same training round are the same. Thus, the parameter node 20 can consider the training rounds of the parameter processing tasks when scheduling the parameter processing tasks in the task queue, which is conducive to synchronously processing the parameter processing tasks sent by the same working node in the same training round.

[0037] Optionally, for any working node, when generating a task identifier for the parameter processing task of each training round, a random generation algorithm can be used to generate an ID of a certain length for the parameter processing task of each training round; or, a UUID (Universally Unique Identifier) ​​can be generated for the parameter processing task of each training round, which is not limited in this embodiment.

[0038] In some exemplary embodiments, the parameter node 20 may schedule the execution order of the multiple parameter processing tasks according to the task identifiers of the multiple parameter processing tasks and the arrival times of the multiple parameter processing tasks.

[0039] Optionally, when scheduling the execution order of the multiple parameter processing tasks according to the task identifiers of the multiple parameter processing tasks and the time information of the multiple parameter processing tasks, the parameter node 20 may divide the multiple parameter processing tasks into at least one task group according to the arrival time of the multiple parameter processing tasks. For example, the parameter processing tasks whose arrival time is within the set time range may be divided into one task group. For another example, the parameter processing tasks whose arrival time interval is less than the set interval threshold may be divided into one task group.

[0040] Among them, the operation of dividing multiple parameter processing tasks into at least one task group according to the arrival time of the parameter processing tasks can be regarded as an operation of scheduling the parameter processing tasks in the time dimension. After the multiple parameter processing tasks are divided into at least one task group, the parameter processing tasks can be sorted within each task group to schedule the parameter processing tasks from the source dimension of the parameter processing tasks. Taking any task group as an example, the parameter node 20 can sort the parameter processing tasks in the task group according to the task identifier of the parameter processing tasks in the task group to obtain the execution order of the parameter processing tasks in the task group.

[0041] After performing the above-mentioned intra-group scheduling operation on the at least one task group, the parameter node 20 may obtain the execution order of the multiple parameter processing tasks, and may execute the multiple parameter processing tasks in sequence according to the execution order.

[0042] In some exemplary embodiments, the parameter node 20 may divide the time axis according to a set time interval to obtain multiple time windows before dividing the multiple parameter processing tasks into at least one task group according to the arrival time of the multiple parameter processing tasks. The time axis is used to describe the time span of asynchronous training. The starting point of the time axis may be the starting time of the model asynchronous training; the length of the time axis extends backward as the asynchronous training proceeds. Figure 3 As shown, the parameter nodes PS_1, PS_1...PS_M can divide the time axis into multiple time windows, such as time window t1 and time window t2, in time sequence.

[0043] Optionally, when the time axis is divided according to a set time interval, the time interval used is associated with the request frequency of the at least one working node. In some embodiments, the length of the time interval may be negatively correlated with the request frequency. That is, when the request frequency of the at least one working node is low, the time interval may be a larger value, thereby saving the number of scheduling calculations; when the request frequency of the at least one working node is low, the time interval may be a smaller value, so as to reasonably schedule a large number of parameter processing tasks in a timely manner and reduce the waiting time of the working node.

[0044] After obtaining multiple time windows, for any parameter processing task, the parameter node 20 can determine the target time window corresponding to the parameter processing task from the multiple time windows according to the arrival time of the parameter processing task, and add a window identifier corresponding to the target time window to the parameter processing task. Figure 3Taking the illustrated time window as an example, parameter node PS_1 can add a window identifier corresponding to time window t1 for tasks whose arrival time falls within time window t1, and can add a window identifier corresponding to time window t2 for tasks whose arrival time falls within time window t2.

[0045] The window identifier of any time window may be the number of the time window. For example, after the time axis is divided into multiple time windows according to a set time interval, the multiple time windows may be numbered to obtain window identifiers corresponding to the multiple time windows.

[0046] Accordingly, when the parameter node 20 divides the multiple parameter processing tasks into at least one task group according to the arrival time of the multiple parameter processing tasks, the parameter node 20 may divide the multiple parameter processing tasks into at least one task group according to the window identifiers corresponding to each of the multiple parameter processing tasks.

[0047] In some optional embodiments, when the parameter node 20 groups multiple parameter processing tasks, the parameter processing tasks with the same window identifier may be grouped into one task group. This embodiment is applicable to the case where the number of parameter processing tasks in the task queue of the parameter node 20 is small.

[0048] In other optional embodiments, when the parameter node 20 groups and divides multiple parameter processing tasks, if the number of the multiple parameter processing tasks is greater than a set quantity threshold, then for any time window in the multiple time windows, the parameter node 20 may determine a specified number (K) of time windows after the time window, and divide the parameter processing tasks having the window identifiers corresponding to the time window and the specified number of time windows into a task group. The set quantity threshold may be set according to the number of parameter processing tasks per unit time during the asynchronous training of the model, and this embodiment does not impose any restrictions. The value of K may be set according to the number of time windows spanned by the parameter processing tasks in the task queue and a preset upper limit value. For example, continuing with Figure 3 Taking the illustrated time window as an example, if the number of request tasks in the task queue is greater than a certain number threshold, the parameter processing tasks with window identifiers corresponding to the time window t1 and the time window t2 can be divided into the same task group.

[0049] In this implementation, when the number of multiple parameter processing tasks is large, the tasks arranged in the task queue will span multiple time windows. When grouping multiple parameter processing tasks, the parameter processing tasks within the current time window and the K time windows after the current time window are divided into one task group, which realizes the flexible processing of parameter processing tasks across time windows, so that parameter processing tasks with relatively close arrival times can be processed synchronously, which is conducive to reducing the waiting time of the working nodes corresponding to the parameter processing tasks and improving the computing efficiency of the working nodes.

[0050] Optionally, for any task group, the parameter node 20 can sort the parameter processing tasks in the task group in ascending or descending order according to the task identifier of the parameter processing tasks in the task group. For example, when the task identifier is implemented as an ID carried by the parameter processing task, ID ascending or ID descending sorting can be performed in any task group, so that the parameter processing tasks of the same working node in the task group are arranged close to each other, which is convenient for synchronous execution of the parameter processing tasks of the same working node or shortening the difference in execution time of the parameter processing tasks of the same working node.

[0051] The following will be combined with Figure 4 , the functions of the parameter nodes are further illustrated.

[0052] like Figure 4 As shown, any parameter node may include the following modules: a network transceiver module, which is used to receive the parameter processing tasks sent by the working node side and send the task processing results to the working node; a task queue, which is used to put the received parameter processing tasks of the working node into the task queue, waiting for the task scheduling module to schedule and execute; a task scheduling module, which is used to schedule tasks to be executed according to a preset algorithm; a task execution module, which is responsible for executing the tasks in the task queue according to the instructions issued by the task scheduling module.

[0053] Among them, the scheduling algorithm executed by the task scheduling module mainly includes the following steps:

[0054] S1. Receive parameter processing tasks from working nodes. Each parameter node carries the task ID corresponding to the working node.

[0055] Among them, each working node independently initiates parameter processing tasks to multiple parameter nodes. Each parameter processing task is accompanied by a random task ID, and the task ID meets the following requirements: parameter processing tasks of different working nodes carry different task IDs; parameter processing tasks of the same round of training of the same working node carry the same task ID; parameter processing tasks of different rounds of training of the same working node carry different task IDs.

[0056] S2. Add the received parameter processing task to the task queue.

[0057] S3. Reorder the parameter processing tasks in the task queue and execute the parameter processing tasks according to the ordering results.

[0058] The sorting process is as follows:

[0059] S31, the time axis is divided into time windows (time_window) according to fixed time intervals, and each (time_window) is numbered in an increasing sequence, and the number is recorded as tid. The size of the time window can be flexibly set according to actual conditions.

[0060] S32: When a new parameter processing task arrives, the tid of the current time window is marked on the parameter processing task.

[0061] S33. At the end of each time window, the parameter processing tasks falling in the same time window are sorted in ascending or descending order according to the task IDs they carry.

[0062] S4. The task scheduling module executes the parameter processing tasks in the task queue in sequence according to the sorted order.

[0063] When there are many requests in the task queue, the tasks sorted in the task queue will span multiple time windows. The parameter node can include tasks within K time windows after the current time window in the sorting of the current time window. The K value can be flexibly set according to the actual situation.

[0064] Repeat the above steps until the asynchronous training process of the model is completed.

[0065] Based on this implementation, the parameter processing tasks received by the parameter nodes can be scheduled without relying on global synchronization, allowing the parameter nodes to complete the requests of the same working nodes as synchronously as possible, shortening the request response time and improving the overall training speed.

[0066] Optionally, the deep learning models applicable to the embodiments of the present application may include, but are not limited to, one or more of convolutional neural networks (CNN), deep neural networks (DNN), graph convolutional neural networks (GCN), recurrent neural networks (RNN), and long short-term memory neural networks (LSTM), or other models that can be deformed from one or more of the above neural networks, which are not limited in this embodiment.

[0067] In addition to the task scheduling system for asynchronous training described in the aforementioned embodiments, the embodiments of the present application also provide a task scheduling method for asynchronous training, which will be exemplarily described below.

[0068] Figure 5 A flowchart of a task scheduling method for asynchronous training provided by an exemplary embodiment of the present application is shown in FIG. Figure 5 As shown, the method includes:

[0069] Step 501, determine multiple parameter processing tasks in a task queue; the multiple parameter processing tasks are sent by at least one working node; wherein the parameter processing task sent by any working node carries a task identifier corresponding to the working node.

[0070] Step 502: Schedule the execution order of the multiple parameter processing tasks according to the task identifiers of the multiple parameter processing tasks and the time information of the multiple parameter processing tasks to obtain the execution order of the multiple parameter processing tasks.

[0071] Step 503: Execute the multiple parameter processing tasks in sequence according to the execution order of the multiple parameter processing tasks.

[0072] In some exemplary embodiments, a method for scheduling the execution order of the multiple parameter processing tasks according to the task identifiers of the multiple parameter processing tasks and the time information of the multiple parameter processing tasks includes: dividing the multiple parameter processing tasks into at least one task group according to the arrival time of the multiple parameter processing tasks; for any task group, sorting the parameter processing tasks in the task group according to the task identifiers of the parameter processing tasks in the task group to obtain the execution order of the parameter processing tasks in the task group.

[0073] In some exemplary embodiments, before dividing the multiple parameter processing tasks into at least one task group according to the time period in which the arrival times of the multiple parameter processing tasks fall, it also includes: dividing the time axis according to the set time interval to obtain multiple time windows; the time interval is associated with the request frequency of the at least one working node; for any parameter processing task, when the parameter processing task arrives, according to the arrival time of the parameter processing task, determining the target time window corresponding to the parameter processing task from the multiple time windows; adding a window identifier corresponding to the target time window to the parameter processing task; accordingly, a method of dividing the multiple parameter processing tasks into at least one task group according to the arrival times of the multiple parameter processing tasks may include: dividing the multiple parameter processing tasks into at least one task group according to the window identifiers corresponding to each of the multiple parameter processing tasks.

[0074] In some exemplary embodiments, a method of dividing the multiple parameter processing tasks into at least one task group according to the window identifiers corresponding to the multiple parameter processing tasks may include: dividing the parameter processing tasks with the same window identifier into one task group.

[0075] In some exemplary embodiments, a method of dividing the multiple parameter processing tasks into at least one task group according to the window identifiers corresponding to each of the multiple parameter processing tasks may include: for any time window among the multiple time windows, if the number of tasks corresponding to the task queue is greater than a set number threshold, determining a specified number of time windows after the time window; and dividing the parameter processing tasks having the window identifiers corresponding to the time window and the specified number of time windows into a task group.

[0076] In some exemplary embodiments, the parameter processing tasks in the task group are sorted according to the task identifiers of the parameter processing tasks in the task group, including: sorting the parameter processing tasks in the task group in ascending order or descending order according to the identifiers of the parameter processing tasks in the task group.

[0077] In some exemplary embodiments, among the at least one working node, the task identifiers of the parameter processing tasks sent by any working node in different training rounds are different, and the task identifiers of the parameter processing tasks sent in the same training round are the same.

[0078] In this embodiment, after the parameter node receives multiple parameter processing tasks corresponding to at least one working node, the execution order of the multiple parameter processing tasks can be scheduled according to the task identifiers of the multiple parameter processing tasks and the time information of the multiple parameter processing tasks, and the multiple parameter processing tasks can be executed in sequence according to the scheduling results. Among them, the task identifier of the parameter processing task corresponds to the source working node of the parameter processing task. In this embodiment, when scheduling the execution order of multiple parameter processing tasks, the source working node of the parameter processing task and the time information of the parameter processing task are considered at the same time, so that the parameter node can be scheduled to process the parameter processing tasks of the same working node in a timely manner within the same or similar time range as much as possible, thereby reducing the waiting time of the working node and improving the computing efficiency of the working node.

[0079] It should be noted that in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or may be executed in parallel. The serial numbers of the operations, such as 501, 502, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. The execution subjects of each step of the method provided in the above embodiments can be the same device, or the method can also be executed by different devices. For example, the execution subject of steps 501 to 504 can be device A; for another example, the execution subject of steps 501 and 502 can be device A, and the execution subject of steps 503 and 504 can be device B; and so on.

[0080] In addition, these processes may include more or fewer operations, and these operations may be performed sequentially or in parallel. It should be noted that the descriptions of "first", "second", etc. in this document are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do they limit "first" and "second" to different types.

[0081] Figure 6 A schematic diagram of the structure of a server device provided by an exemplary embodiment of the present application is shown in FIG. Figure 6 As shown, the server device may include: a memory 601 , a processor 602 , a communication component 603 , and a power supply component 604 . Figure 6 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 6 Components shown.

[0082] The memory 601 may be configured to store various other data to support operations on the parameter node 20. Examples of such data include instructions for any application or method operating on the parameter node 20, contact data, phone book data, messages, pictures, videos, etc. The memory may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0083] In this embodiment, the memory 601 is used to store one or more computer instructions.

[0084] Processor 602, coupled to memory 601, is used to execute one or more computer instructions in memory 601, so as to: determine multiple parameter processing tasks in a task queue; the multiple parameter processing tasks are sent by at least one working node; wherein the parameter processing tasks sent by any working node carry a task identifier corresponding to the working node; according to the respective task identifiers of the multiple parameter processing tasks and the respective time information of the multiple parameter processing tasks, the execution order of the multiple parameter processing tasks is scheduled to obtain the execution order of the multiple parameter processing tasks; according to the execution order of the multiple parameter processing tasks, the multiple parameter processing tasks are executed in sequence.

[0085] Further optionally, when the processor 602 schedules the execution order of the multiple parameter processing tasks according to the task identifiers of the multiple parameter processing tasks and the time information of the multiple parameter processing tasks, it is specifically configured to:

[0086] Dividing the plurality of parameter processing tasks into at least one task group according to arrival times of the plurality of parameter processing tasks;

[0087] For any task group, the parameter processing tasks in the task group are sorted according to the task identifiers of the parameter processing tasks in the task group to obtain the execution order of the parameter processing tasks in the task group.

[0088] Further optionally, before the processor 602 divides the multiple parameter processing tasks into at least one task group according to the time period in which the arrival time of the multiple parameter processing tasks is located, it is also used to: divide the time axis according to the set time interval to obtain multiple time windows; the time interval is associated with the request frequency of the at least one working node; for any parameter processing task, when the parameter processing task arrives, according to the arrival time of the parameter processing task, determine the target time window corresponding to the parameter processing task from the multiple time windows; add a window identifier corresponding to the target time window to the parameter processing task. Accordingly, when the processor 602 divides the multiple parameter processing tasks into at least one task group according to the arrival time of the multiple parameter processing tasks, it is specifically used to: divide the multiple parameter processing tasks into at least one task group according to the window identifiers corresponding to each of the multiple parameter processing tasks.

[0089] Further optionally, when the processor 602 divides the multiple parameter processing tasks into at least one task group according to the window identifiers corresponding to the multiple parameter processing tasks, the processor 602 is specifically configured to: divide the parameter processing tasks having the same window identifier into one task group.

[0090] Further optionally, when the processor 602 divides the multiple parameter processing tasks into at least one task group according to the window identifiers corresponding to each of the multiple parameter processing tasks, it is specifically used to: for any time window among the multiple time windows, if the number of tasks corresponding to the task queue is greater than a set number threshold, determine a specified number of time windows after the time window; and divide the parameter processing tasks having the window identifiers corresponding to the time window and the specified number of time windows into a task group.

[0091] Further optionally, when the processor 602 sorts the parameter processing tasks in the task group according to the task identifiers of the parameter processing tasks in the task group, it is specifically used to: sort the parameter processing tasks in the task group in ascending order or descending order according to the identifiers of the parameter processing tasks in the task group.

[0092] Further optionally, among the at least one working node, the task identifiers of the parameter processing tasks sent by any working node in different training rounds are different, and the task identifiers of the parameter processing tasks sent in the same training round are the same.

[0093] Among them, the communication component 603 is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component can be implemented based on near field communication (NFC) technology, radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0094] The power supply component 605 provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0095] In this embodiment, after the parameter node receives multiple parameter processing tasks corresponding to at least one working node, the execution order of the multiple parameter processing tasks can be scheduled according to the task identifiers of the multiple parameter processing tasks and the time information of the multiple parameter processing tasks, and the multiple parameter processing tasks can be executed in sequence according to the scheduling results. Among them, the task identifier of the parameter processing task corresponds to the source working node of the parameter processing task. In this embodiment, when scheduling the execution order of multiple parameter processing tasks, the source working node of the parameter processing task and the time information of the parameter processing task are considered at the same time, so that the parameter node can be scheduled to process the parameter processing tasks of the same working node in a timely manner within the same or similar time range as much as possible, thereby reducing the waiting time of the working node and improving the computing efficiency of the working node.

[0096] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement each step of the method embodiment that can be executed by the server in the above method embodiment.

[0097] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0098] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0099] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0101] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0102] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0103] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0104] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0105] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A task scheduling method for asynchronous training, It is characterized in that include: Determine multiple parameter processing tasks in the task queue; The multiple parameter processing tasks are sent by at least one working node; wherein the parameter processing task sent by any working node carries a task identifier corresponding to the working node; According to the arrival time of the plurality of parameter processing tasks, the plurality of parameter processing tasks are divided into at least one task group; for any task group, the parameter processing tasks in the task group are sorted according to the identifiers of the parameter processing tasks in the task group to obtain the execution order of the parameter processing tasks in the task group; The multiple parameter processing tasks are executed in sequence according to the execution order of the multiple parameter processing tasks.

2. The method according to claim 1, It is characterized in that Before dividing the multiple parameter processing tasks into at least one task group according to the arrival time of the multiple parameter processing tasks, the method further includes: dividing the time axis according to a set time interval to obtain a plurality of time windows; the time interval is associated with the request frequency of the at least one working node; for any parameter processing task, when the parameter processing task arrives, according to the arrival time of the parameter processing task, determining a target time window corresponding to the parameter processing task from the multiple time windows, and adding a window identifier corresponding to the target time window to the parameter processing task; According to the arrival time of the multiple parameter processing tasks, the multiple parameter processing tasks are divided into at least one task group, including: according to the window identifiers corresponding to each of the multiple parameter processing tasks, the multiple parameter processing tasks are divided into at least one task group.

3. The method according to claim 2, It is characterized in that According to the window identifiers corresponding to the plurality of parameter processing tasks, the plurality of parameter processing tasks are divided into at least one task group, including: Parameter processing tasks with the same window identifier are divided into a task group.

4. The method according to claim 2, It is characterized in that According to the window identifiers corresponding to the plurality of parameter processing tasks, the plurality of parameter processing tasks are divided into at least one task group, including: If the number of tasks corresponding to the task queue is greater than a set number threshold, then for any time window among the multiple time windows, determining a specified number of time windows after the time window; The parameter processing tasks having the time window and the window identifiers corresponding to each of the specified number of time windows are divided into a task group.

5. The method according to any one of claims 2 to 4, It is characterized in that Sorting the parameter processing tasks in the task group according to the task identifiers of the parameter processing tasks in the task group includes: The parameter processing tasks in the task group are sorted in ascending order or descending order according to the identifiers of the parameter processing tasks in the task group.

6. The method according to any one of claims 1 to 4, It is characterized in that Among the at least one working node, the task identifiers of the parameter processing tasks sent by any working node in different training rounds are different, and the task identifiers of the parameter processing tasks sent in the same training round are the same.

7. A task scheduling system for asynchronous training, It is characterized in that include: At least one working node and parameter node; The at least one working node is used to: send a parameter processing task to the parameter node; The parameter processing task sent by any working node carries the task identifier corresponding to the working node; The parameter node is used to: determine multiple parameter processing tasks in a task queue; the multiple parameter processing tasks are sent by at least one working node; divide the multiple parameter processing tasks into at least one task group according to the arrival time of the multiple parameter processing tasks; for any task group, sort the parameter processing tasks in the task group according to the identifiers of the parameter processing tasks in the task group to obtain the execution order of the parameter processing tasks in the task group; and execute the multiple parameter processing tasks in sequence according to the execution order of the multiple parameter processing tasks.

8. A server device, It is characterized in that include: Memory, processor, and communication components; The memory is used to store one or more computer instructions; The processor is used to execute one or more computer instructions to execute the steps in the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, It is characterized in that When the computer program is executed, the steps in the method according to any one of claims 1 to 6 can be implemented.