A distributed training method, device and equipment based on communication optimization

Abstract

The present disclosure provides a communication optimization-based distributed training method, device and equipment, which can be applied to the field of machine learning. The method comprises: obtaining a training task with a waiting time greater than or equal to a first preset threshold; processing first transceiving data volume information between an initial parameter server and a worker node based on a first target optimization function to obtain a first parameter ratio; determining at least one target parameter server from a plurality of initial parameter servers according to the first parameter ratio of each initial parameter server; processing second transceiving data volume information between the target parameter server and the worker node based on a second target optimization function to obtain a second parameter ratio and a minimum transmission time; generating a first bandwidth value, a second bandwidth value and a remaining completion time according to the second parameter ratio and the minimum transmission time of the target parameter server; and obtaining a bandwidth allocation scheme based on a minimum remaining completion time priority principle to execute the training task to obtain a training result.

Description

Technical Field

[0001] This disclosure relates to the field of machine learning, and specifically to a distributed training method, apparatus, and device based on communication optimization. Background Technology

[0002] With the continuous advancement of machine learning technology, training ultra-large-scale machine learning models has exceeded the capabilities of single machines, and distributed machine learning has begun to play a crucial role. Among these, the parameter server architecture is one of the most commonly used communication architectures. A parameter server architecture comprises multiple nodes with different functions, forming a distributed training architecture for training machine learning models. Nodes in this distributed training architecture can perform training tasks for the machine learning model through data interaction, thereby obtaining the trained model. During this distributed training process, the data transmission speed between nodes becomes a key factor affecting the training efficiency of the machine learning model. Summary of the Invention

[0003] In view of the above problems, this disclosure provides a distributed training method, apparatus and device based on communication optimization.

[0004] According to a first aspect of this disclosure, a distributed training method based on communication optimization is provided, comprising:

[0005] In response to a data processing request, acquire training tasks with a waiting time greater than or equal to a first preset threshold;

[0006] For each training task, the first data transmission and reception information between the initial parameter server and the worker node is processed based on the first objective optimization function to obtain the first parameter ratio of the initial parameter server. The initial parameter server and the worker node are configured to perform at least one training round for each training task. The first parameter ratio represents the ratio between the parameters held by the initial parameter server and the total parameters of the trained model.

[0007] Based on the first parameter ratio of each of the multiple initial parameter servers, at least one target parameter server is determined from the multiple initial parameter servers;

[0008] For each training task, the second data transmission and reception information between the target parameter server and the worker node is processed based on the second objective optimization function to obtain the second parameter ratio of the target parameter server and the minimum transmission duration of the transmission sub-round, where the transmission sub-round represents the data transmission process in the training round.

[0009] For each training task, a first bandwidth value, a second bandwidth value, and a remaining completion time are generated based on the second parameter ratio of the target parameter server and the minimum transmission time. The first bandwidth value represents the bandwidth resources corresponding to the target parameter server, and the second bandwidth value represents the bandwidth resources corresponding to the worker node.

[0010] The bandwidth scheme is obtained by processing the remaining completion time of each training task based on the principle of prioritizing the minimum remaining completion time; and

[0011] Bandwidth resources are allocated to the target parameter server and worker nodes according to the bandwidth allocation scheme to facilitate the execution of training tasks and obtain training results.

[0012] According to embodiments of this disclosure, for each training task, the first data transmission and reception volume information between the initial parameter server and the worker node is processed based on a first objective optimization function to obtain the first parameter ratio of the initial parameter server, including:

[0013] For each training task, the data transmission and reception information of multiple initial parameter servers and multiple worker nodes are processed based on the first constraint condition corresponding to the first objective optimization function to obtain the first parameter ratio, wherein the first constraint condition includes at least one of the following:

[0014] The first constraint condition includes a first constraint sub-condition, which indicates that the amount of data sent by each working node is less than or equal to the rated amount of data sent by each working node. The rated amount of data sent by each working node is obtained based on the product between the node's sending bandwidth and the intermediate transmission time. The optimization objective of the first objective optimization function is to minimize the intermediate transmission time.

[0015] The first constraint condition includes a second constraint subcondition, which indicates that the amount of data received by each working node is less than or equal to the rated amount of data received by each working node. The rated amount of data received by each working node is obtained based on the product between the node's receiving bandwidth and the intermediate transmission duration.

[0016] The first constraint condition includes a third constraint subcondition, which indicates that the amount of data received by each initial parameter server is less than or equal to the rated amount of data received by each initial parameter server. The rated amount of data received by the initial parameter server is obtained based on the product between the receiving bandwidth of the initial parameter server and the intermediate transmission time.

[0017] The first constraint condition includes a fourth constraint subcondition, which indicates that the amount of data sent by each initial parameter server is less than or equal to the rated amount of data sent by each initial parameter server. The rated amount of data sent by the initial parameter server is obtained based on the product between the initial parameter server's sending bandwidth and the intermediate transmission duration.

[0018] According to embodiments of this disclosure, determining at least one target parameter server from a plurality of initial parameter servers based on a first parameter ratio of each initial parameter server includes:

[0019] The multiple first parameter ratios are sorted from largest to smallest according to the values ​​represented by the first parameter ratios to obtain the first parameter ratio sequence;

[0020] The first parameter proportions of the top n items in the sorted first parameter proportion sequence are determined as the target first parameter proportions, where n ≥ 1; and

[0021] The target parameter server is determined from multiple initial parameter servers based on the target first parameter ratio.

[0022] According to embodiments of this disclosure, for each training task, the second data transmission and reception volume information between the target parameter server and the worker node is processed based on the second objective optimization function to obtain the second parameter ratio of the target parameter server and the minimum transmission duration of the transmission sub-round, including:

[0023] For each training task, the data transmission and reception information of multiple target parameter servers and multiple worker nodes are processed based on the second constraint conditions corresponding to the second objective optimization function to obtain the second parameter ratio, wherein the second constraint conditions include at least one of the following:

[0024] The second constraint includes the fifth constraint sub-condition, which indicates that the amount of data sent by each working node is less than or equal to the rated amount of data sent by each working node. The rated amount of data sent by each working node is obtained based on the product of the node's sending bandwidth and the transmission duration. The optimization objective of the second objective optimization function is to minimize the transmission duration.

[0025] The second constraint includes a sixth constraint sub-condition, which indicates that the amount of data received by each working node is less than or equal to the rated amount of data received by each working node. The rated amount of data received by a working node is obtained based on the product of the node's receiving bandwidth and the transmission duration.

[0026] The second constraint includes the seventh constraint sub-condition, which indicates that the amount of data received by each target parameter server is less than or equal to the rated amount of data received by each target parameter server. The rated amount of data received by the target parameter server is obtained based on the product of the target parameter server's receiving bandwidth and transmission duration.

[0027] The second constraint includes the eighth constraint sub-condition, which indicates that the amount of data sent by each target parameter server is less than or equal to the rated amount of data sent by each target parameter server. The rated amount of data sent by the target parameter server is obtained based on the product of the target parameter server's sending bandwidth and transmission duration.

[0028] According to embodiments of this disclosure, for each training task, a first bandwidth value, a second bandwidth value, and a remaining completion time are generated based on the second parameter ratio of the target parameter server and the minimum transmission duration, including:

[0029] For each training task, the amount of data in the transmission sub-round is generated according to the proportion of the second parameter of the target parameter server;

[0030] A first bandwidth value and a second bandwidth value are generated based on the amount of data received in the transmission sub-round and the minimum transmission duration.

[0031] The remaining completion time is generated based on the minimum transmission duration of the transmission sub-round.

[0032] According to embodiments of this disclosure, the remaining completion time is generated based on the minimum transmission duration of the transmission sub-round, including:

[0033] For each training task, the number of remaining training rounds is obtained based on the target number of training rounds.

[0034] For each training task, the remaining completion time for the current round is obtained based on the amount of data remaining in the current round.

[0035] For each training task, the remaining completion time is generated based on the number of remaining training rounds, the remaining completion time of the current round, and the minimum transmission time of the transmission sub-rounds.

[0036] According to embodiments of this disclosure, for each training task, the remaining number of training rounds is obtained based on the target number of training rounds, including:

[0037] When the target number of training rounds is unknown, the loss function is used to process the number of historical training rounds to generate a loss function curve.

[0038] If the loss function curve converges, the target number of training rounds is determined based on the loss function curve; and

[0039] The number of remaining training rounds is obtained based on the target number of training rounds.

[0040] According to embodiments of this disclosure, the remaining completion time of each training task is processed based on the principle of prioritizing the minimum remaining completion time to obtain a bandwidth allocation scheme, including:

[0041] Based on the principle of prioritizing the minimum remaining completion time, the first training task is determined from multiple training tasks according to their respective remaining completion times.

[0042] A bandwidth allocation scheme is obtained based on the first bandwidth value and the second bandwidth value corresponding to the first training task.

[0043] A second aspect of this disclosure provides a communication-optimized distributed training apparatus, comprising:

[0044] The acquisition module is used to acquire training tasks with a waiting time greater than or equal to a first preset threshold in response to data processing requests.

[0045] The first processing module is used to process the first data transmission and reception volume information between the initial parameter server and the worker node based on the first objective optimization function for each training task, and obtain the first parameter ratio of the initial parameter server. The initial parameter server and the worker node are configured to perform at least one training round for each training task. The first parameter ratio represents the ratio between the parameters held by the initial parameter server and the total parameters of the trained model.

[0046] The first determining module is used to determine at least one target parameter server from the multiple initial parameter servers according to the first parameter ratio of each of the multiple initial parameter servers.

[0047] The second processing module is used to process the second data transmission and reception volume information between the target parameter server and the worker node based on the second objective optimization function for each training task, so as to obtain the second parameter ratio of the target parameter server and the minimum transmission duration of the transmission sub-round, wherein the transmission sub-round represents the data transmission process in the training round.

[0048] The generation module is used to generate a first bandwidth value, a second bandwidth value, and a remaining completion time for each training task, based on the second parameter ratio of the target parameter server and the minimum transmission time. The first bandwidth value represents the bandwidth resources corresponding to the target parameter server, and the second bandwidth value represents the bandwidth resources corresponding to the worker node.

[0049] The second determining module is used to process the remaining completion time of each training task based on the principle of prioritizing the minimum remaining completion time, and obtain the bandwidth allocation scheme.

[0050] The training module is used to allocate bandwidth resources to the target parameter server and worker nodes according to the bandwidth allocation scheme, so as to execute training tasks and obtain training results.

[0051] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the aforementioned communication-optimized distributed training method.

[0052] According to the distributed training method, apparatus, and device based on communication optimization provided in this disclosure, by acquiring training tasks in the entire machine learning cluster that are greater than or equal to a first preset threshold, the first objective optimization function is used to solve for the first parameter ratio held by the initial parameter server, thereby calculating the target parameter server required for each training task; then, by combining the second objective optimization function to solve for the second parameter ratio held by the target parameter server and the minimum transmission duration of the transmission sub-round, the bandwidth resources allocated to it are calculated, and the remaining completion time of each training task is obtained based on the minimum transmission duration of the transmission sub-round, thereby determining the priority task bandwidth allocation scheme based on the minimum remaining completion time priority principle. Since the training tasks in the entire machine learning cluster are scheduled, the overall optimization of the execution order of each training task is achieved, and resources are allocated according to this order. Simultaneously, by combining the scheduling method of the objective optimization function, the parameter ratio held by each parameter server is dynamically adjusted, allowing parameter servers with more bandwidth resources to hold more parameters, reducing communication time, and solving the problem of minimizing the average task completion time of the machine learning cluster. Attached Figure Description

[0053] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0054] Figure 1 The illustration schematically shows a principle diagram of implementing a communication-optimized distributed training method according to an embodiment of the present disclosure;

[0055] Figure 2 A flowchart illustrating a communication-optimized distributed training method according to an embodiment of the present disclosure is shown schematically.

[0056] Figure 3 A flowchart illustrating a method for generating remaining completion time according to an embodiment of the present disclosure is shown schematically.

[0057] Figure 4 A schematic diagram illustrates a structural block diagram of a communication-optimized distributed training device according to embodiments of the present disclosure; and

[0058] Figure 5 The diagram illustrates an electronic device suitable for implementing a communication-optimized distributed training method according to embodiments of the present disclosure. Detailed Implementation

[0059] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0060] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0061] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0062] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0063] In implementing this disclosure, it was discovered that a parameter server architecture can contain multiple nodes with different functions: parameter servers and worker nodes. Due to limitations in computing and network resources, this architecture requires the introduction of multiple parameter servers to share the burden of computing and network resources. However, the communication time between worker nodes and parameter servers does not decrease indefinitely with the increase in the number of parameter servers, hindering the speed of model training. In cloud platform training services provided by relevant organizations, only the data transmission acceleration of a single training task is generally focused on, while neglecting to reduce the average completion time of tasks, thus failing to improve the utilization rate of the entire cluster resources. Therefore, how to optimize the task scheduling process of the entire distributed machine learning cluster is an urgent problem to be solved.

[0064] In view of this, embodiments of this disclosure provide a distributed training method, apparatus, and device based on communication optimization. The method includes: in response to a data processing request, acquiring training tasks with a waiting time greater than or equal to a first preset threshold; for each training task, processing first data transmission and reception volume information between an initial parameter server and worker nodes based on a first objective optimization function to obtain a first parameter ratio of the initial parameter server, wherein the initial parameter server and worker nodes are configured to execute at least one training round for each training task, and the first parameter ratio characterizes the ratio between the parameters held by the initial parameter server and the total parameters of the trained model; determining at least one target parameter server from the plurality of initial parameter servers based on their respective first parameter ratios; and for each training task, processing the target parameter server and worker nodes based on a second objective optimization function. The second data transmission and reception information between the target parameter server and the minimum transmission duration of the transmission sub-round is obtained, where the transmission sub-round represents the data transmission process in the training round. For each training task, a first bandwidth value, a second bandwidth value, and a remaining completion duration are generated based on the second parameter ratio of the target parameter server and the minimum transmission duration. The first bandwidth value represents the bandwidth resource corresponding to the target parameter server, and the second bandwidth value represents the bandwidth resource corresponding to the worker node. The remaining completion duration of each training task is processed based on the principle of prioritizing the minimum remaining completion duration to obtain a bandwidth allocation scheme. Bandwidth resources are allocated to the target parameter server and the worker node respectively according to the bandwidth allocation scheme to facilitate the execution of training tasks and obtain training results.

[0065] In the technical solution of this invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0066] Figure 1 The illustration schematically shows a principle diagram of implementing a communication-optimized distributed training method according to an embodiment of the present disclosure.

[0067] like Figure 1 As shown in the schematic diagram of this embodiment, it may include m+1 worker nodes and s+1 parameter servers. The specific steps for performing at least one training round include:

[0068] The working node m trains its own data based on the distributed machine learning model, that is, performs local training on data, for local model parameters {w1,w2,…,w n}, to obtain the corresponding model gradient

[0069] Working node m pushes the model gradient to all parameter servers, that is, each working node pushes the model gradient obtained from local training to all parameter servers.

[0070] Each parameter server is responsible for updating different model parameters, therefore, each parameter server only aggregates model gradients corresponding to the model parameters it is responsible for that are trained from each working node. For example, the model parameter that parameter server s is responsible for updating is {w ls+1 ,w ls+2 ,…,w n}, wherein 1<s<n, then parameter server s aggregates model gradients corresponding to the model parameters it is responsible for that are trained from m+1 working nodes and obtains through aggregation thereby updating the model parameters {w ls+1 ,w ls+2 ,…,w n}.

[0071] Each working node pulls the updated model parameters from all parameter servers, then updates the trained model.

[0072] Figure 2 schematically shows a flowchart of a communication optimization-based distributed training method according to an embodiment of the present disclosure.

[0073] As shown in Figure 2 , the method 200 includes operations S210 to S260.

[0074] In operation S210, in response to a data processing request, a training task whose waiting duration is greater than or equal to a first preset threshold is acquired.

[0075] In operation S220, for each training task, first transmission and reception data volume information between initial parameter servers and working nodes is processed based on a first objective optimization function to obtain a first parameter ratio of the initial parameter servers, wherein the initial parameter servers and working nodes are configured to perform at least one training round for each training task, and the first parameter ratio characterizes the ratio between parameters held by the initial parameter servers and the total parameters of the model to be trained.

[0076] According to embodiments of this disclosure, a user can upload multiple datasets to an application platform. The application platform can be configured with an initial parameter server and worker nodes. The initial parameter server and worker nodes can use at least one of the multiple datasets to train a machine learning model. The training task can characterize the process of training the machine learning model.

[0077] According to embodiments of this disclosure, a first preset threshold represents the maximum time a user can wait, for example, the first preset threshold is 1 day. Multiple training tasks with a waiting time greater than or equal to 1 day are obtained from the application platform, and these tasks are prioritized for training. When no training task with a waiting time greater than or equal to the first preset threshold exists, all tasks with a waiting time less than the first preset threshold are directly scheduled for training.

[0078] According to embodiments of this disclosure, during the training of a machine learning model, the training task can be performed based on a parameter server architecture containing two types of distributed nodes with different functions. These two types of distributed nodes may include, for example, an initial parameter server and worker nodes. Parallel computation by multiple initial parameter servers and multiple worker nodes can accelerate the training process of the machine learning task. Data communication is required between the initial parameter server and the worker nodes to synchronize the model parameters.

[0079] According to embodiments of this disclosure, the entire machine learning cluster may contain multiple training tasks, each training task may contain multiple training rounds, and multiple initial parameter servers and multiple worker nodes may be configured to perform at least one training round for each training task.

[0080] According to embodiments of this disclosure, the training process of a training round in a parameter server architecture can include four steps: first, each worker node calculates the model gradient based on the computation results output by the machine learning model; second, each worker node pushes the model gradient to the initial parameter server; third, the initial parameter server aggregates the model gradients from all worker nodes and updates the model parameters; and fourth, each worker node pulls the updated model parameters from the initial parameter server and then updates the model being trained.

[0081] According to embodiments of this disclosure, the data transmission and reception volume information represents the amount of model gradients sent and parameters received by each worker node in a training round, and the amount of model gradients received and parameters sent by each initial parameter server.

[0082] According to embodiments of this disclosure, the first parameter ratio represents the proportion of model parameters held by the initial parameter server, that is, the ratio between the parameters held by the initial parameter server and the total parameters of the trained model, and multiple initial parameter servers correspond to multiple first parameter ratios.

[0083] According to embodiments of this disclosure, a machine cluster may contain multiple physical servers, with each initial parameter server placed on a corresponding physical server. The physical server is used to perform gradient aggregation and parameter update operations on the parameter portion held by the initial parameter server.

[0084] According to embodiments of this disclosure, the multiple physical servers in the machine cluster are heterogeneous and can have different inbound and outbound bandwidths. Therefore, the selection of which parameter servers to perform gradient aggregation and parameter updates among multiple initial parameter servers will affect the communication time. That is, when the initial parameter server with more available inbound and outbound bandwidth resources holds more parameters, the communication time can be reduced to a greater extent and the data transmission efficiency can be improved.

[0085] In operation S230, at least one target parameter server is determined from the multiple initial parameter servers based on the proportion of their respective first parameters.

[0086] According to embodiments of this disclosure, based on the first parameter ratio of each of the multiple initial parameter servers, n parameter servers required for this training task are selected from the multiple initial parameter servers to obtain n target parameter servers, wherein the n parameter servers are respectively placed on n physical servers.

[0087] In operation S240, for each training task, the second data transmission and reception volume information between the target parameter server and the worker node is processed based on the second objective optimization function to obtain the second parameter ratio of the target parameter server and the minimum transmission duration of the transmission sub-round, wherein the transmission sub-round represents the data transmission process in the training round.

[0088] According to embodiments of this disclosure, in a single training round, for the parameter server, the reception of model gradients and the transmission of model parameters can be performed simultaneously. Therefore, the duration of a single training round can be jointly determined by the maximum gradient push duration and the maximum parameter retrieval duration of the worker node.

[0089] According to embodiments of this disclosure, a transmission sub-round can be a process of gradient pushing or parameter fetching by a worker node, and the duration of gradient pushing and parameter fetching cannot exceed the transmission duration of the transmission sub-round.

[0090] According to embodiments of this disclosure, the minimum transmission duration of a transmission sub-round is obtained, which reduces the data transmission communication time, thereby enabling the minimum transmission duration of a single training round to be obtained.

[0091] In operation S250, for each training task, a first bandwidth value, a second bandwidth value, and a remaining completion time are generated based on the second parameter ratio of the target parameter server and the minimum transmission time. The first bandwidth value represents the bandwidth resources corresponding to the target parameter server, and the second bandwidth value represents the bandwidth resources corresponding to the worker node.

[0092] According to embodiments of this disclosure, a first bandwidth value characterizes the available inbound and outbound bandwidth resources required by the server in a transmission sub-round.

[0093] According to embodiments of this disclosure, a first bandwidth value characterizes the available inbound and outbound bandwidth resources required by this working node in a transmission sub-round.

[0094] According to embodiments of this disclosure, the remaining completion time represents the completion time of the remaining training rounds for each training task.

[0095] In operation S260, the remaining completion time of each training task is processed based on the principle of prioritizing the minimum remaining completion time, resulting in a bandwidth allocation scheme.

[0096] When operating S270, bandwidth resources are allocated to the target parameter server and worker nodes according to the bandwidth allocation scheme to facilitate the execution of training tasks and obtain training results.

[0097] According to embodiments of this disclosure, task scheduling follows the principle of prioritizing tasks with the shortest remaining completion time. For a task with the shortest remaining completion time, if there are sufficient available resources in the machine learning cluster, the scheduler prioritizes allocating corresponding bandwidth resources to this task to enable it to complete the remaining training rounds.

[0098] According to embodiments of this disclosure, by acquiring training tasks in the entire machine learning cluster that are greater than or equal to a first preset threshold, the first objective optimization function is used to solve for the first parameter ratio held by the initial parameter server, thereby calculating the target parameter server subset required for each training task; then, the second objective optimization function is combined to solve for the second parameter ratio held by the target parameter server and the minimum transmission duration of the transmission sub-round, and the bandwidth resources allocated to it are calculated. At the same time, the remaining completion time of each training task is obtained according to the minimum transmission duration of the transmission sub-round, thereby determining the priority task bandwidth allocation scheme based on the minimum remaining completion time priority principle.

[0099] According to the embodiments of this disclosure, by scheduling the training tasks in the entire machine learning cluster, the overall execution order of each training task is optimized and resources are allocated in this order. At the same time, by combining the scheduling method of the objective optimization function, the proportion of parameters held by each parameter server is dynamically adjusted, so that the parameter server with more bandwidth resources holds more parameters, reducing communication time and solving the problem of minimizing the average task completion time of the machine learning cluster.

[0100] According to embodiments of this disclosure, for each training task, the first data transmission and reception volume information between the initial parameter server and the worker node is processed based on a first objective optimization function to obtain the first parameter ratio of the initial parameter server, including:

[0101] For each training task, the data transmission and reception information of multiple initial parameter servers and multiple worker nodes are processed based on the first constraint condition corresponding to the first objective optimization function to obtain the first parameter ratio. The first constraint condition includes at least one of the following: first constraint sub-condition, second constraint sub-condition, third constraint sub-condition, and fourth constraint sub-condition.

[0102] The first constraint sub-condition indicates that the amount of data sent by each working node is less than or equal to the rated amount of data sent by each working node. The rated amount of data sent by each working node is obtained based on the product of the node's sending bandwidth and the intermediate transmission time. The optimization objective of the first objective optimization function is to minimize the intermediate transmission time.

[0103] The second constraint subcondition indicates that the amount of data received by each working node is less than or equal to the rated amount of data received by each working node, which is obtained based on the product of the node's receiving bandwidth and the intermediate transmission time.

[0104] The third constraint subcondition indicates that the amount of data received by each initial parameter server is less than or equal to the rated amount of data received by each initial parameter server. The rated amount of data received by the initial parameter server is obtained based on the product of the initial parameter server's receiving bandwidth and the intermediate transmission time.

[0105] The fourth constraint subcondition indicates that the amount of data sent by each initial parameter server is less than or equal to the rated amount of data sent by each initial parameter server. The rated amount of data sent by the initial parameter server is obtained based on the product of the initial parameter server's sending bandwidth and the intermediate transmission time.

[0106] According to embodiments of this disclosure, for each training task, in one training round, the data transmission and reception information of multiple initial parameter servers and multiple worker nodes are processed based on the first constraint condition corresponding to the first objective optimization function to obtain the first parameter ratio, wherein the first constraint condition includes a first constraint sub-condition, a second constraint sub-condition, a third constraint sub-condition, and a fourth constraint sub-condition.

[0107] In one embodiment, the first sub-constraint 1a is formulated as shown in (1):

[0108]

[0109] Where u represents the amount of data for the model parameters, and W represents the set of worker nodes. T1 represents the node transmission bandwidth of worker node w, and T1 represents the intermediate transmission duration in the transmission sub-round.

[0110] According to an embodiment of this disclosure, the first constraint sub-condition 1a represents the amount of data that the working node w must send at least within time T1, which is u, i.e., the amount of data transmission generated when the working node sends the model gradient to all parameter servers.

[0111] The formula for the second sub-constraint 1b is shown in (2):

[0112]

[0113] Where u represents the amount of data for the model parameters, and W represents the set of worker nodes. The node receiving bandwidth represents the working node w.

[0114] According to an embodiment of this disclosure, the second constraint sub-condition 1b represents that within time T1, the minimum amount of parameter data that the working node w must receive is u, that is, the amount of parameter data generated when the working node receives all parameter servers.

[0115] According to the embodiments of this disclosure, each worker node sends the trained model gradients to all parameter servers. Since the model parameters that each parameter server is responsible for aggregating do not overlap, the amount of model gradient data sent by worker node w to all parameter servers in the transmission sub-round is u. Therefore, the amount of parameters received by worker node w in the transmission sub-round can be u.

[0116] In one embodiment, the third constraint sub-condition 1c is formulated as shown in (3):

[0117]

[0118] Among them, P s The initial parameter server s represents the proportion of model parameters held by the initial parameter server s, where S represents the set of multiple initial parameter servers, and m represents the number of worker nodes. Characterizes the receiving bandwidth of the initial parameter server s.

[0119] According to an embodiment of this disclosure, the third sub-constraint 1c represents that the initial parameter server s must receive at least umP of data within time T1. s This refers to the amount of data transmitted when the initial parameter server receives gradients from all worker nodes.

[0120] The formula for the fourth constraint subcondition 1d is shown in (4):

[0121]

[0122] Among them, P sThe initial parameter server s represents the proportion of model parameters held by the initial parameter server s, where S represents the set of multiple initial parameter servers, and m represents the number of worker nodes. Characterizes the sending bandwidth of the initial parameter server s.

[0123] According to an embodiment of this disclosure, the fourth constraint sub-condition 1d represents that the initial parameter server s must send at least umP of parameter data within time T1. s This refers to the amount of parameter data generated when the initial parameter server sends it to all worker nodes.

[0124] According to embodiments of this disclosure, in a parameter server architecture, each initial parameter server needs to send the parameters it is responsible for aggregating to all worker nodes. The initial parameter server s needs to send up to one worker node. s Given the model parameters of size, the amount of parameters sent by the initial parameter server s to all worker nodes in the transmission sub-round can be calculated as ump. s Therefore, the amount of model gradient data received by the initial parameter server s from all worker nodes during the transmission sub-round can be ump. s .

[0125] In one embodiment, the first objective optimization function is as shown in (5):

[0126] Minimize T1 (1)

[0127] Subject to:

[0128]

[0129] Among them, the first constraint subcondition 1a and the third constraint subcondition 1c are generated when the working node sends the model gradient to the initial parameter server; the second constraint subcondition 1b and the fourth constraint subcondition 1d are generated when the initial parameter server sends the model parameters back to the working node; constraint 1e represents that the sum of the proportions of all parameters held by the initial parameter servers is 1; constraint 1f indicates that the range of the proportions of parameters held by the initial parameter servers is [0,1].

[0130] According to embodiments of this disclosure, the objective (1) of the first objective optimization function is optimized and solved to obtain the first parameter ratios (p1,...,p) of each of the multiple initial parameter servers. s ,…,p S ).

[0131] According to embodiments of this disclosure, determining at least one target parameter server from a plurality of initial parameter servers based on a first parameter ratio of each initial parameter server includes:

[0132] Sort multiple first parameter ratios in descending order of their numerical values ​​to obtain a first parameter ratio sequence; determine the first parameter ratios that are the top n in the first parameter ratio sequence as the target first parameter ratios, where n≥1; and determine the target parameter server from multiple initial parameter servers based on the target first parameter ratios.

[0133] According to embodiments of this disclosure, the first parameter ratios (p1,...,p) of each of the multiple initial parameter servers are set to... s ,…,p s Sort the data from largest to smallest to obtain the first parameter proportion sequence.

[0134] According to embodiments of this disclosure, it is assumed that n parameter servers (1≤n≤S) are required for a training task.

[0135] According to the embodiments of this disclosure, the initial parameter servers corresponding to the top n of the first parameter ratio sequence are selected to obtain n target parameter servers, and the n target parameter servers are respectively placed on the corresponding n physical servers, and the physical servers perform gradient aggregation and parameter update operations on the parameter parts held by the target parameter servers.

[0136] According to embodiments of this disclosure, by solving the first objective optimization function, the first parameter ratios of each of the multiple initial parameter servers are obtained, and then n target parameter servers are obtained. That is, the parameter ratio held by each parameter server is dynamically adjusted, and the parameter server with more bandwidth resources holds more parameters. Based on this, the physical server where the target parameter server is placed performs parameter updates, etc., which can alleviate the data communication time bottleneck in the training task.

[0137] According to embodiments of this disclosure, for each training task, the second data transmission and reception volume information between the target parameter server and the worker node is processed based on the second objective optimization function to obtain the second parameter ratio of the target parameter server and the minimum transmission duration of the transmission sub-round, including:

[0138] For each training task, the data transmission and reception information of multiple target parameter servers and multiple worker nodes are processed based on the second constraint conditions corresponding to the second objective optimization function to obtain the second parameter ratio. The second constraint conditions include at least one of the following: the fifth constraint sub-condition, the sixth constraint sub-condition, the seventh constraint sub-condition, and the eighth constraint sub-condition.

[0139] The fifth constraint subcondition indicates that the amount of data sent by each working node is less than or equal to the rated amount of data sent by each working node. The rated amount of data sent by each working node is obtained based on the product of the node's sending bandwidth and the transmission duration. The optimization objective of the second objective function is to minimize the transmission duration.

[0140] The sixth constraint subcondition indicates that the amount of data received by each working node is less than or equal to the rated amount of data received by each working node. The rated amount of data received by a working node is obtained based on the product of the node's receiving bandwidth and the transmission duration.

[0141] The seventh constraint subcondition indicates that the amount of data received by each target parameter server is less than or equal to the rated amount of data received by each target parameter server. The rated amount of data received by the target parameter server is obtained based on the product of the target parameter server's receiving bandwidth and transmission duration.

[0142] The eighth constraint subcondition indicates that the amount of data sent by each target parameter server is less than or equal to the rated amount of data sent by each target parameter server. The rated amount of data sent by the target parameter server is obtained based on the product of the target parameter server's sending bandwidth and transmission duration.

[0143] According to embodiments of this disclosure, for each training task, in one training round, the data transmission and reception information of multiple target parameter servers and multiple worker nodes is processed based on the second constraint conditions corresponding to the second objective optimization function to obtain the second parameter ratio, wherein the second constraint conditions include the fifth constraint sub-condition, the sixth constraint sub-condition, the seventh constraint sub-condition, and the eighth constraint sub-condition.

[0144] In one embodiment, the fifth constraint subcondition 2a is formulated as shown in (6):

[0145]

[0146] in, T represents the node transmission bandwidth of worker node w, and T represents the transmission duration in the transmission sub-round.

[0147] The formula for the sixth constraint subcondition 2b is shown in (7):

[0148]

[0149] in, The node receive bandwidth representing the working node w

[0150] According to embodiments of this disclosure, multiple initial parameter servers are replaced with n target parameter servers, and parameters are updated based on the target parameter servers in the transmission sub-round to obtain the seventh constraint sub-condition and the eighth constraint sub-condition.

[0151] In one embodiment, the formula for the seventh constraint sub-condition 2c is as shown in (8):

[0152]

[0153] Among them, S aThe set of servers representing the n target parameters after the replacement. Characterizes the receiving bandwidth of the target parameter server s.

[0154] The formula for the eighth constraint subcondition 2d is shown in (9):

[0155]

[0156] Among them, S a The set of servers representing the n target parameters after the replacement. Characterizes the sending bandwidth of the target parameter server s.

[0157] In one embodiment, S in the first objective optimization function is replaced with S a The new second objective function is obtained as shown in (10):

[0158] Minimize T (2)

[0159] Subject to:

[0160]

[0161] According to embodiments of this disclosure, the minimum transmission duration of the transmission sub-round is solved based on the objective (2) of the second objective optimization function, and the proportion of the second parameter held by each of the multiple objective parameter servers is re-solved to obtain (p1,...,p s ,…,p n ).

[0162] According to embodiments of this disclosure, a parameter server is placed on each physical server, that is, the number of parameters held by the parameter server can be the number of parameters that the corresponding physical server needs to process.

[0163] According to embodiments of this disclosure, multiple initial parameter servers are replaced with n target parameter servers, and the optimality of solving for the minimum transmission duration of the transmission sub-round based on the objective (2) of the second objective optimization function is proven as follows:

[0164] Let the obtained target parameter server subset be S. a , and i∈S a , Assume the optimal set of parameter servers is U, and j∈U, In the objective (2) of the second objective function, there are constraints. and Where p j This represents the parameter ratio of parameter server j in the new optimization objective (2).

[0165] In the original optimization objective (1), there are constraints. and Where p i ′、p j ′ and T′ represent the parameter proportions of parameter server i, parameter server j, and the completion time of the transmission sub-round, respectively, in the optimization objective (1). For parameter server i, and The third and fourth constraints, which characterize the bandwidth requirement, have one that is equal. This is because if both constraints were inequalities, then p... i ′ can continue to increase while satisfying the constraints. Because the sum of the parameter ratios of all parameter servers is 1, this means that the ratios of the other parameters p s T′ can be reduced, causing constraints (1c) and (1d) to become relaxed. At this point, T′ can continue to decrease, which contradicts the fact that T′ is the optimal solution. Therefore, one of the constraints (1c) and (1d) corresponding to a parameter server is an equality. The constraints of parameter servers i and j are discussed separately below.

[0166] First scenario: By p i ′≥p j With ′ and two equality constraints, we get Therefore, in the new optimization constraint (2c), replacing j with i, we can see that... Constraints are satisfied. (By...) and achievable because Learned This holds true. In summary, under these circumstances, replacing j with i does not change the constraint of the new optimization objective (2).

[0167] The second scenario: By p i ′≥p j With ′ and two equality constraints, we get Therefore, in the new optimization constraint (2d), replacing j with i, we can see that... Constraints are satisfied. (By...) and achievable because Learned This holds true. In summary, under these circumstances, replacing j with i does not change the constraint of the new optimization objective (2).

[0168] Regarding the situation Similarly, it can be deduced that if j is replaced with i, the constraint of the new optimization objective (2) still holds.

[0169] In summary, in the new optimization objective (2), replacing server j in the selected parameter server set U with server i still holds the constraint. Therefore, the minimum transmission duration T of the transmission sub-round obtained after the replacement is still valid. i Less than or equal to the minimum transmission duration T of the transmission sub-rounds obtained before the replacement j Therefore, in the optimization objective (1), the initial parameter server is replaced with the target parameter server, and the initial parameter server corresponding to the top n in the first parameter ratio is selected to obtain n target parameter servers. Then, gradient aggregation and parameter update are performed through the physical servers where they are placed to obtain the optimal processing scheme, and the proof ends.

[0170] According to the embodiments of this disclosure, under the premise of determining the target parameter server, that is, the parameter server with more bandwidth resources holds more parameters, the target parameter server is optimized and selected. Then, the second parameter ratio of the target parameter server and the minimum transmission time of the transmission sub-round are solved by the second objective optimization function. The communication time of the training task is further minimized by using the parameter ratio held by the target parameter server and the minimum transmission time of the transmission sub-round.

[0171] According to embodiments of this disclosure, for each training task, a first bandwidth value, a second bandwidth value, and a remaining completion time are generated based on the second parameter ratio of the target parameter server and the minimum transmission duration, including:

[0172] For each training task, the amount of data in the transmission sub-round is generated according to the proportion of the second parameter of the target parameter server;

[0173] A first bandwidth value and a second bandwidth value are generated based on the amount of data received in the transmission sub-round and the minimum transmission duration.

[0174] The remaining completion time is generated based on the minimum transmission duration of the transmission sub-round.

[0175] According to embodiments of this disclosure, the amount of data the target parameter server generates in a transmission sub-round can be obtained by multiplying the amount of model parameter data by the number of worker nodes by the proportion of the second parameter. For example, the number of parameter servers s in a transmission sub-round is ump. s .

[0176] In one embodiment, the first bandwidth value d1 required by each target parameter server is given by the formula (11):

[0177]

[0178] Among them, T min The minimum transmission duration of the transmission sub-rounds obtained in the second objective optimization function is represented.

[0179] The formula for the second bandwidth value d2 required by each working node is shown in (12):

[0180]

[0181] Among them, T min The minimum transmission duration of the transmission sub-rounds obtained in the second multi-objective optimization function is represented.

[0182] Figure 3 A flowchart illustrating a method for generating remaining completion time according to an embodiment of the present disclosure is shown schematically.

[0183] like Figure 3 As shown, the remaining completion time is generated based on the minimum transmission duration of the transmission sub-round. The method 300 includes operations S310 to S350.

[0184] In operation S310, for each training task, the number of remaining training rounds is obtained based on the target number of training rounds.

[0185] When operating the S320, for each training task, the remaining completion time of the current round is obtained based on the amount of data remaining in the current round.

[0186] In operation S330, for each training task, the remaining completion time is generated based on the number of remaining training rounds, the remaining completion time of the current round, and the minimum transmission time of the transmission sub-rounds.

[0187] According to embodiments of this disclosure, for each training task, the number of target training rounds is subtracted from the number of already trained rounds to obtain the number of remaining training rounds.

[0188] According to embodiments of this disclosure, for each training task, the remaining data volume in the current round is divided by the transmission rate to obtain the remaining completion time of the current round, wherein the remaining data volume and transmission rate in the current round can be directly obtained from the system.

[0189] In one embodiment, the remaining completion time T for each training task l The formula is shown in (13):

[0190]

[0191] Where E represents the number of remaining training rounds, Q represents the amount of data remaining in the current round, v represents the transmission rate, and T represents the transmission rate. min The minimum transmission duration of the transmission sub-rounds obtained in the second objective optimization function is represented.

[0192] According to embodiments of this disclosure, the remaining completion time of each training task is obtained based on the minimum transmission duration of the transmission sub-round, and then the execution order of each training task and the allocated resources are scheduled to reduce the communication time of the training tasks and minimize the average task completion time of the machine learning cluster.

[0193] According to embodiments of this disclosure, for each training task, the remaining number of training rounds is obtained based on the target number of training rounds, including:

[0194] When the target number of training rounds is unknown, the loss function is used to process the number of historical training rounds to generate a loss function curve.

[0195] If the loss function curve converges, the target number of training rounds is determined based on the loss function curve; and

[0196] The number of remaining training rounds is obtained based on the target number of training rounds.

[0197] According to embodiments of this disclosure, when the target number of training rounds is unknown, the actual value of the number of historical training rounds is processed using a loss function to obtain historical training data, and then the historical data is used to generate a loss function curve.

[0198] In one embodiment, the loss function l(k) is as shown in (14):

[0199]

[0200] Where β0, β1, and β2 represent the first, second, and third non-negative parameters, respectively, and k represents the number of training rounds.

[0201] According to embodiments of this disclosure, the convergence rate of the loss function model is O(1 / k).

[0202] According to an embodiment of this disclosure, the target number of training rounds is determined when the loss function curve converges after M consecutive training rounds.

[0203] In one embodiment, the predicted value of the target number of training rounds K can be the smallest integer that satisfies the convergence condition, and the convergence condition of the loss function curve is shown in (15):

[0204] l(KM)-l(K)<ε (15)

[0205] Where M represents the number of consecutive training rounds recently, and ε represents the preset convergence threshold, for example, ε is 0.3.

[0206] According to an embodiment of this disclosure, the number of training rounds K is subtracted from the number of training rounds k to obtain the number of remaining training rounds E for each training task.

[0207] According to an embodiment of this disclosure, when the target number of training rounds is known, the number of training rounds already completed is directly subtracted to obtain the remaining number of training rounds.

[0208] According to embodiments of this disclosure, the remaining completion time of each training task is processed based on the principle of prioritizing the minimum remaining completion time, resulting in a priority task bandwidth allocation scheme, including:

[0209] Based on the principle of prioritizing the minimum remaining completion time, the first training task is determined from multiple training tasks according to their respective remaining completion times.

[0210] A bandwidth allocation scheme is obtained based on the first bandwidth value and the second bandwidth value corresponding to the first training task.

[0211] According to embodiments of this disclosure, the minimum remaining completion time priority principle indicates that, for multiple training tasks, the task with the minimum remaining completion time is preferred.

[0212] According to embodiments of this disclosure, a first training task is determined based on the principle of minimum remaining completion time priority, and bandwidth resources are allocated for training according to the first bandwidth value and the second bandwidth value corresponding to the first training task.

[0213] According to embodiments of this disclosure, in response to a data processing request, a training task with a waiting time greater than or equal to a first preset threshold is obtained from the application platform. The above steps are repeated to find the next task with the minimum remaining completion time until there are no training tasks greater than or equal to the first preset threshold in the entire machine learning cluster.

[0214] According to embodiments of this disclosure, when there are no training tasks in the entire machine learning cluster that are greater than or equal to a first preset threshold, all training tasks in the entire machine learning cluster that are less than the first preset threshold are directly scheduled for training.

[0215] According to embodiments of this disclosure, the remaining completion time of each training task is processed based on the principle of prioritizing the minimum remaining completion time, resulting in a priority task bandwidth allocation scheme. That is, while performing distributed cluster resource scheduling, communication optimization is considered, thereby minimizing the average task completion time of the entire cluster.

[0216] Figure 4 A schematic block diagram of a communication-optimized distributed training device according to an embodiment of the present disclosure is shown.

[0217] Based on the aforementioned communication-optimized distributed training method, this disclosure also provides a communication-optimized distributed training apparatus. The following will combine... Figure 4 The device is described in detail.

[0218] like Figure 4 As shown, the distributed training device 400 of this embodiment includes an acquisition module 410, a first processing module 420, a first determination module 430, a second processing module 440, a generation module 450, a second determination module 460, and a training module 470.

[0219] The acquisition module 410 is used to acquire training tasks with a waiting time greater than or equal to a first preset threshold in response to a data processing request. In one embodiment, the acquisition module 410 can be used to perform the operation S210 described above, which will not be repeated here.

[0220] The first processing module 420 is configured to, for each training task, process the first data transmission and reception volume information between the initial parameter server and the worker nodes based on a first objective optimization function to obtain a first parameter ratio of the initial parameter server. The initial parameter server and the worker nodes are configured to execute at least one training round for each training task, and the first parameter ratio characterizes the ratio between the parameters held by the initial parameter server and the total parameters of the trained model. In one embodiment, the first processing module 420 may be used to execute the operation S220 described above, which will not be repeated here.

[0221] The first determining module 430 is configured to determine at least one target parameter server from among the multiple initial parameter servers based on the first parameter ratio of each of the multiple initial parameter servers. In one embodiment, the first determining module 430 may be used to perform the operation S230 described above, which will not be repeated here.

[0222] The second processing module 440 is used to process the second data transmission and reception volume information between the target parameter server and the worker nodes based on the second objective optimization function for each training task, to obtain the second parameter ratio of the target parameter server and the minimum transmission duration of the transmission sub-round, wherein the transmission sub-round represents the data transmission process in the training round. In one embodiment, the second processing module 440 can be used to execute the operation S240 described above, which will not be repeated here.

[0223] The generation module 450 is used to generate a first bandwidth value, a second bandwidth value, and a remaining completion time for each training task, based on the second parameter ratio of the target parameter server and the minimum transmission time. The first bandwidth value represents the bandwidth resources corresponding to the target parameter server, and the second bandwidth value represents the bandwidth resources corresponding to the worker node. In one embodiment, the generation module 450 can be used to perform the operation S250 described above, which will not be repeated here.

[0224] The second determining module 460 is used to process the remaining completion time of each training task based on the principle of prioritizing the minimum remaining completion time, and obtain a bandwidth allocation scheme. In one embodiment, the second determining module 460 can be used to perform the operation S260 described above, which will not be repeated here.

[0225] The training module 470 is used to allocate bandwidth resources to the target parameter server and worker nodes according to the bandwidth allocation scheme, so as to execute the training task and obtain the training results. In one embodiment, the training module 470 can be used to perform the operation S270 described above, which will not be repeated here.

[0226] According to embodiments of this disclosure, by acquiring training tasks in the entire machine learning cluster that are greater than or equal to a first preset threshold, a first objective optimization function is used to solve for the first parameter ratio held by the initial parameter server, thereby calculating the target parameter server subset required for each training task. Then, a second objective optimization function is combined to solve for the second parameter ratio held by the target parameter server and the minimum transmission duration of the transmission sub-round, and the bandwidth resources allocated to it are calculated. Simultaneously, the remaining completion time of each training task is obtained based on the minimum remaining completion time priority principle, thereby determining a priority task bandwidth allocation scheme. Because the training tasks in the entire machine learning cluster are scheduled, the overall execution order of each training task is optimized, and resources are allocated according to this order. Simultaneously, the scheduling method of the objective optimization function dynamically adjusts the parameter ratio held by each parameter server, allowing parameter servers with more bandwidth resources to hold more parameters, reducing communication time, and solving the problem of minimizing the average task completion time in the machine learning cluster.

[0227] According to embodiments of this disclosure, the first determining module 430 includes a first determining submodule, a second determining submodule, and a third determining submodule.

[0228] The first determining submodule is used to sort multiple first parameter ratios from largest to smallest according to the values ​​represented by the first parameter ratios, to obtain a first parameter ratio sequence.

[0229] The second determining submodule is used to determine the first parameter ratio of the first n items in the first parameter ratio sequence as the target first parameter ratio, where n≥1.

[0230] The third determination submodule is used to determine the target parameter server from multiple initial parameter servers based on the target first parameter ratio.

[0231] According to embodiments of this disclosure, the generation module 450 includes a first generation submodule, a second generation submodule, and a third generation submodule.

[0232] The first generation submodule is used to generate the amount of data in the transmission sub-round for each training task, based on the proportion of the second parameter of the target parameter server.

[0233] The second generation submodule is used to generate a first bandwidth value and a second bandwidth value based on the amount of data received in the transmission sub-round and the minimum completion time.

[0234] The third generation submodule generates the remaining completion time based on the minimum transmission duration of the transmission sub-round.

[0235] According to embodiments of this disclosure, the third generation submodule includes a first generation unit, a second generation unit, and a third generation unit.

[0236] The first generation unit is used to obtain the number of remaining training rounds for each training task, based on the target number of training rounds.

[0237] The second generation unit is used to obtain the remaining completion time of the current round for each training task based on the amount of data remaining in the current round.

[0238] The third generation unit is used to generate the remaining completion time for each training task based on the number of remaining training rounds, the remaining completion time of the current round, and the minimum transmission time of the transmission sub-rounds.

[0239] According to embodiments of this disclosure, the first generation unit includes a loss function curve generation subunit, a target training round number determination subunit, and a remaining training round number generation subunit.

[0240] The loss function curve generation sub-unit is used to generate a loss function curve when the target number of training rounds is unknown, by processing the number of historical training rounds using the loss function.

[0241] The target number of training rounds determination subunit is used to determine the target number of training rounds based on the loss function curve, provided that the loss function curve converges.

[0242] The remaining training rounds number generation sub-unit is used to obtain the remaining training rounds number based on the target training rounds number.

[0243] According to embodiments of this disclosure, the second determining module 460 includes a first training task determining unit and a bandwidth allocation scheme determining unit.

[0244] The first training task determination unit is used to determine the first training task from multiple training tasks based on the principle of minimum remaining completion time priority and according to the remaining completion time of each of the multiple training tasks.

[0245] The bandwidth allocation scheme determination unit is used to obtain a bandwidth allocation scheme based on the first bandwidth value and the second bandwidth value corresponding to the first training task.

[0246] According to embodiments of this disclosure, any multiple modules among the acquisition module 410, the first processing module 420, the first determining module 430, the second processing module 440, the generation module 450, the second determining module 460, and the training module 470 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the acquisition module 410, the first processing module 420, the first determining module 430, the second processing module 440, the generation module 450, the second determining module 460, and the training module 470 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in any one of the three methods of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, at least one of the acquisition module 410, the first processing module 420, the first determining module 430, the second processing module 440, the generation module 450, the second determining module 460, and the training module 470 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0247] Figure 5 The diagram illustrates an electronic device suitable for implementing a communication-optimized distributed training method according to embodiments of the present disclosure.

[0248] like Figure 5 As shown, an electronic device 500 according to an embodiment of the present disclosure includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage portion 508 into a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0249] RAM 503 stores various programs and data required for the operation of electronic device 500. Processor 501, ROM 502, and RAM 503 are interconnected via bus 504. Processor 501 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 502 and / or RAM 503. It should be noted that programs may also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0250] According to embodiments of this disclosure, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to a bus 504. The electronic device 500 may also include one or more of the following components connected to the I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0251] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0252] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 502 and / or RAM 503 and / or one or more memories other than ROM 502 and RAM 503 described above.

[0253] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the distributed training method provided in the embodiments of this disclosure.

[0254] When the computer program is executed by the processor 501, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0255] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0256] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0257] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0258] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0259] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0260] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A distributed training method based on communication optimization, comprising: In response to a data processing request, acquire training tasks with a waiting time greater than or equal to a first preset threshold; For each training task, the first data transmission and reception volume information between the initial parameter server and the worker node is processed based on the first objective optimization function to obtain the first parameter ratio of the initial parameter server, wherein the initial parameter server and the worker node are configured to perform at least one training round for each training task, and the first parameter ratio represents the ratio between the parameters held by the initial parameter server and the total parameters of the trained model. Based on the first parameter ratio of each of the plurality of initial parameter servers, at least one target parameter server is determined from the plurality of initial parameter servers; For each training task, the second data transmission and reception information between the target parameter server and the worker node is processed based on the second objective optimization function to obtain the second parameter ratio of the target parameter server and the minimum transmission duration of the transmission sub-round, wherein the transmission sub-round characterizes the data transmission process in the training round; For each training task, a first bandwidth value, a second bandwidth value, and a remaining completion time are generated based on the second parameter ratio of the target parameter server and the minimum transmission time. The first bandwidth value represents the bandwidth resources corresponding to the target parameter server, and the second bandwidth value represents the bandwidth resources corresponding to the worker node. The remaining completion time of each training task is processed based on the principle of prioritizing the minimum remaining completion time, resulting in a bandwidth allocation scheme; and According to the bandwidth allocation scheme, bandwidth resources are allocated to the target parameter server and the worker node respectively, so as to execute the training task and obtain the training results.

2. The method according to claim 1, wherein, For each training task, the first data transmission and reception volume information between the initial parameter server and the worker nodes is processed based on the first objective optimization function to obtain the first parameter ratio of the initial parameter server, including: For each training task, the data transmission and reception information of multiple initial parameter servers and multiple worker nodes are processed based on the first constraint condition corresponding to the first objective optimization function to obtain the first parameter ratio, wherein the first constraint condition includes at least one of the following: The first constraint condition includes a first constraint sub-condition, which indicates that the amount of data sent by each working node is less than or equal to the rated amount of data sent by each working node. The rated amount of data sent by the working node is obtained based on the product between the node's sending bandwidth and the intermediate transmission time. The optimization objective of the first objective optimization function is to minimize the intermediate transmission time. The first constraint condition includes a second constraint sub-condition, which indicates that the amount of data received by each working node is less than or equal to the rated amount of data received by each working node, and the rated amount of data received by the working node is obtained based on the product between the node receiving bandwidth and the intermediate transmission duration. The first constraint condition includes a third constraint sub-condition, which indicates that the amount of data received by each of the initial parameter servers is less than or equal to the rated amount of data received by each of the initial parameter servers. The rated amount of data received by the initial parameter server is obtained based on the product between the receiving bandwidth of the initial parameter server and the intermediate transmission duration. The first constraint condition includes a fourth constraint sub-condition, which indicates that the amount of data sent by each of the initial parameter servers is less than or equal to the rated amount of data sent by each of the initial parameter servers. The rated amount of data sent by the initial parameter server is obtained based on the product between the initial parameter server's sending bandwidth and the intermediate transmission duration.

3. The method according to claim 1, wherein, Based on the first parameter ratio of each of the multiple initial parameter servers, at least one target parameter server is determined from the multiple initial parameter servers, including: The multiple first parameter ratios are sorted from largest to smallest according to the values ​​represented by the first parameter ratios to obtain the first parameter ratio sequence; The first parameter ratio of the top n items in the first parameter ratio sequence is determined as the target first parameter ratio, where n ≥ 1; and The target parameter server is determined from a plurality of initial parameter servers based on the target first parameter ratio.

4. The method according to claim 1, wherein, For each training task, the second data transmission and reception information between the target parameter server and the worker node is processed based on the second objective optimization function to obtain the second parameter ratio of the target parameter server and the minimum transmission duration of the transmission sub-round, including: For each training task, the data transmission and reception information of multiple target parameter servers and multiple worker nodes are processed based on the second constraint condition corresponding to the second objective optimization function to obtain the second parameter ratio, wherein the second constraint condition includes at least one of the following: The second constraint condition includes a fifth constraint sub-condition, which indicates that the amount of data sent by each working node is less than or equal to the rated amount of data sent by each working node. The rated amount of data sent by each working node is obtained based on the product of the node's transmission bandwidth and the transmission duration. The optimization objective of the second objective optimization function is to minimize the transmission duration. The second constraint includes a sixth constraint sub-condition, which indicates that the amount of data received by each working node is less than or equal to the rated amount of data received by each working node, and the rated amount of data received by the working node is obtained based on the product of the node receiving bandwidth and the transmission duration. The second constraint condition includes a seventh constraint sub-condition, which indicates that the amount of data received by each of the target parameter servers is less than or equal to the rated amount of data received by each of the target parameter servers. The rated amount of data received by the target parameter server is obtained based on the product between the target parameter server's receiving bandwidth and the transmission duration. The second constraint condition includes an eighth constraint sub-condition, which indicates that the amount of data sent by each of the target parameter servers is less than or equal to the rated amount of data sent by each of the target parameter servers. The rated amount of data sent by the target parameter server is obtained based on the product between the target parameter server's transmission bandwidth and transmission duration.

5. The method according to claim 1, wherein, For each training task, based on the second parameter ratio of the target parameter server and the minimum transmission duration, a first bandwidth value, a second bandwidth value, and a remaining completion time are generated, including: For each training task, the amount of data in the transmission sub-round is generated according to the second parameter ratio of the target parameter server; A first bandwidth value and a second bandwidth value are generated based on the amount of data received in the transmission sub-round and the minimum transmission duration. The remaining completion time is generated based on the minimum transmission duration of the transmission sub-round.

6. The method according to claim 5, wherein, The remaining completion time is generated based on the minimum transmission duration of the aforementioned transmission sub-round, including: For each training task, the number of remaining training rounds is obtained based on the target number of training rounds. For each training task, the remaining completion time for the current round is obtained based on the amount of data remaining in the current round. For each training task, the remaining completion time is generated based on the number of remaining training rounds, the remaining completion time of the current round, and the minimum transmission time of the transmission sub-round.

7. The method according to claim 6, wherein for each training task, the number of remaining training rounds is obtained based on the target number of training rounds, comprising: When the target number of training rounds is unknown, the loss function is used to process the number of historical training rounds to generate a loss function curve. If the loss function curve converges, the target number of training rounds is determined based on the loss function curve; and The number of remaining training rounds is obtained based on the target number of training rounds.

8. The method according to claim 1, wherein, The remaining completion time of each training task is processed based on the principle of prioritizing the minimum remaining completion time to obtain a bandwidth scheme, including: Based on the principle of prioritizing the minimum remaining completion time, the first training task is determined from the multiple training tasks according to their respective remaining completion times. A bandwidth scheme is obtained based on the first bandwidth value and the second bandwidth value corresponding to the first training task.

9. A distributed training device based on communication optimization, comprising: The acquisition module is used to acquire training tasks with a waiting time greater than or equal to a first preset threshold in response to data processing requests. The first processing module is used to process the first data transmission and reception volume information between the initial parameter server and the worker node based on the first objective optimization function for each training task, and obtain the first parameter ratio of the initial parameter server, wherein the initial parameter server and the worker node are configured to perform at least one training round for each training task, and the first parameter ratio represents the ratio between the parameters held by the initial parameter server and the total parameters of the trained model. The first determining module is used to determine at least one target parameter server from the plurality of initial parameter servers according to the first parameter ratio of each of the plurality of initial parameter servers; The second processing module is used to process the second data transmission and reception volume information between the target parameter server and the worker node based on the second objective optimization function for each training task, and obtain the second parameter ratio of the target parameter server and the minimum transmission duration of the transmission sub-round, wherein the transmission sub-round represents the data transmission process in the training round; The generation module is used to generate a first bandwidth value, a second bandwidth value, and a remaining completion time for each training task, based on the second parameter ratio of the target parameter server and the minimum transmission time, wherein the first bandwidth value represents the bandwidth resources corresponding to the target parameter server, and the second bandwidth value represents the bandwidth resources corresponding to the worker node. The second determining module is used to process the remaining completion time of each training task based on the principle of prioritizing the minimum remaining completion time, and obtain a bandwidth allocation scheme. The training module is used to allocate bandwidth resources to the target parameter server and the worker node according to the bandwidth allocation scheme, so as to execute the training task and obtain training results.

10. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 8.

Images