A load balancing method and device for computing nodes

Abstract

The present application relates to the technical field of distributed training, in particular to a load balancing method and device for computing nodes.A load balancing method for computing nodes comprises: collecting data by a performance analysis tool to obtain node performance data; constructing a model based on an improved MMoE model to obtain a to-be-trained multi-objective joint prediction model; dynamically partitioning the node performance data to process the data to obtain training data; training the to-be-trained multi-objective joint prediction model using the training data to obtain a trained multi-objective joint prediction model; verifying the trained multi-objective joint prediction model by an elastic relaxation scheme, and obtaining a multi-objective joint prediction model when the verification result is passed; and inputting to-be-balanced node performance data to perform load balancing by the multi-objective joint prediction model.The present application is a load balancing method for computing nodes with high processing efficiency and strong robustness.

Description

Technical Field

[0001] This invention relates to the field of distributed training technology, and in particular to a load balancing method and apparatus for computing nodes. Background Technology

[0002] With the development of deep learning and big data technologies, distributed training has become an important way to improve computational efficiency. However, in a distributed training environment, performance differences between heterogeneous computing nodes and interference from irrelevant tasks may lead to uneven system load, causing a "lag phenomenon," which manifests as some nodes processing slower, thereby reducing the overall training efficiency of the system.

[0003] To address the demands of large-scale computing, existing methods for accelerating training primarily fall into two categories: adding more resources to a single machine and adding more nodes to the system. However, centralized training on a single node presents numerous challenges, particularly as the computing power of a single node is insufficient to meet the requirements of training time and accuracy. Therefore, to shorten training time and improve model performance, algorithm designers utilize the method of adding nodes to the system to construct a distributed system, thereby enhancing training efficiency.

[0004] In recent years, researchers have mostly focused on building distributed environments to improve resource utilization through parallelization of node computation and communication. The load imbalance problem among computing nodes in distributed training has become a significant factor limiting the efficiency and performance improvement of model training.

[0005] In the existing technology, there is a lack of a load balancing method that is efficient and robust for computing nodes. Summary of the Invention

[0006] This invention provides a load balancing method and apparatus for computing nodes. The technical solution is as follows:

[0007] On the one hand, a load balancing method for computing nodes is provided, which is implemented by electronic devices and includes:

[0008] Data is collected using performance analysis tools to obtain node performance data;

[0009] Model construction is carried out based on the improved MMoE model to obtain a multi-objective joint prediction model to be trained;

[0010] The node performance data is dynamically partitioned to obtain dynamically adjusted data;

[0011] The dynamically adjusted data is processed to obtain training data;

[0012] Using the training data, the multi-objective joint prediction model to be trained is trained to obtain the trained multi-objective joint prediction model.

[0013] The trained multi-objective joint prediction model was validated using an elastic relaxation scheme. When the validation result passed, the multi-objective joint prediction model was obtained.

[0014] Input the performance data of the nodes to be balanced, and perform load balancing through the multi-objective joint prediction model.

[0015] The node performance data includes the size of the subset of data in the current training round of the node, the total training time of the current training round, the training time of the node's iteration rounds, and the number of iteration rounds of the node under the current synchronization window.

[0016] The multi-objective joint prediction model to be trained includes an input layer, a feature extraction layer, and a private task top layer.

[0017] The input layer is used to divide the input data of the multi-objective joint prediction model to be trained into time-series data and cross-sectional data.

[0018] The time series data is the training behavior information of nodes in the input data;

[0019] The cross-sectional data is the training information of the nodes in the input data during the current iteration cycle.

[0020] The feature extraction layer includes a unique bottom-level module and a common bottom-level module.

[0021] The unique underlying module includes an LSTM feature extraction network and an MLP feature extraction network; the input data of the LSTM feature extraction network is time-series data; the input data of the MLP feature extraction network is cross-sectional data.

[0022] The shared underlying module is used to extract common features related to subtasks.

[0023] The private task top layer is used to convert the output of the MMoE layer in the multi-objective joint prediction model to be trained into the prediction result of a specific sub-task.

[0024] The top layer of the private task consists of a dedicated network of subtasks, and the correspondence between the subtasks and the dedicated network is unique.

[0025] The dedicated network is used to fit the prediction results corresponding to the sub-tasks.

[0026] Optionally, the step of dynamically partitioning the node performance data to obtain dynamically adjusted data includes:

[0027] The node performance data is dynamically adjusted in terms of sample size to obtain sample size adjustment data.

[0028] The sample size adjustment data is partitioned into datasets to obtain dynamically adjusted data.

[0029] Optionally, after validating the trained multi-objective joint prediction model using an elastic relaxation scheme, the method further includes:

[0030] If the verification result is that the verification fails, continue to collect node performance data and iteratively train the multi-objective joint prediction model to be trained.

[0031] The elastic relaxation scheme is used to verify the load balancing performance of the multi-objective joint prediction model.

[0032] The elastic relaxation scheme controls the communication pace of the load nodes by adjusting the synchronization threshold.

[0033] On the other hand, a compute node-oriented load balancing device is provided, which is applied to a compute node-oriented load balancing method. The device includes:

[0034] The data acquisition module is used to collect data through performance analysis tools to obtain node performance data.

[0035] The model building module is used to build a model based on the improved MMoE model to obtain a multi-objective joint prediction model to be trained.

[0036] The data dynamic adjustment module is used to dynamically partition the node performance data to obtain dynamic adjustment data.

[0037] The data processing module is used to process the dynamically adjusted data to obtain training data;

[0038] The model training module is used to train the multi-objective joint prediction model to be trained using the training data, so as to obtain the trained multi-objective joint prediction model.

[0039] The model validation module is used to validate the trained multi-objective joint prediction model through an elastic relaxation scheme. When the validation result is passed, the multi-objective joint prediction model is obtained.

[0040] The load balancing module is used to input the performance data of the nodes to be balanced and perform load balancing through the multi-objective joint prediction model.

[0041] The node performance data includes the size of the subset of data in the current training round of the node, the total training time of the current training round, the training time of the node's iteration rounds, and the number of iteration rounds of the node under the current synchronization window.

[0042] The multi-objective joint prediction model to be trained includes an input layer, a feature extraction layer, and a private task top layer.

[0043] The input layer is used to divide the input data of the multi-objective joint prediction model to be trained into time-series data and cross-sectional data.

[0044] The time series data is the training behavior information of nodes in the input data;

[0045] The cross-sectional data is the training information of the nodes in the input data during the current iteration cycle.

[0046] The feature extraction layer includes a unique bottom-level module and a common bottom-level module.

[0047] The unique underlying module includes an LSTM feature extraction network and an MLP feature extraction network; the input data of the LSTM feature extraction network is time-series data; the input data of the MLP feature extraction network is cross-sectional data.

[0048] The shared underlying module is used to extract common features related to subtasks.

[0049] The private task top layer is used to convert the output of the MMoE layer in the multi-objective joint prediction model to be trained into the prediction result of a specific sub-task.

[0050] The top layer of the private task consists of a dedicated network of subtasks, and the correspondence between the subtasks and the dedicated network is unique.

[0051] The dedicated network is used to fit the prediction results corresponding to the sub-tasks.

[0052] Optionally, the data dynamic adjustment module is further used for:

[0053] The node performance data is dynamically adjusted in terms of sample size to obtain sample size adjustment data.

[0054] The sample size adjustment data is partitioned into datasets to obtain dynamically adjusted data.

[0055] Optionally, the model verification module is further configured to:

[0056] If the verification result is that the verification fails, continue to collect node performance data and iteratively train the multi-objective joint prediction model to be trained.

[0057] The elastic relaxation scheme is used to verify the load balancing performance of the multi-objective joint prediction model.

[0058] The elastic relaxation scheme controls the communication pace of the load nodes by adjusting the synchronization threshold.

[0059] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the aforementioned load balancing method for computing nodes.

[0060] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the above-described load balancing method for computing nodes.

[0061] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0062] This invention proposes a load balancing method for computing nodes, optimizing distributed training from two perspectives: aggregation scheme and node computing load. Through a proposed multi-objective joint prediction model, a joint prediction strategy is designed to balance node load based on synchronization thresholds and data volume. The elastic relaxation synchronization scheme effectively addresses the lag problem caused by cluster environments with significant differences in node performance, and this strategy exhibits good robustness and can optimize the training efficiency of distributed training. This invention provides a highly efficient and robust load balancing method for computing nodes. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a flowchart of a load balancing method for computing nodes provided in an embodiment of the present invention;

[0065] Figure 2 This is a block diagram of a load balancing device for computing nodes provided in an embodiment of the present invention;

[0066] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0067] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0068] This invention provides a load balancing method for computing nodes, which can be implemented by an electronic device, such as a terminal or a server. Figure 1 The flowchart shown represents a load balancing method for compute nodes. The processing flow of this method may include the following steps:

[0069] S1. Collect data using performance analysis tools to obtain node performance data.

[0070] The node performance data includes the size of the subset of data for the current training round of the node, the total training time for the current training round, the training time for the node's iteration rounds, and the number of iteration rounds the node performs within the current synchronization window.

[0071] In one feasible implementation, the node performance in this invention is based on a defined node performance factor P. This definition is based on the assumption that node performance is relatively stable in the short term. Under this assumption, the performance of node i in the j-th training cycle can be calculated using the size of the subset of data available in that training cycle. and the duration of this round of training To estimate, where ||·|| represents the magnitude of the value, as shown in the following formula (1):

[0072]

[0073] To conserve computational resources, the performance factor P is recalculated only after each training cycle. To maintain load balancing in distributed training, the amount of data held by each node in the subset should be dynamically adjusted based on P.

[0074] S2. Based on the improved MMoE model, a model is constructed to obtain a multi-objective joint prediction model to be trained.

[0075] The multi-objective joint prediction model to be trained includes an input layer, a feature extraction layer, and a private task top layer.

[0076] In one feasible implementation, the technical problem to be solved by the present invention is to provide a load balancing method for computing nodes that can effectively optimize distributed training. Based on this, the present invention addresses the load balancing problem during computing node training by introducing a joint prediction strategy based on synchronization thresholds and data volume, and a multi-objective joint prediction model, thus designing a load balancing method for computing nodes.

[0077] The input layer is used to divide the input data of the multi-objective joint prediction model to be trained into time series data and cross-sectional data.

[0078] Time series data is the training behavior information of nodes in the input data;

[0079] Cross-sectional data is the training information of nodes in the input data during the current iteration cycle.

[0080] In one feasible implementation, the collected node performance data is used as the model input in the input layer, and the preprocessed data is passed to the shared feature extraction layer. The input data is divided into temporal data and cross-sectional data. Temporal data contains node behavior information from the historical training process, while cross-sectional data includes node training information from the current iteration.

[0081] The feature extraction layer includes a unique bottom-level module and a common bottom-level module.

[0082] The unique underlying modules include an LSTM feature extraction network and an MLP feature extraction network; the input data for the LSTM feature extraction network is time-series data; the input data for the MLP feature extraction network is cross-sectional data.

[0083] There are underlying modules for extracting common features related to subtasks.

[0084] In one feasible implementation, the task's feature information is extracted at the feature extraction layer. However, negative transfer can easily occur when task relevance is weak, causing a significant task-sample dependency problem. To mitigate the impact of negative transfer, we consider explicitly separating the unique underlying module and the common underlying module, and training the two modules in parallel.

[0085] The unique underlying module contains two task-specific feature extraction networks. The inputs of the feature extraction networks take into account the differences in the tasks, using temporal data as the input to the Long Short-Term Memory Network (LSTM), while the input to the Multi-Layer Perceptron (MLP) is cross-sectional data.

[0086] The data partitioning subtask uses two LSTM networks with 16 and 8 neurons respectively. To capture the temporal relationships in the time-series data, ReLU activation functions are used between layers. Furthermore, DropOut layers with a random deactivation rate of 0.3 are introduced between layers to force some neurons to be randomly deactivated during training.

[0087] The delayed synchronization subtask uses a dual-hidden-layer MLP network with an input dimension of 2 and an output dimension of 1. Since the prediction of the synchronization threshold is a regression problem, the MES loss function is used to measure the difference between the predicted and actual values.

[0088] The common underlying modules focus on the correlation between tasks, extract common features related to all tasks from the input data, and use a multi-gate Mixture of Experts (MMoE) model to perform multi-task learning on sub-tasks.

[0089] The MMoE layer, as a core component of the fusion prediction architecture, consists of a set of expert networks and gating networks. Its core task is to learn a general low-level representation for the sub-tasks. This invention employs cross-validation for hyperparameter setting and uses four 32-neuron expert networks to learn the correlation between the input general features and the two prediction sub-tasks. The output is then passed to the task-specific low-level layer according to the weight allocation of the gating network.

[0090] Among them, the private task top layer is used to convert the output of the MMoE layer in the multi-objective joint prediction model to be trained into the prediction result of a specific sub-task;

[0091] The top layer of a private task consists of a dedicated network of subtasks, and the correspondence between subtasks and dedicated networks is unique.

[0092] A dedicated network is used to fit the prediction results corresponding to the subtask.

[0093] In one feasible implementation, the private task top layer is located after the MMoE layer and is used to convert the output of the MMoE layer into the prediction result of a specific subtask.

[0094] The top layer of the private task consists of dedicated networks for each subtask, with each task having its own dedicated network. These dedicated networks are designed according to the specific needs of each subtask and are used to fit the predictions corresponding to that subtask. Their inputs consist of two parts: subtask-specific vector mappings extracted by the unique bottom-level modules, and shared bottom-level representations extracted by the common bottom-level modules.

[0095] The dedicated networks consist of fully connected layers and a sigmoid activation function, which reduce the dimensionality of the feature vector formed by concatenating the general and specific vector mappings for the task, and finally obtain the prediction results.

[0096] S3. Dynamically partition the node performance data to obtain dynamically adjusted data.

[0097] Optionally, node performance data can be dynamically partitioned to obtain dynamically adjusted data, including:

[0098] The node performance data is dynamically adjusted in terms of sample size to obtain sample size adjustment data.

[0099] The sample size was adjusted by partitioning the dataset to obtain dynamically adjusted data.

[0100] In one feasible implementation, when the dataset is large, it is difficult to load the dataset into memory all at once for each training cycle. Therefore, the dataset D needs to be loaded in batches with a certain sample size B. Thus, when discussing the amount of data in the subset held by a node, we should start from the perspective of sample size. The entire dynamic adjustment process of the data volume can be divided into two stages: dynamic adjustment of sample size and dataset partitioning.

[0101] Formula (2) defines the number of samples held by node i in the (j+1)th training cycle. The proportion of sample size B.

[0102]

[0103] Furthermore, if the sample size is maintained in the next training cycle... If the training time for a node is T, then The mathematical expressions should be satisfied as follows: (3) and (4):

[0104]

[0105]

[0106] Substituting equation (4) into equation (3) and rearranging them, we can obtain the estimated training time T for the number of samples held, as shown in equation (5) below:

[0107]

[0108] Then the number of samples held by node i Percentage in the total sample It can be expressed as a mathematical expression as shown in equation (6):

[0109]

[0110] The dynamic sample size adjustment algorithm approximates the proportion of the dataset to the proportion of the sample size. Therefore, the dataset of nodes can be indirectly partitioned by dynamically adjusting the sample size. It should be represented as an integer.

[0111] However, the value calculated using the ratio might be a floating-point number. Therefore, it is necessary to minimize the introduction of errors. Convert to an integer. Then round down. The error should be minimized while keeping the total number of samples constant, and the mathematical relationships it satisfies are as follows: (7) and (8):

[0112]

[0113]

[0114] Use rounding for non-integer values. Process it. First, [do something]. Round down to get The processed sample size contains errors, and the mathematical expression for the errors is shown in equation (9) below:

[0115]

[0116] To reduce the impact of rounding on training, the error k is fed back into the algorithm, and the k processed sample values ​​are rounded up. However, to satisfy formula (8) and ensure that the rounding variance is minimized, the original sample values ​​should be rounded up before rounding up. Sort the sample values. The sorting operation focuses on the decimal parts of the sample values. Select the first k sorted items Round up.

[0117] Because at this time B′ j+1 The sum of these values ​​should be approximately equal to the original total sample size B. Therefore, the normalized result can be approximated as the adjusted proportion of the subset. The dataset can be based on B′. j+1 Divide the data. The return value consists of two lists L. j+1 and U j+1 The vector [L] j+1 U j+1 The formula for calculating the subset boundary of node i can be expressed as:

[0118]

[0119] S4. Process the dynamically adjusted data to obtain training data.

[0120] In one feasible implementation, the input to the optimization task model in this invention is the real-time performance data of the nodes. This data provides the basis for the adaptability of the optimization task. Specifically, the performance data that the optimization task focuses on is the iteration cycle duration (IterTime) of the computation node, which is the time interval between two consecutive push requests received by the parameter service node from the node.

[0121] Synchronization threshold th G Defined as the optimal value th dynamically selected from a fixed threshold range. G ∈[th G，L ,th G，U Each time a global synchronization threshold is selected, an iteration number difference E = th will occur between it and the lower threshold. G -th G，LTherefore, the model's output can be defined as the optimal iteration tolerance E in the current iteration. The delayed synchronization subtask determines the iteration tolerance when the user-specified lower limit of the lag threshold th is exceeded. G，L At that time, how many more iterations should a node in the current synchronization window continue to run after its iteration ends?

[0122] The iteration time intervals of each computing node are randomly simulated in different training scenarios, and corresponding TS values ​​are generated based on these iteration time intervals. These TS values ​​are then input into the synchronization controller to calculate the corresponding E as the target output. This yields a dataset containing the iteration period IterTime and iteration tolerance E of the leading node.

[0123] To generate the dataset for training the prediction model, the ensemble runs a data partitioning algorithm to obtain the actual performance of each computing node in each training iteration. The size of the subset of data for each node during each training iteration is then collected. Training duration and sample size

[0124] S5. Using the training data, train the multi-objective joint prediction model to obtain the trained multi-objective joint prediction model.

[0125] In one feasible implementation, a historical training dataset is constructed through feature engineering. Assuming there are N nodes, a 3×N dimensional vector sequence can be created in each training cycle. It consists of the DataSize, EpochTime, and BatchSize of N nodes in each training cycle. In the first training cycle, since the training data lacks historical data for dynamic partitioning, the first batch of samples in DataSize should consist of the evenly divided training data.

[0126] Since the dynamic partitioning task predicts the size of the next sub-dataset to be allocated, NextDataSize, based on information such as the data volume of the previous round's sub-dataset, NextDataSize can be obtained by advancing the DataSize by one training cycle in advance, as shown in the figure. Based on formula (11), the four-dimensional vector sequence NextDataSize corresponding to each training cycle can be obtained.

[0127] NextDataSet j，i =DataSet j+1，i (11)

[0128] The selected feature datasets are DataSize, EpochTime, and BatchSize, and the label dataset is [not specified]. Since the input to the LSTM model needs to be time-series data, a feature dataset with time series characteristics needs to be constructed. With T time steps of data, each time step contains three features: EpochTime, the historically assigned subset size DataSize, and the historically assigned sample size BatchSize. Therefore, the input layer size is (B, T, 3), where B represents the total sample size. Based on the obtained training data, the multi-objective joint prediction model is trained to obtain the trained multi-objective joint prediction model.

[0129] S6. Validate the trained multi-objective joint prediction model using an elastic relaxation scheme. If the validation result is successful, the multi-objective joint prediction model is obtained.

[0130] Optionally, after validating the trained multi-objective joint prediction model using an elastic relaxation scheme, the method further includes:

[0131] If the verification result is that the verification fails, continue to collect node performance data and iteratively train the multi-objective joint prediction model to be trained.

[0132] In one feasible implementation, after obtaining the multi-objective joint prediction model, an elastic relaxation scheme is used for verification. If the performance of load balancing cannot meet the expected requirements, node performance data is collected again for iterative training.

[0133] Among them, the elastic relaxation scheme is used to verify the load balancing performance of the multi-objective joint prediction model;

[0134] The elastic relaxation scheme controls the communication pace of load nodes by adjusting the synchronization threshold.

[0135] In one feasible implementation, the core idea of ​​the elastic relaxation synchronization scheme is to allow computing nodes to iterate to a certain extent ahead of lagging nodes. Simultaneously, this lead time is given adaptive characteristics, namely, by adaptively adjusting the amount of data held by each computing node and the synchronization conditions, the waiting time of leading nodes is minimized, thereby improving overall training efficiency.

[0136] To improve node resource utilization, the elastic relaxation synchronization scheme first performs a data partitioning subtask before the start of each training cycle to adaptively adjust the subset of data allocated to the computing nodes.

[0137] To address the lagging performance of the node cluster during parameter aggregation, the model executes a delayed synchronization subtask to adaptively adjust the synchronization threshold and control the communication pace of the computing nodes. This adjustment is based on predictions from the previous training round.

[0138] S7. Input the performance data of the nodes to be balanced, and perform load balancing through a multi-objective joint prediction model.

[0139] In one feasible implementation, the present invention can achieve information sharing among multiple optimization objectives. Simultaneously, the elastic relaxation scheme can provide predictive capabilities, effectively predicting the next round of training data based on historical training data, thus providing more information for optimizing the training process.

[0140] When sub-task networks corresponding to the optimization objectives are jointly incorporated into a multi-objective joint prediction model for multi-task learning, if the training data for a particular task is of poor quality or contains noise, the multi-objective joint prediction model can mitigate these adverse effects through information from other tasks, thereby improving the model's performance across all tasks. Furthermore, this multi-task learning can also help the model learn more shared knowledge, improving its adaptability to new data.

[0141] This invention proposes a load balancing method for computing nodes, optimizing distributed training from two perspectives: aggregation scheme and node computing load. Through a proposed multi-objective joint prediction model, a joint prediction strategy is designed to balance node load based on synchronization thresholds and data volume. The elastic relaxation synchronization scheme effectively addresses the lag problem caused by cluster environments with significant differences in node performance, and this strategy exhibits good robustness and can optimize the training efficiency of distributed training. This invention provides a highly efficient and robust load balancing method for computing nodes.

[0142] Figure 2 This is a block diagram of a load balancing device for computing nodes, according to an exemplary embodiment. (Refer to...) Figure 2 The device includes:

[0143] Data acquisition module 210 is used to collect data through performance analysis tools to obtain node performance data;

[0144] The model building module 220 is used to build a model based on the improved MMoE model to obtain a multi-objective joint prediction model to be trained.

[0145] The data dynamic adjustment module 230 is used to dynamically partition node performance data to obtain dynamic adjustment data.

[0146] Data processing module 240 is used to process dynamically adjusted data to obtain training data;

[0147] The model training module 250 is used to train the multi-objective joint prediction model to be trained using training data, and obtain the trained multi-objective joint prediction model.

[0148] The model validation module 260 is used to validate the trained multi-objective joint prediction model through an elastic relaxation scheme. When the validation result is passed, the multi-objective joint prediction model is obtained.

[0149] The load balancing module 270 is used to input the performance data of the nodes to be balanced and to perform load balancing through a multi-objective joint prediction model.

[0150] The node performance data includes the size of the subset of data for the current training round of the node, the total training time for the current training round, the training time for the node's iteration rounds, and the number of iteration rounds the node performs within the current synchronization window.

[0151] The multi-objective joint prediction model to be trained includes an input layer, a feature extraction layer, and a private task top layer.

[0152] The input layer is used to divide the input data of the multi-objective joint prediction model to be trained into time series data and cross-sectional data.

[0153] Time series data is the training behavior information of nodes in the input data;

[0154] Cross-sectional data is the training information of nodes in the input data during the current iteration cycle.

[0155] The feature extraction layer includes a unique bottom-level module and a common bottom-level module.

[0156] The unique underlying modules include an LSTM feature extraction network and an MLP feature extraction network; the input data for the LSTM feature extraction network is time-series data; the input data for the MLP feature extraction network is cross-sectional data.

[0157] There are underlying modules for extracting common features related to subtasks.

[0158] Among them, the private task top layer is used to convert the output of the MMoE layer in the multi-objective joint prediction model to be trained into the prediction result of a specific sub-task;

[0159] The top layer of a private task consists of a dedicated network of subtasks, and the correspondence between subtasks and dedicated networks is unique.

[0160] A dedicated network is used to fit the prediction results corresponding to the subtask.

[0161] Optionally, the data dynamic adjustment module 230 is further used for:

[0162] The node performance data is dynamically adjusted in terms of sample size to obtain sample size adjustment data.

[0163] The sample size was adjusted by partitioning the dataset to obtain dynamically adjusted data.

[0164] Optionally, the model validation module 260 is also used for:

[0165] If the verification result is that the verification fails, continue to collect node performance data and iteratively train the multi-objective joint prediction model to be trained.

[0166] Among them, the elastic relaxation scheme is used to verify the load balancing performance of the multi-objective joint prediction model;

[0167] The elastic relaxation scheme controls the communication pace of load nodes by adjusting the synchronization threshold.

[0168] This invention proposes a load balancing method for computing nodes, optimizing distributed training from two perspectives: aggregation scheme and node computing load. Through a proposed multi-objective joint prediction model, a joint prediction strategy is designed to balance node load based on synchronization thresholds and data volume. The elastic relaxation synchronization scheme effectively addresses the lag problem caused by cluster environments with significant differences in node performance, and this strategy exhibits good robustness and can optimize the training efficiency of distributed training. This invention provides a highly efficient and robust load balancing method for computing nodes.

[0169] Figure 3 This is a schematic diagram of the structure of an electronic device 300 provided in an embodiment of the present invention. The electronic device 300 may vary considerably due to different configurations or performance. It may include one or more central processing units (CPUs) 301 and one or more memories 302. The memory 302 stores at least one instruction, which is loaded and executed by the processor 301 to implement the steps of the above-described load balancing method for computing nodes.

[0170] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to perform the aforementioned load balancing method for computing nodes. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0171] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0172] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A load balancing method for computing nodes, characterized in that, The method includes: Data is collected using performance analysis tools to obtain node performance data; Model construction is carried out based on the improved MMoE model to obtain a multi-objective joint prediction model to be trained; The multi-objective joint prediction model to be trained includes an input layer, a feature extraction layer, and a private task top layer; The input layer is used to divide the input data of the multi-objective joint prediction model to be trained into time series data and cross-sectional data; The time series data is the training behavior information of nodes in the input data; The cross-sectional data is the training information of the nodes in the input data in the current iteration cycle; The feature extraction layer includes a unique bottom-level module and a common bottom-level module; The unique underlying module includes an LSTM feature extraction network and an MLP feature extraction network; the input data of the LSTM feature extraction network is time-series data; the input data of the MLP feature extraction network is cross-sectional data. The common underlying module is used to extract common features related to subtasks; The private task top layer is used to convert the output of the MMoE layer in the multi-objective joint prediction model to be trained into the prediction result of a specific sub-task; The top layer of the private task consists of a dedicated network of subtasks, and the correspondence between the subtasks and the dedicated network is unique. The dedicated network is used to fit the prediction results corresponding to the sub-tasks; The node performance data is dynamically partitioned to obtain dynamically adjusted data, including: The node performance data is dynamically adjusted in terms of sample size to obtain sample size adjustment data. The sample size adjustment data is partitioned into datasets to obtain dynamically adjusted data; The dynamically adjusted data is processed to obtain training data; Using the training data, the multi-objective joint prediction model to be trained is trained to obtain the trained multi-objective joint prediction model. The trained multi-objective joint prediction model was validated using an elastic relaxation scheme. When the validation result passed, the multi-objective joint prediction model was obtained. The elastic relaxation scheme is used to verify the load balancing performance of the multi-objective joint prediction model. The elastic relaxation scheme controls the communication pace of the load nodes by adjusting the synchronization threshold; Input the performance data of the nodes to be balanced, and perform load balancing through the multi-objective joint prediction model.

2. The load balancing method for computing nodes according to claim 1, characterized in that, The node performance data includes the size of the subset of data in the current training round of the node, the total training time of the current training round, the training time of the node's iteration rounds, and the number of iteration rounds of the node under the current synchronization window.

3. The load balancing method for computing nodes according to claim 1, characterized in that, After validating the trained multi-objective joint prediction model using an elastic relaxation scheme, the method further includes: If the verification result is that the verification fails, continue to collect node performance data and iteratively train the multi-objective joint prediction model to be trained.

4. A load balancing device for computing nodes, characterized in that, The device includes: The data acquisition module is used to collect data through performance analysis tools to obtain node performance data. The model building module is used to build a model based on the improved MMoE model to obtain a multi-objective joint prediction model to be trained. The multi-objective joint prediction model to be trained includes an input layer, a feature extraction layer, and a private task top layer; The input layer is used to divide the input data of the multi-objective joint prediction model to be trained into time series data and cross-sectional data; The time series data is the training behavior information of nodes in the input data; The cross-sectional data is the training information of the nodes in the input data in the current iteration cycle; The feature extraction layer includes a unique bottom-level module and a common bottom-level module; The unique underlying module includes an LSTM feature extraction network and an MLP feature extraction network; the input data of the LSTM feature extraction network is time-series data; the input data of the MLP feature extraction network is cross-sectional data. The common underlying module is used to extract common features related to subtasks; The private task top layer is used to convert the output of the MMoE layer in the multi-objective joint prediction model to be trained into the prediction result of a specific sub-task; The top layer of the private task consists of a dedicated network of subtasks, and the correspondence between the subtasks and the dedicated network is unique. The dedicated network is used to fit the prediction results corresponding to the sub-tasks; The data dynamic adjustment module is used to dynamically partition the node performance data to obtain dynamic adjustment data. The data dynamic adjustment module is further used for: The node performance data is dynamically adjusted in terms of sample size to obtain sample size adjustment data. The sample size adjustment data is partitioned into datasets to obtain dynamically adjusted data; The data processing module is used to process the dynamically adjusted data to obtain training data; The model training module is used to train the multi-objective joint prediction model to be trained using the training data, so as to obtain the trained multi-objective joint prediction model. The model validation module is used to validate the trained multi-objective joint prediction model through an elastic relaxation scheme. When the validation result is passed, the multi-objective joint prediction model is obtained. The elastic relaxation scheme is used to verify the load balancing performance of the multi-objective joint prediction model. The elastic relaxation scheme controls the communication pace of the load nodes by adjusting the synchronization threshold; The load balancing module is used to input the performance data of the nodes to be balanced and perform load balancing through the multi-objective joint prediction model.

Images