A resource prediction and scheduling system for large model training

Abstract

The present invention provides a resource prediction and scheduling system in large model training, which relates to the field of electrical digital data processing, and comprises a resource monitoring module, a prediction and analysis module, a task scheduling module and a feedback display module. The resource monitoring module is used to monitor and collect the status of resources, the prediction and analysis module is used to predict the demand for resources, the task scheduling module schedules tasks based on the prediction results, and the feedback display module is used to collect and display performance information during training. The system can predict the resource usage trend of training tasks, allocate tasks based on the prediction results, and improve the fluency and efficiency of training tasks during operation.

Description

Technical Field

[0001] The invention relates to the field of electrical digital data processing, and in particular to a resource prediction and scheduling system in large model training. Background Art

[0002] At present, many large-scale deep learning tasks have extremely different demands for computing resources during the training process, especially in different time periods of the training phase. For example, in the early stage of training, the model parameters are relatively small and the computing demand is relatively low, while in the later stage of training, as the model gradually converges, the computing demand increases rapidly. This dynamic change in resource demand makes the scheduling and allocation of computing resources complicated. Therefore, traditional resource scheduling methods cannot fully cope with this dynamic and complex resource allocation problem, which often leads to idle or overloaded resources, affecting training efficiency and task completion time.

[0003] Many resource prediction and scheduling systems have been developed. After a lot of searching and reference, it is found that the existing prediction and scheduling systems are such as the system disclosed in the publication number CN114579270B. These systems generally include obtaining tasks to be executed; predicting the tasks to be executed through the prediction model to obtain the resource requirements of the tasks to be executed and the time required to execute the tasks to be executed; scheduling the tasks to be executed through the scheduling model based on the predicted resource requirements of the tasks to be executed and the time required to execute the tasks to be executed, combined with the working status of all executors. However, this system only predicts a fixed resource requirement and time, and is not suitable for training tasks whose resource requirements will change during operation. Summary of the invention

[0004] The purpose of the present invention is to propose a resource prediction and scheduling system in large model training to address the existing deficiencies.

[0005] The present invention adopts the following technical solution:

[0006] A resource prediction and scheduling system in large model training, including a resource monitoring module, a prediction analysis module, a task scheduling module and a feedback display module;

[0007] The resource monitoring module is used to monitor and collect the status of resources, the prediction analysis module is used to predict the demand for resources, the task scheduling module is used to schedule tasks based on the prediction results, and the feedback display module is used to collect and display the performance information during training;

[0008] The resource monitoring module includes a resource collection unit, a history storage unit and an anomaly detection unit, wherein the resource collection unit is used to collect resource usage data, the history storage unit is used to store historical resource status data, and the anomaly detection unit is used to detect anomalies in resource usage;

[0009] The prediction analysis module includes a task receiving unit, a comparison prediction unit and a prediction output unit, wherein the task receiving unit is used to receive task information, the comparison prediction unit is used to compare and analyze historical data to obtain prediction information, and the prediction output unit is used to output a resource demand curve of the task;

[0010] The task scheduling module includes a priority management unit, a task allocation unit and a scheduling execution unit. The priority management unit is used to allocate the priority information of the task. The task allocation unit allocates the training task to the appropriate node based on the prediction result. The scheduling execution unit is used to perform data interaction in the task allocation process.

[0011] The feedback display module includes a performance acquisition unit, an error analysis unit and a display interaction unit. The performance acquisition unit is used to collect performance data of actual training tasks, the error analysis unit is used to analyze the difference between prediction and actual use, and the display interaction unit is used to display performance data and difference information.

[0012] Further, the task receiving unit includes a task information register, a task parsing processor and a reference acquisition processor, wherein the task information register is used to receive and store task information, the task parsing processor is used to parse the received task information to obtain type parameters, and the reference acquisition processor is used to obtain historical data with the same type parameters from the history storage unit as reference information;

[0013] The reference information is a number of groups of functions, each group of functions includes a task function S(t) and a node function P(t), the task function is the amount of resource usage during the task running time period, and the node function is the resource usage rate of the node during the corresponding time.

[0014] Furthermore, the comparison prediction unit includes a function alignment processor, a function segmentation processor and a function prediction processor. The function alignment processor is used to align the function duration in the reference information, the function segmentation processor is used to divide the function into a stable segment and a fluctuating segment, and the function prediction processor is used to analyze the fluctuating segment and obtain the final prediction function.

[0015] Furthermore, the process of processing the task function by the function segment processor includes the following steps:

[0016] S1. Calculate the comprehensive mean function Sa(t) according to the following formula:

[0017] ;

[0018] Where n is the number of function groups in the reference information, Se(t) irepresents the task function after the i-th alignment;

[0019] S2. Calculate the stability function w(t) according to the following formula:

[0020] ;

[0021] in, To stabilize the ratio;

[0022] S3. The time period in which the stability function is less than 1 is set as the stable time period, and the remaining time periods are set as the fluctuating time periods.

[0023] Furthermore, the process of the function prediction processor processing the task function and the node function includes the following steps:

[0024] S21, obtaining the fluctuation segment of the task function and the node function in the corresponding time period;

[0025] S22. Calculate the wave function Sb(t) according to the following formula:

[0026] ;

[0027] S23, calculate the transformation parameters a and b according to the following formula:

[0028] ;

[0029] ;

[0030] Among them, [t1, t2] is the interval of the fluctuation time period;

[0031] S24. Generate a replacement function Rp(t) according to the following formula:

[0032] ;

[0033] Among them, x is the node factor;

[0034] The replacement function replaces the corresponding fluctuation segment in the comprehensive mean function;

[0035] S25. Repeat steps S21 to S24 until all fluctuation segments in the comprehensive mean function are processed.

[0036] The beneficial effects achieved by the present invention are:

[0037] This system directly predicts the resource usage of the task during the entire running process at the beginning of task allocation, and allocates tasks according to the prediction results. Compared with predicting during task running, although the accuracy of the prediction is reduced, the consumption of resources during the prediction process is greatly reduced, so that more resources can be used in running training tasks. At the same time, through comparative analysis of historical data, the prediction function is divided into stable segment and fluctuating segment, and factor adjustment is introduced for the fluctuating segment to improve the accuracy of the prediction, so that the accuracy can be close to the effect of real-time prediction.

[0038] To further understand the features and technical contents of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are only for reference and description and are not intended to limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the overall structural framework of the present invention;

[0040] Figure 2 This is a schematic diagram of the resource monitoring module of the present invention;

[0041] Figure 3 This is a schematic diagram of the forecast analysis module of the present invention;

[0042] Figure 4 This is a schematic diagram of the task scheduling module of the present invention;

[0043] Figure 5 It is a schematic diagram of the feedback display module of the present invention;

[0044] Figure 6 This is a comparison chart of the experimental test results of the present invention. DETAILED DESCRIPTION

[0045] The following is an explanation of the embodiments of the present invention through specific embodiments. Those skilled in the art can understand the advantages and effects of the present invention from the contents disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments, and the details in this specification can also be modified and changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. In addition, the drawings of the present invention are only simple schematic illustrations and are not depicted according to actual sizes. It is stated in advance. The following embodiments will further explain the relevant technical contents of the present invention in detail, but the disclosed contents are not intended to limit the scope of protection of the present invention.

[0046] Embodiment 1: This embodiment provides a resource prediction and scheduling system in large model training, combined with Figure 1 , including resource monitoring module, prediction and analysis module, task scheduling module and feedback display module.

[0047] The resource monitoring module is used to monitor and collect the status of resources, the prediction analysis module is used to predict the demand for resources, the task scheduling module schedules tasks based on the prediction results, and the feedback display module is used to collect and display performance information during training.

[0048] The resource monitoring module includes a resource collection unit, a history storage unit and an anomaly detection unit. The resource collection unit is used to collect resource usage data, the history storage unit is used to store historical resource status data, and the anomaly detection unit is used to detect anomalies in resource usage.

[0049] The prediction analysis module includes a task receiving unit, a comparison prediction unit and a prediction output unit. The task receiving unit is used to receive task information, the comparison prediction unit is used to compare and analyze historical data to obtain prediction information, and the prediction output unit is used to output the resource demand curve of the task.

[0050] The task scheduling module includes a priority management unit, a task allocation unit and a scheduling execution unit. The priority management unit is used to allocate task priority information. The task allocation unit allocates training tasks to appropriate nodes based on prediction results. The scheduling execution unit is used to perform data interaction in the task allocation process.

[0051] The feedback display module includes a performance acquisition unit, an error analysis unit and a display interaction unit. The performance acquisition unit is used to collect performance data of actual training tasks, the error analysis unit is used to analyze the difference between prediction and actual use, and the display interaction unit is used to display performance data and difference information.

[0052] The task receiving unit includes a task information register, a task parsing processor and a reference acquisition processor. The task information register is used to receive and store task information. The task parsing processor is used to parse the received task information to obtain type parameters. The reference acquisition processor is used to obtain historical data with the same type parameters from the historical storage unit as reference information.

[0053] The reference information is a number of groups of functions, each group of functions includes a task function S(t) and a node function P(t), the task function is the amount of resource usage during the task running time period, and the node function is the resource usage rate of the node during the corresponding time.

[0054] The comparison prediction unit includes a function alignment processor, a function segmentation processor and a function prediction processor. The function alignment processor is used to align the function duration in the reference information, the function segmentation processor is used to divide the function into a stable segment and a fluctuating segment, and the function prediction processor is used to analyze the fluctuating segment and obtain the final prediction function.

[0055] The process of processing the task function by the function segment processor comprises the following steps:

[0056] S1. Calculate the comprehensive mean function Sa(t) according to the following formula:

[0057] .

[0058] Where n is the number of function groups in the reference information, Se(t) i Represents the task function after the i-th alignment.

[0059] S2. Calculate the stability function w(t) according to the following formula:

[0060] .

[0061] in, To stabilize the ratio.

[0062] S3. The time period in which the stability function is less than 1 is set as the stable time period, and the remaining time periods are set as the fluctuating time periods.

[0063] The process of the function prediction processor processing the task function and the node function includes the following steps:

[0064] S21. Obtain the fluctuation segment of the task function and the node function in the corresponding time period.

[0065] S22. Calculate the wave function Sb(t) according to the following formula:

[0066] .

[0067] S23, calculate the transformation parameters a and b according to the following formula:

[0068] .

[0069] .

[0070] Among them, [t1, t2] is the interval of the fluctuation time period.

[0071] S24. Generate a replacement function Rp(t) according to the following formula:

[0072] .

[0073] Among them, x is the node factor.

[0074] The replacement function replaces the corresponding fluctuation segment in the comprehensive mean function.

[0075] S25. Repeat steps S21 to S24 until all fluctuation segments in the comprehensive mean function are processed.

[0076] Embodiment 2: This embodiment includes all the contents of Embodiment 1, and provides a resource prediction and scheduling system in large model training, including a resource monitoring module, a prediction and analysis module, a task scheduling module and a feedback display module.

[0077] The resource monitoring module is used to monitor and collect the status of resources, the prediction analysis module is used to predict the demand for resources, the task scheduling module schedules tasks based on the prediction results, and the feedback display module is used to collect and display performance information during training.

[0078] Combination Figure 2 The resource monitoring module includes a resource collection unit, a history storage unit and an anomaly detection unit. The resource collection unit is used to collect resource usage data, the history storage unit is used to store historical resource status data, and the anomaly detection unit is used to detect anomalies in resource usage.

[0079] Combination Figure 3 The prediction and analysis module includes a task receiving unit, a comparison prediction unit and a prediction output unit. The task receiving unit is used to receive task information, the comparison prediction unit is used to compare and analyze historical data to obtain prediction information, and the prediction output unit is used to output the resource demand curve of the task.

[0080] Combination Figure 4 The task scheduling module includes a priority management unit, a task allocation unit and a scheduling execution unit. The priority management unit is used to allocate task priority information. The task allocation unit allocates training tasks to appropriate nodes based on prediction results. The scheduling execution unit is used to perform data interaction in the task allocation process.

[0081] Combination Figure 5 The feedback display module includes a performance acquisition unit, an error analysis unit and a display interaction unit. The performance acquisition unit is used to collect performance data of actual training tasks, the error analysis unit is used to analyze the difference between prediction and actual use, and the display interaction unit is used to display performance data and difference information.

[0082] The resource acquisition unit includes a node management processor, a monitoring acquisition processor and a data preprocessor. The node management processor is used to manage resource node information, the monitoring acquisition processor is used to collect real-time data of each resource node, and the data preprocessor is used to format the collected real-time data.

[0083] The historical storage unit includes a resource status register, a task status register and a mapping management processor. The resource status register is used to store resource status information of resource nodes, the task status register is used to store resource usage status information of tasks during runtime, and the mapping management processor is used to map the task usage status information with the node's resource status information.

[0084] The abnormality detection unit includes an abnormality judgment processor and a data deletion processor. The abnormality judgment processor is used to judge whether the resource usage of the node handles the abnormal state, and the data deletion processor is used to delete the data in the historical storage unit that is in the abnormal state.

[0085] The task receiving unit includes a task information register, a task parsing processor and a reference acquisition processor. The task information register is used to receive and store task information. The task parsing processor is used to parse the received task information to obtain type parameters. The reference acquisition processor is used to obtain historical data with the same type parameters from the historical storage unit as reference information.

[0086] The reference information is a number of groups of functions, each group of functions includes a task function S(t) and a node function P(t), the task function is the amount of resource usage during the task running time period, and the node function is the resource usage rate of the node during the corresponding time.

[0087] The comparison prediction unit includes a function alignment processor, a function segmentation processor and a function prediction processor. The function alignment processor is used to align the function duration in the reference information, the function segmentation processor is used to divide the function into a stable segment and a fluctuating segment, and the function prediction processor is used to analyze the fluctuating segment and obtain the final prediction function.

[0088] The function alignment processor performs alignment conversion on each group of functions according to the following formula:

[0089] .

[0090] .

[0091] Among them, T1 is the task running time, T0 is the alignment time, Se(t) is the task function after alignment, and Pe(t) is the node function after alignment.

[0092] The process of processing the task function by the function segment processor comprises the following steps:

[0093] S1. Calculate the comprehensive mean function Sa(t) according to the following formula:

[0094] .

[0095] Where n is the number of function groups in the reference information, Se(t) i Represents the task function after the i-th alignment.

[0096] S2. Calculate the stability function w(t) according to the following formula:

[0097] .

[0098] in, To stabilize the ratio.

[0099] S3. The time period in which the stability function is less than 1 is set as the stable time period, and the remaining time periods are set as the fluctuating time periods.

[0100] The process of the function prediction processor processing the task function and the node function includes the following steps:

[0101] S21. Obtain the fluctuation segment of the task function and the node function in the corresponding time period.

[0102] S22. Calculate the wave function Sb(t) according to the following formula:

[0103] .

[0104] S23, calculate the transformation parameters a and b according to the following formula:

[0105] .

[0106] .

[0107] Among them, [t1, t2] is the interval of the fluctuation time period.

[0108] S24. Generate a replacement function Rp(t) according to the following formula:

[0109] .

[0110] Where x is the node factor.

[0111] The replacement function replaces the corresponding fluctuation segment in the comprehensive mean function.

[0112] S25. Repeat steps S21 to S24 until all fluctuation segments in the comprehensive mean function are processed.

[0113] The prediction output unit includes a feature extraction processor, a prediction packaging processor and a data packet output processor. The feature extraction processor is used to receive the comprehensive mean function and extract feature information therein. The prediction packaging processor is used to package the feature information, function information and task information to obtain a prediction data packet. The data packet output processor is used to send the prediction data packet to the task scheduling module.

[0114] The priority management unit includes a priority rule register, a priority assessment processor and a task sorting processor. The priority rule register is used to store priority assessment rules. The priority assessment processor rates tasks in the predicted data packet based on the assessment rules. The task sorting processor sorts tasks based on the rating results.

[0115] The task allocation unit includes an allocation record processor, a factor screening processor and a node selection processor. The allocation record processor is used to record the prediction function information of the task being run by each node, the factor screening processor is used to obtain the node factor and screen out the target node, and the node selection processor is used to select the node to which the task is allocated from the target node.

[0116] The factor screening processor substitutes the real-time resource utilization of the node as the node factor into the comprehensive mean function to obtain a complete prediction function. When the remaining resource amount of the node can meet the requirements of the prediction function, the node is used as the target node.

[0117] The node selection processor uses the target node with the lowest real-time resource utilization after running the task as the actually allocated node.

[0118] The scheduling execution unit includes a data management processor and a task delivery processor. The data management processor acquires and manages the data required for executing the task according to the task information, and the task delivery processor is used to send the task data to the assigned node.

[0119] The i appearing in the above text is an ordinal number used to indicate a sequence number.

[0120] Some code information of this system is described as follows:

[0121] class PredictionAnalyzer:

[0122] def __init__(self, historical_data):

[0123] self.historical_data = historical_data # Store historical data

[0124] def receive_task_info(self, task_info):

[0125] """

[0126] Receive task information (such as task type, resource requirements, etc.)

[0127] """

[0128] # Example: Print task information

[0129] print("Received task information:", task_info)

[0130] def compare_and_predict(self, task_info):

[0131] """

[0132] Compare historical data to forecast resource demand

[0133] """

[0134] predicted_resource_usage = {

[0135] 'CPU': 75, # Predicted CPU demand

[0136] 'GPU': 60, # Predicted GPU requirements

[0137] 'Memory': 8 # Predicted memory requirements

[0138] }

[0139] return predicted_resource_usage

[0140] def output_prediction(self, predicted_resources):

[0141] """

[0142] Resource requirement curve for output tasks (printing is used as an example here)

[0143] """

[0144] print("Predicted resource usage:", predicted_resources)

[0145] class TaskScheduler:

[0146] def __init__(self):

[0147] self.tasks = []

[0148] def assign_priority(self, task_info):

[0149] """

[0150] Assign priorities to tasks based on their importance, priority, etc.

[0151] """

[0152] priority = "High" if task_info['CPU'] > 70 else "Low"

[0153] task_info['Priority'] = priority

[0154] return task_info

[0155] def assign_task(self, task_info):

[0156] """

[0157] Assign tasks to appropriate nodes based on prediction results

[0158] """

[0159] # Here we simulate task allocation (assuming nodes are allocated based on resource requirements)

[0160] node = "Node_1" if task_info['GPU'] < 70 else "Node_2"

[0161] task_info['Assigned_Node'] = node

[0162] self.tasks.append(task_info)

[0163] def execute_schedule(self):

[0164] """

[0165] Execute task scheduling (simulate scheduling process)

[0166] """

[0167] for task in self.tasks:

[0168] print(f"Executing task on {task['Assigned_Node']} with priority {task['Priority']}").

[0169] Now, 10 sample tasks are processed in the common system and this system respectively, and the resource consumption in the prediction process is measured and compared. Figure 6 The effect diagram shown.

[0170] The contents disclosed above are only preferred feasible embodiments of the present invention, and do not limit the protection scope of the present invention. Therefore, all equivalent technical changes made using the contents of the present invention specification and drawings are included in the protection scope of the present invention. In addition, the elements therein can be updated as technology develops.

Claims

1. A resource prediction and scheduling system in large model training, characterized in that: It includes resource monitoring module, prediction analysis module, task scheduling module and feedback display module; The resource monitoring module is used to monitor and collect the status of resources, the prediction analysis module is used to predict the demand for resources, the task scheduling module is used to schedule tasks based on the prediction results, and the feedback display module is used to collect and display the performance information during training; The resource monitoring module includes a resource collection unit, a history storage unit and an anomaly detection unit, wherein the resource collection unit is used to collect resource usage data, the history storage unit is used to store historical resource status data, and the anomaly detection unit is used to detect anomalies in resource usage; The prediction analysis module includes a task receiving unit, a comparison prediction unit and a prediction output unit, wherein the task receiving unit is used to receive task information, the comparison prediction unit is used to compare and analyze historical data to obtain prediction information, and the prediction output unit is used to output a resource demand curve of the task; The task scheduling module includes a priority management unit, a task allocation unit and a scheduling execution unit. The priority management unit is used to allocate the priority information of the task. The task allocation unit allocates the training task to the appropriate node based on the prediction result. The scheduling execution unit is used to perform data interaction in the task allocation process. The feedback display module includes a performance acquisition unit, an error analysis unit and a display interaction unit, wherein the performance acquisition unit is used to collect performance data of the actual training task, the error analysis unit is used to analyze the difference between the prediction and the actual use, and the display interaction unit is used to display the performance data and the difference information; The task receiving unit includes a task information register, a task parsing processor and a reference acquisition processor, wherein the task information register is used to receive and store task information, the task parsing processor is used to parse the received task information to obtain type parameters, and the reference acquisition processor is used to obtain historical data with the same type parameters from the history storage unit as reference information; The reference information is a number of groups of functions, each group of functions includes a task function S(t) and a node function P(t), the task function is the amount of resource usage during the task running time period, and the node function is the resource usage rate of the node during the corresponding time; The comparison prediction unit includes a function alignment processor, a function segmentation processor and a function prediction processor, wherein the function alignment processor is used to align the function duration in the reference information, the function segmentation processor is used to divide the function into a stable segment and a fluctuating segment, and the function prediction processor is used to analyze the fluctuating segment and obtain a final prediction function; The function alignment processor performs alignment conversion on each group of functions according to the following formula: ； ； Where T1 is the task running time, T0 is the alignment time, Se(t) is the task function after alignment, and Pe(t) is the node function after alignment; The process of processing the task function by the function segment processor comprises the following steps: S1. Calculate the comprehensive mean function Sa(t) according to the following formula: ； Where n is the number of function groups in the reference information, Se(t) i represents the task function after the i-th alignment; S2. Calculate the stability function w(t) according to the following formula: ； in, To stabilize the ratio; S3, setting the time period when the stability function is less than 1 as the stable time period, and the remaining time periods as the fluctuating time periods; The process of the function prediction processor processing the task function and the node function includes the following steps: S21, obtaining the fluctuation segment of the task function and the node function in the corresponding time period; S22. Calculate the wave function Sb(t) according to the following formula: ； S23, calculate the transformation parameters a and b according to the following formula: ； ； Among them, [t1, t2] is the interval of the fluctuation time period; S24. Generate a replacement function Rp(t) according to the following formula: ； Among them, x is the node factor; The replacement function replaces the corresponding fluctuation segment in the comprehensive mean function; S25. Repeat steps S21 to S24 until all fluctuation segments in the comprehensive mean function are processed.

Images