Resource prediction and scheduling system in large model training

Abstract

The invention provides a resource prediction and scheduling system in large model training, which relates to the field of electric digital data processing, and comprises a resource monitoring module, a prediction analysis module, a task scheduling module and a feedback display module, the prediction analysis module is used for predicting resource demands, the task scheduling module is used for scheduling tasks based on prediction results, and the feedback display module is used for collecting performance information during training and displaying the performance information; according to the system, the resource use trend of the training tasks can be predicted, the tasks are allocated based on the prediction result, and the fluency and the operation efficiency of the training tasks during operation are improved.

Description

Technical Field

[0001] The present invention relates to the field of electronic digital data processing, and in particular to a resource prediction and scheduling system in large model training. Background Art

[0002] At present, many large-scale deep learning tasks have extremely different demands for computing resources during the training process, especially in different time periods of the training phase. For example, in the early stages of training, the model parameters are relatively small and the computing demand is low. In the later stages of training, as the model gradually converges, the computing demand increases rapidly. This dynamic change in resource demand makes the scheduling and allocation of computing resources complicated. Therefore, traditional resource scheduling methods cannot fully cope with this dynamic and complex resource allocation problem, often resulting in idle or overloaded resources, affecting training efficiency and task completion time.

[0003] Numerous resource prediction and scheduling systems have been developed. After extensive research and reference, I discovered existing prediction and scheduling systems, such as the one disclosed in Publication No. CN114579270B. These systems generally involve obtaining pending tasks; using a prediction model to predict the pending tasks' resource requirements and the time required to execute them; and then scheduling the pending tasks using a scheduling model based on the predicted resource requirements and time required, combined with the operating status of all executors. However, this system only predicts a fixed resource requirement and time, making it unsuitable for training tasks whose resource requirements can change during runtime. Summary of the Invention

[0004] The purpose of the present invention is to address the existing deficiencies and propose a resource prediction and scheduling system in large model training.

[0005] The present invention adopts the following technical solutions: A resource prediction and scheduling system for large model training, including a resource monitoring module, a prediction and analysis module, a task scheduling module, and a feedback display module; The resource monitoring module is used to monitor and collect resource status, the prediction analysis module is used to predict resource demand, the task scheduling module is used to schedule tasks based on the prediction results, and the feedback display module is used to collect and display performance information during training; The resource monitoring module includes a resource collection unit, a history storage unit and an anomaly detection unit, wherein the resource collection unit is used to collect resource usage data, the history storage unit is used to store historical resource status data, and the anomaly detection unit is used to detect anomalies in resource usage; The prediction analysis module includes a task receiving unit, a comparison and prediction unit, and a prediction output unit. The task receiving unit is used to receive task information, the comparison and prediction unit is used to compare and analyze historical data to obtain prediction information, and the prediction output unit is used to output the resource demand curve of the task; The task scheduling module includes a priority management unit, a task allocation unit, and a scheduling execution unit. The priority management unit is used to allocate priority information of tasks. The task allocation unit allocates training tasks to appropriate nodes based on prediction results. The scheduling execution unit is used to execute data interaction during the task allocation process. The feedback display module includes a performance acquisition unit, an error analysis unit and a display interaction unit. The performance acquisition unit is used to collect performance data of actual training tasks, the error analysis unit is used to analyze the difference between prediction and actual use, and the display interaction unit is used to display performance data and difference information.

[0006] Furthermore, the task receiving unit includes a task information register, a task parsing processor, and a reference acquisition processor. The task information register is used to receive and store task information. The task parsing processor is used to parse the received task information to obtain type parameters. The reference acquisition processor is used to obtain historical data with the same type parameters from the history storage unit as reference information. The reference information is a number of groups of functions, each group of functions includes a task function S(t) and a node function P(t), the task function is the amount of resource usage during the task running time period, and the node function is the resource utilization rate of the node during the corresponding time.

[0007] Furthermore, the comparison prediction unit includes a function alignment processor, a function segmentation processor and a function prediction processor. The function alignment processor is used to align the function duration in the reference information, the function segmentation processor is used to divide the function into stable segments and fluctuating segments, and the function prediction processor is used to analyze the fluctuating segments and obtain the final prediction function.

[0008] Furthermore, the process of processing the task function by the function segment processor includes the following steps: S1. Calculate the comprehensive mean function Sa(t) according to the following formula: ; Where n is the number of function groups in the reference information, Se(t) i represents the task function after the i-th alignment; S2. Calculate the stability function w(t) according to the following formula: ; in, To stabilize the ratio; S3. The time period in which the stability function is less than 1 is set as the stable time period, and the remaining time periods are set as the fluctuating time period.

[0009] Furthermore, the process of the function prediction processor processing the task function and the node function includes the following steps: S21, obtaining the fluctuation segment of the task function and the node function within the corresponding time period; S22. Calculate the wave function Sb(t) according to the following formula: ; S23. Calculate the transformation parameters a and b according to the following formula: ; ; Among them, [t1, t2] is the interval of the fluctuation time period; S24. Generate a replacement function Rp(t) according to the following formula: ; Where x is the node factor; The replacement function replaces the corresponding fluctuation segment in the comprehensive mean function; S25. Repeat steps S21 to S24 until all fluctuation segments in the comprehensive mean function are processed.

[0010] The beneficial effects achieved by the present invention are: This system directly predicts the resource usage of the task during the entire operation process at the beginning of the task assignment, and assigns tasks based on the prediction results. Compared with predicting during task execution, although this reduces the accuracy of the prediction, it greatly reduces the consumption of resources during the prediction process, allowing more resources to be used in running training tasks. At the same time, through comparative analysis of historical data, the prediction function is divided into stable and fluctuating segments, and factor adjustment is introduced for the fluctuating segment to improve the accuracy of the prediction, making the accuracy close to that of real-time prediction.

[0011] To further understand the features and technical contents of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are only for reference and illustration and are not intended to limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a schematic diagram of the overall structural framework of the present invention; Figure 2 This is a schematic diagram of the resource monitoring module of the present invention; Figure 3 This is a schematic diagram of the forecast analysis module of the present invention; Figure 4 This is a schematic diagram of the task scheduling module of the present invention; Figure 5 This is a schematic diagram of the feedback display module of the present invention; Figure 6 This is a comparison chart of the experimental test results of the present invention. DETAILED DESCRIPTION

[0013] The following is an explanation of the embodiments of the present invention through specific embodiments. Those skilled in the art can understand the advantages and effects of the present invention from the contents disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments, and the details in this specification can also be modified and changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. In addition, the drawings of the present invention are only simple schematic illustrations and are not depicted in actual size. It is stated in advance. The following embodiments will further explain the relevant technical content of the present invention in detail, but the disclosed content is not intended to limit the scope of protection of the present invention.

[0014] Example 1: This example provides a resource prediction and scheduling system for large model training, combining Figure 1 , including resource monitoring module, prediction analysis module, task scheduling module and feedback display module.

[0015] The resource monitoring module is used to monitor and collect the status of resources, the prediction analysis module is used to predict the demand for resources, the task scheduling module is used to schedule tasks based on the prediction results, and the feedback display module is used to collect and display performance information during training.

[0016] The resource monitoring module includes a resource collection unit, a history storage unit and an anomaly detection unit. The resource collection unit is used to collect resource usage data, the history storage unit is used to store historical resource status data, and the anomaly detection unit is used to detect anomalies in resource usage.

[0017] The prediction analysis module includes a task receiving unit, a comparison prediction unit and a prediction output unit. The task receiving unit is used to receive task information, the comparison prediction unit is used to compare and analyze historical data to obtain prediction information, and the prediction output unit is used to output the resource demand curve of the task.

[0018] The task scheduling module includes a priority management unit, a task allocation unit and a scheduling execution unit. The priority management unit is used to allocate task priority information. The task allocation unit allocates training tasks to appropriate nodes based on prediction results. The scheduling execution unit is used to execute data interaction in the task allocation process.

[0019] The feedback display module includes a performance acquisition unit, an error analysis unit and a display interaction unit. The performance acquisition unit is used to collect performance data of actual training tasks, the error analysis unit is used to analyze the difference between prediction and actual use, and the display interaction unit is used to display performance data and difference information.

[0020] The task receiving unit includes a task information register, a task parsing processor and a reference acquisition processor. The task information register is used to receive and store task information. The task parsing processor is used to parse the received task information to obtain type parameters. The reference acquisition processor is used to obtain historical data with the same type parameters from the history storage unit as reference information.

[0021] The reference information is a number of groups of functions, each group of functions includes a task function S(t) and a node function P(t), the task function is the amount of resource usage during the task running time period, and the node function is the resource utilization rate of the node during the corresponding time.

[0022] The comparison prediction unit includes a function alignment processor, a function segmentation processor and a function prediction processor. The function alignment processor is used to align the function duration in the reference information, the function segmentation processor is used to divide the function into a stable segment and a fluctuating segment, and the function prediction processor is used to analyze the fluctuating segment and obtain the final prediction function.

[0023] The process of processing the task function by the function segment processor includes the following steps: S1. Calculate the comprehensive mean function Sa(t) according to the following formula: .

[0024] Where n is the number of function groups in the reference information, Se(t) i Represents the task function after the i-th alignment.

[0025] S2. Calculate the stability function w(t) according to the following formula: .

[0026] in, To stabilize the ratio.

[0027] S3. The time period in which the stability function is less than 1 is set as the stable time period, and the remaining time periods are set as the fluctuating time period.

[0028] The process of the function prediction processor processing the task function and the node function includes the following steps: S21. Obtain the fluctuation segment of the task function and the node function within the corresponding time period.

[0029] S22. Calculate the wave function Sb(t) according to the following formula: .

[0030] S23. Calculate the transformation parameters a and b according to the following formula: .

[0031] .

[0032] Among them, [t1, t2] is the interval of the fluctuation time period.

[0033] S24. Generate a replacement function Rp(t) according to the following formula: .

[0034] Where x is the node factor.

[0035] The replacement function replaces the corresponding fluctuation segment in the comprehensive mean function.

[0036] S25. Repeat steps S21 to S24 until all fluctuation segments in the comprehensive mean function are processed.

[0037] Example 2: This example includes all the contents of Example 1, and provides a resource prediction and scheduling system for large model training, including a resource monitoring module, a prediction analysis module, a task scheduling module and a feedback display module.

[0038] The resource monitoring module is used to monitor and collect the status of resources, the prediction analysis module is used to predict the demand for resources, the task scheduling module is used to schedule tasks based on the prediction results, and the feedback display module is used to collect and display performance information during training.

[0039] Combine Figure 2 The resource monitoring module includes a resource collection unit, a history storage unit and an anomaly detection unit. The resource collection unit is used to collect resource usage data, the history storage unit is used to store historical resource status data, and the anomaly detection unit is used to detect anomalies in resource usage.

[0040] Combine Figure 3 The prediction analysis module includes a task receiving unit, a comparison prediction unit and a prediction output unit. The task receiving unit is used to receive task information, the comparison prediction unit is used to compare and analyze historical data to obtain prediction information, and the prediction output unit is used to output the resource demand curve of the task.

[0041] Combine Figure 4The task scheduling module includes a priority management unit, a task allocation unit and a scheduling execution unit. The priority management unit is used to allocate the priority information of the task. The task allocation unit allocates the training task to the appropriate node based on the prediction results. The scheduling execution unit is used to perform data interaction in the task allocation process.

[0042] Combine Figure 5 The feedback display module includes a performance acquisition unit, an error analysis unit and a display interaction unit. The performance acquisition unit is used to collect performance data of actual training tasks, the error analysis unit is used to analyze the difference between prediction and actual use, and the display interaction unit is used to display performance data and difference information.

[0043] The resource acquisition unit includes a node management processor, a monitoring and acquisition processor and a data preprocessor. The node management processor is used to manage resource node information, the monitoring and acquisition processor is used to collect real-time data of each resource node, and the data preprocessor is used to format the collected real-time data.

[0044] The historical storage unit includes a resource status register, a task status register and a mapping management processor. The resource status register is used to store the resource status information of the resource node, the task status register is used to store the resource usage status information of the task during operation, and the mapping management processor is used to map the task usage status information with the node's resource status information.

[0045] The abnormality detection unit includes an abnormality judgment processor and a data deletion processor. The abnormality judgment processor is used to judge whether the resource usage of the node handles the abnormal state, and the data deletion processor is used to delete the data in the history storage unit that is in the abnormal state.

[0046] The task receiving unit includes a task information register, a task parsing processor and a reference acquisition processor. The task information register is used to receive and store task information. The task parsing processor is used to parse the received task information to obtain type parameters. The reference acquisition processor is used to obtain historical data with the same type parameters from the history storage unit as reference information.

[0047] The reference information is a number of groups of functions, each group of functions includes a task function S(t) and a node function P(t), the task function is the amount of resource usage during the task running time period, and the node function is the resource utilization rate of the node during the corresponding time.

[0048] The comparison prediction unit includes a function alignment processor, a function segmentation processor and a function prediction processor. The function alignment processor is used to align the function duration in the reference information, the function segmentation processor is used to divide the function into a stable segment and a fluctuating segment, and the function prediction processor is used to analyze the fluctuating segment and obtain the final prediction function.

[0049] The function alignment processor performs alignment conversion on each group of functions according to the following formula: .

[0050] .

[0051] Among them, T1 is the task running time, T0 is the alignment time, Se(t) is the task function after alignment, and Pe(t) is the node function after alignment.

[0052] The process of processing the task function by the function segment processor includes the following steps: S1. Calculate the comprehensive mean function Sa(t) according to the following formula: .

[0053] Where n is the number of function groups in the reference information, Se(t) i Represents the task function after the i-th alignment.

[0054] S2. Calculate the stability function w(t) according to the following formula: .

[0055] in, To stabilize the ratio.

[0056] S3. The time period in which the stability function is less than 1 is set as the stable time period, and the remaining time periods are set as the fluctuating time period.

[0057] The process of the function prediction processor processing the task function and the node function includes the following steps: S21. Obtain the fluctuation segment of the task function and the node function within the corresponding time period.

[0058] S22. Calculate the wave function Sb(t) according to the following formula: .

[0059] S23. Calculate the transformation parameters a and b according to the following formula: .

[0060] .

[0061] Among them, [t1, t2] is the interval of the fluctuation time period.

[0062] S24. Generate a replacement function Rp(t) according to the following formula: .

[0063] Where x is the node factor.

[0064] The replacement function replaces the corresponding fluctuation segment in the comprehensive mean function.

[0065] S25. Repeat steps S21 to S24 until all fluctuation segments in the comprehensive mean function are processed.

[0066] The prediction output unit includes a feature extraction processor, a prediction packaging processor and a data packet output processor. The feature extraction processor is used to receive the comprehensive mean function and extract the feature information therein. The prediction packaging processor is used to package the feature information, function information and task information to obtain a prediction data packet. The data packet output processor is used to send the prediction data packet to the task scheduling module.

[0067] The priority management unit includes a priority rule register, a priority assessment processor and a task sorting processor. The priority rule register is used to store priority assessment rules. The priority assessment processor rates tasks in the predicted data packet based on the assessment rules. The task sorting processor sorts tasks based on the rating results.

[0068] The task allocation unit includes an allocation record processor, a factor screening processor and a node selection processor. The allocation record processor is used to record the prediction function information of the task running on each node, the factor screening processor is used to obtain the node factor and filter out the target node, and the node selection processor is used to select the node to which the task is allocated from the target node.

[0069] The factor screening processor substitutes the real-time resource utilization of the node as the node factor into the comprehensive mean function to obtain a complete prediction function. When the remaining resource amount of the node can meet the requirements of the prediction function, the node is used as the target node.

[0070] The node selection processor uses the target node with the lowest real-time resource utilization after running the task as the actually allocated node.

[0071] The scheduling execution unit includes a data management processor and a task delivery processor. The data management processor obtains and manages the data required for executing the task according to the task information. The task delivery processor is used to send the task data to the assigned node.

[0072] The i appearing above is an ordinal number used to represent a sequence number.

[0073] Some code information of this system is described as follows: class PredictionAnalyzer: def __init__(self, historical_data): self.historical_data = historical_data # Store historical data def receive_task_info(self, task_info): """ Receive task information (such as task type, resource requirements, etc.) """ # Example: Print task information print("Received task information:", task_info) def compare_and_predict(self, task_info): """ Compare historical data to predict resource demand """ predicted_resource_usage = { 'CPU': 75, # predicted CPU demand 'GPU': 60, # predicted GPU requirements 'Memory': 8 # predicted memory requirements } return predicted_resource_usage def output_prediction(self, predicted_resources): """ Output task resource demand curve (printing is used as an example here) """ print("Predicted resource usage:", predicted_resources) class TaskScheduler: def __init__(self): self.tasks = [] def assign_priority(self, task_info): """ Assign priorities to tasks based on their importance, priority, etc. """ priority = "High" if task_info['CPU'] > 70 else "Low" task_info['Priority'] = priority return task_info def assign_task(self, task_info): """ Assign tasks to appropriate nodes based on prediction results """ # Here we simulate task allocation (assuming nodes are allocated based on resource requirements) node = "Node_1" if task_info['GPU'] < 70 else "Node_2" task_info['Assigned_Node'] = node self.tasks.append(task_info) def execute_schedule(self): """ Execute task scheduling (simulate scheduling process) """ for task in self.tasks: print(f"Executing task on {task['Assigned_Node']} with priority {task['Priority']}").

[0074] Now 10 sample tasks are processed in the common system and this system respectively, and the resource consumption in the prediction process is measured and compared. Figure 6 The effect diagram shown.

[0075] The contents disclosed above are only preferred feasible embodiments of the present invention and do not limit the protection scope of the present invention. Therefore, all equivalent technical changes made using the contents of the present invention description and drawings are included in the protection scope of the present invention. In addition, the elements therein can be updated as technology develops.

Claims

1. A resource prediction and scheduling system in large model training, characterized in that: It includes resource monitoring module, prediction analysis module, task scheduling module and feedback display module; The resource monitoring module is used to monitor and collect the status of resources, the prediction analysis module is used to predict the demand for resources, the task scheduling module is used to schedule tasks based on the prediction results, and the feedback display module is used to collect and display the performance information during training; The resource monitoring module includes a resource collection unit, a history storage unit and an anomaly detection unit, wherein the resource collection unit is used to collect resource usage data, the history storage unit is used to store historical resource status data, and the anomaly detection unit is used to detect anomalies in resource usage; The prediction analysis module includes a task receiving unit, a comparison prediction unit and a prediction output unit, wherein the task receiving unit is used to receive task information, the comparison prediction unit is used to compare and analyze historical data to obtain prediction information, and the prediction output unit is used to output a resource demand curve of the task; The task scheduling module includes a priority management unit, a task allocation unit and a scheduling execution unit. The priority management unit is used to allocate the priority information of the task. The task allocation unit allocates the training task to the appropriate node based on the prediction result. The scheduling execution unit is used to perform data interaction in the task allocation process. The feedback display module includes a performance acquisition unit, an error analysis unit and a display interaction unit. The performance acquisition unit is used to collect performance data of actual training tasks, the error analysis unit is used to analyze the difference between prediction and actual use, and the display interaction unit is used to display performance data and difference information.

2. A resource prediction and scheduling system in large model training as claimed in claim 1, characterized in that: The task receiving unit includes a task information register, a task parsing processor and a reference acquisition processor, wherein the task information register is used to receive and store task information, the task parsing processor is used to parse the received task information to obtain type parameters, and the reference acquisition processor is used to obtain historical data with the same type parameters from the history storage unit as reference information; The reference information is a number of groups of functions, each group of functions includes a task function S(t) and a node function P(t), the task function is the amount of resource usage during the task running time period, and the node function is the resource usage rate of the node during the corresponding time.

3. A resource prediction and scheduling system in large model training as described in claim 2, characterized in that: The comparison prediction unit includes a function alignment processor, a function segmentation processor and a function prediction processor. The function alignment processor is used to align the function duration in the reference information, the function segmentation processor is used to divide the function into a stable segment and a fluctuating segment, and the function prediction processor is used to analyze the fluctuating segment and obtain the final prediction function.

4. A resource prediction and scheduling system in large model training as claimed in claim 3, characterized in that: The process of processing the task function by the function segment processor comprises the following steps: S1. Calculate the comprehensive mean function Sa(t) according to the following formula: ； Where n is the number of function groups in the reference information, Se(t) i represents the task function after the i-th alignment; S2. Calculate the stability function w(t) according to the following formula: ； in, To stabilize the ratio; S3. The time period in which the stability function is less than 1 is set as the stable time period, and the remaining time periods are set as the fluctuating time periods.

5. A resource prediction and scheduling system in large model training as claimed in claim 4, characterized in that: The process of the function prediction processor processing the task function and the node function includes the following steps: S21, obtaining the fluctuation segment of the task function and the node function in the corresponding time period; S22. Calculate the wave function Sb(t) according to the following formula: ； S23, calculate the transformation parameters a and b according to the following formula: ； ； Among them, [t1, t2] is the interval of the fluctuation time period; S24. Generate a replacement function Rp(t) according to the following formula: ； Among them, x is the node factor; The replacement function replaces the corresponding fluctuation segment in the comprehensive mean function; S25. Repeat steps S21 to S24 until all fluctuation segments in the comprehensive mean function are processed.

Images