An adaptive computing power scheduling system for large model training

Abstract

The present invention provides an adaptive computing power scheduling system for large model training, which relates to the field of electrical digital data processing, including a computing power management module, a task analysis module, an adaptive scheduling module and a data interaction module. The computing power management module is responsible for the management and monitoring of the overall computing power resources, the task analysis module is responsible for analyzing the needs and characteristics of training tasks, the adaptive scheduling module performs computing power scheduling according to real-time resource conditions and task requirements, and the data interaction module is used to manage data interaction between different computing power nodes. The system can effectively improve the training efficiency by dispatching training tasks to different nodes and adjusting the allocated resources during the execution of tasks.

Description

Technical Field

[0001] The present invention relates to the field of electrical digital data processing, and in particular to an adaptive computing power scheduling system for large model training. Background Art

[0002] In the field of artificial intelligence and deep learning, large models have become an important tool for solving complex tasks. These models are characterized by their large number of parameters and complex network structures. They usually require massive amounts of data for training and are extremely demanding on computing resources. With the continuous growth of model size and the further increase in computing requirements, traditional static resource allocation methods have become difficult to adapt to the needs of large model training. Currently, an adaptive scheduling system is needed that can dynamically adjust resource allocation according to task requirements to optimize the utilization of computing resources, improve the efficiency of task execution, and ensure the robustness and flexibility of the system.

[0003] Now many computing power scheduling systems have been developed. After a lot of searching and reference, it is found that the existing computing power scheduling systems are such as the system disclosed in the publication number CN117873734B. These system methods generally include pre-allocating the allowed number of GPU cards in the distributed training cluster for each algorithm engineer, and forming GPU card allocation information for storage; when creating a model training task, reading the stored GPU card allocation information and the current GPU card usage in the distributed training cluster, and judging the number of GPU cards allowed to be selected by the algorithm engineer; when selecting the GPU card required for the training task, executing the training job scheduling according to the selected GPU card, and judging whether to reduce the number of GPU cards in the training task according to the number of GPU cards selected by the algorithm engineer, and then starting a new training task. However, the system cannot adjust the allocated resources during the training process, resulting in low utilization of computing power resources, and the overall training efficiency needs to be improved. Summary of the invention

[0004] The purpose of the present invention is to propose an adaptive computing power scheduling system for large model training in response to the existing deficiencies.

[0005] The present invention adopts the following technical solution:

[0006] An adaptive computing power scheduling system for large model training, including a computing power management module, a task analysis module, an adaptive scheduling module and a data interaction module;

[0007] The computing power management module is responsible for the management and monitoring of the overall computing power resources, the task analysis module is responsible for analyzing the requirements and characteristics of the training tasks, the adaptive scheduling module performs computing power scheduling according to the real-time resource status and task requirements, and the data interaction module is used to manage the data interaction between different computing power nodes;

[0008] The computing power management module includes a computing power resource monitoring unit, a computing power pool management unit and an estimated resource unit. The computing power resource monitoring unit is used to detect the load and available status of computing power resources. The computing power pool management unit is used to divide the available computing power into resource pools of different levels. The estimated resource unit predicts the overall computing power resources required based on the large model training features.

[0009] The task analysis module includes a task decomposition unit, a demand analysis unit and a priority evaluation unit. The task decomposition unit is used to decompose the large model training task into multiple subtasks and identify the dependencies of the tasks. The demand analysis unit is used to analyze the requirements of each subtask and generate computing power requirement parameters. The priority evaluation unit is used to assign priorities to subtasks.

[0010] The adaptive scheduling module includes a dynamic resource allocation unit, a scheduling strategy generation unit and an adaptive optimization unit. The dynamic resource allocation unit allocates appropriate computing resources to each subtask based on the scheduling strategy. The scheduling strategy generation unit generates a scheduling strategy based on the priority, demand parameters and resource status of the task. The adaptive optimization unit is used to adjust the computing power allocation during the task execution process.

[0011] The data interaction module includes a data cache unit, a data synchronization unit and a transmission optimization unit. The data cache unit is used to cache commonly used data and intermediate results. The data synchronization unit is used to ensure the consistency and synchronization of data between various nodes. The transmission optimization unit is used to select the optimal data transmission path and method.

[0012] Furthermore, the dynamic resource allocation unit includes a policy management processor, a node selection processor and a task allocation processor. The policy management processor is used to receive the scheduling policy, the node selection processor is used to select the corresponding resource node for the subtask in each scheduling policy, and the task allocation processor is used to organize the task data and send it to the corresponding resource node.

[0013] Furthermore, the process of the node selection processor selecting a resource node includes the following steps:

[0014] S1. Filter out nodes whose resource amount is greater than the actual resource allocation amount in the scheduling strategy, which are called candidate nodes;

[0015] S2. Calculate the data dependency index do of the node to be selected according to the following formula:

[0016] ;

[0017] Among them, ch is the remaining resource amount of the node, cd is the actual resource allocation amount of the subtask, dh is the necessary data amount of the node, and da is the total necessary data amount of the subtask;

[0018] S3. Select the node with the largest data dependency index as the node to allocate resources to execute the subtask.

[0019] Furthermore, the adaptive optimization unit includes an optimization detection processor, a resource change processor and an idle management processor. The optimization detection processor determines whether to optimize based on the use of special resources by subtasks. The resource change processor adjusts and changes the task label of the resource based on the optimization information. The idle management processor is used to manage the use of idle resources by subtasks.

[0020] Furthermore, when the usage rate of the subtask for the special resource exceeds the first threshold for a period of time of T, if there are idle resources at this time, the optimization of adding resources is performed; when the usage rate of the subtask for the special resource is lower than the second threshold for a period of T, the optimization of deleting resources is performed, and T is the optimization judgment time.

[0021] The resource change processor calculates the additional resource amount cd+ according to the following formula:

[0022] ;

[0023] in, As the basic ratio, is the average usage rate within T time, y 1 is the first threshold;

[0024] The resource change processor calculates the amount of deleted resources cd- according to the following formula:

[0025] ;

[0026] Among them, y 2 is the second threshold.

[0027] The beneficial effects achieved by the present invention are:

[0028] This system uses the computing resource monitoring unit to track the status of all computing nodes in the system in real time, dynamically adjust resource allocation, and avoid resource overload or idleness. The computing pool management unit manages resources in a hierarchical manner and dynamically allocates appropriate resource types according to task requirements to achieve refined resource management. The task analysis module identifies the critical path and priority of the task through the demand analysis unit and the priority evaluation unit. During the task execution process, the adaptive scheduling module dynamically optimizes the computing power allocation and scheduling strategy based on real-time feedback to reduce task delays.

[0029] To further understand the features and technical contents of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are only for reference and description and are not intended to limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the overall structural framework of the present invention;

[0031] Figure 2 This is a schematic diagram of the computing power management module of the present invention;

[0032] Figure 3 This is a schematic diagram of the task analysis module of the present invention;

[0033] Figure 4 This is a schematic diagram of the adaptive scheduling module of the present invention;

[0034] Figure 5 This is a schematic diagram of the data interaction module of the present invention;

[0035] Figure 6 This is a comparison diagram of the training and testing effects of the present invention. DETAILED DESCRIPTION

[0036] The following is an explanation of the embodiments of the present invention through specific embodiments. Those skilled in the art can understand the advantages and effects of the present invention from the contents disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments, and the details in this specification can also be modified and changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. In addition, the drawings of the present invention are only simple schematic illustrations and are not depicted according to actual sizes. It is stated in advance. The following embodiments will further explain the relevant technical contents of the present invention in detail, but the disclosed contents are not intended to limit the scope of protection of the present invention.

[0037] Embodiment 1: This embodiment provides an adaptive computing power scheduling system for large model training, combined with Figure 1 , including computing power management module, task analysis module, adaptive scheduling module and data interaction module;

[0038] The computing power management module is responsible for the management and monitoring of the overall computing power resources, the task analysis module is responsible for analyzing the requirements and characteristics of the training tasks, the adaptive scheduling module performs computing power scheduling according to the real-time resource status and task requirements, and the data interaction module is used to manage the data interaction between different computing power nodes;

[0039] The computing power management module includes a computing power resource monitoring unit, a computing power pool management unit and an estimated resource unit. The computing power resource monitoring unit is used to detect the load and available status of computing power resources. The computing power pool management unit is used to divide the available computing power into resource pools of different levels. The estimated resource unit predicts the overall computing power resources required based on the large model training features.

[0040] The task analysis module includes a task decomposition unit, a demand analysis unit and a priority evaluation unit. The task decomposition unit is used to decompose the large model training task into multiple subtasks and identify the dependencies of the tasks. The demand analysis unit is used to analyze the requirements of each subtask and generate computing power requirement parameters. The priority evaluation unit is used to assign priorities to subtasks.

[0041] The adaptive scheduling module includes a dynamic resource allocation unit, a scheduling strategy generation unit and an adaptive optimization unit. The dynamic resource allocation unit allocates appropriate computing resources to each subtask based on the scheduling strategy. The scheduling strategy generation unit generates a scheduling strategy based on the priority, demand parameters and resource status of the task. The adaptive optimization unit is used to adjust the computing power allocation during the task execution process.

[0042] The data interaction module includes a data cache unit, a data synchronization unit and a transmission optimization unit. The data cache unit is used to cache commonly used data and intermediate results. The data synchronization unit is used to ensure the consistency and synchronization of data between various nodes. The transmission optimization unit is used to select the optimal data transmission path and method.

[0043] The dynamic resource allocation unit includes a policy management processor, a node selection processor and a task allocation processor. The policy management processor is used to receive the scheduling policy, the node selection processor is used to select the corresponding resource node for the subtask in each scheduling policy, and the task allocation processor is used to organize the task data and send it to the corresponding resource node.

[0044] The process of selecting a resource node by the node selection processor comprises the following steps:

[0045] S1. Filter out nodes whose resource amount is greater than the actual resource allocation amount in the scheduling strategy, which are called candidate nodes;

[0046] S2. Calculate the data dependency index do of the node to be selected according to the following formula:

[0047] ;

[0048] Among them, ch is the remaining resource amount of the node, cd is the actual resource allocation amount of the subtask, dh is the necessary data amount of the node, and da is the total necessary data amount of the subtask;

[0049] S3. Select the node with the largest data dependency index as the node to allocate resources to execute the subtask.

[0050] The adaptive optimization unit includes an optimization detection processor, a resource change processor and an idle management processor. The optimization detection processor determines whether to optimize based on the usage of special resources by subtasks. The resource change processor adjusts and changes the task label of the resource based on the optimization information. The idle management processor is used to manage the use of idle resources by subtasks.

[0051] When the usage rate of a subtask for a special resource exceeds the first threshold for a period of time T, if there are idle resources at this time, additional resources will be optimized; when the usage rate of a subtask for a special resource is lower than the second threshold for a period of time T, resource reduction will be optimized, where T is the optimization judgment duration;

[0052] The resource change processor calculates the additional resource amount cd+ according to the following formula:

[0053] ;

[0054] in, As the basic ratio, is the average usage rate within T time, y 1 is the first threshold;

[0055] The resource change processor calculates the amount of deleted resources cd- according to the following formula:

[0056] ;

[0057] Among them, y 2 is the second threshold.

[0058] Embodiment 2: This embodiment includes all the contents of Embodiment 1, and provides an adaptive computing power scheduling system for large model training, including a computing power management module, a task analysis module, an adaptive scheduling module and a data interaction module;

[0059] The computing power management module is responsible for the management and monitoring of the overall computing power resources, the task analysis module is responsible for analyzing the requirements and characteristics of the training tasks, the adaptive scheduling module performs computing power scheduling according to the real-time resource status and task requirements, and the data interaction module is used to manage the data interaction between different computing power nodes;

[0060] Combination Figure 2 The computing power management module includes a computing power resource monitoring unit, a computing power pool management unit and an estimated resource unit. The computing power resource monitoring unit is used to detect the load and available status of computing power resources. The computing power pool management unit is used to divide the available computing power into resource pools of different levels. The estimated resource unit predicts the overall computing power resources required based on the training characteristics of the large model.

[0061] Combination Figure 3The task analysis module includes a task decomposition unit, a demand analysis unit and a priority evaluation unit. The task decomposition unit is used to decompose the large model training task into multiple subtasks and identify the dependencies of the tasks. The demand analysis unit is used to analyze the demand of each subtask and generate computing power demand parameters. The priority evaluation unit is used to assign priorities to subtasks.

[0062] Combination Figure 4 The adaptive scheduling module includes a dynamic resource allocation unit, a scheduling strategy generation unit and an adaptive optimization unit. The dynamic resource allocation unit allocates appropriate computing resources to each subtask based on the scheduling strategy. The scheduling strategy generation unit generates a scheduling strategy based on the task's demand parameters and resource status. The adaptive optimization unit is used to adjust the computing power allocation during the task execution process.

[0063] Combination Figure 5 The data interaction module includes a data cache unit, a data synchronization unit and a transmission optimization unit. The data cache unit is used to cache common data and intermediate results. The data synchronization unit is used to ensure the consistency and synchronization of data between various nodes. The transmission optimization unit is used to select the optimal data transmission path and method;

[0064] The resource monitoring unit includes a node status monitoring processor, a performance evaluation processor and a feedback transmission processor. The node status monitoring processor is used to collect resource status data of each node, the performance evaluation processor is used to perform performance evaluation on the collected resource status, and the feedback transmission processor is used to feed back the monitoring information to the computing power pool management unit;

[0065] The computing power pool management unit includes a monitoring receiving processor, a hierarchical statistics processor and an information recording processor, wherein the monitoring receiving processor is used to receive monitoring information of each node, the hierarchical statistics processor divides computing power resources into different resource pools for statistics based on performance evaluation, and the information recording processor is used to record resource information of each resource pool;

[0066] The estimated resource unit includes a historical training register, a comparison estimation processor and a computing power pool allocation processor. The historical training register is used to store case information of actual training. The comparison estimation processor is used to compare the characteristics of the current large model with the case information and determine the resources required for the entire training. The computing power pool allocation processor marks and allocates corresponding resources in the computing power pool based on the estimated resources.

[0067] The task decomposition unit includes a model disassembly processor, a task encoding processor and a task management processor, wherein the model disassembly processor is used to disassemble the training model into multiple subtasks, the task encoding processor performs task encoding based on the dependency relationship of the subtasks, and the task management processor is used to store detailed information of each subtask and manage the dispatch of tasks;

[0068] The demand analysis unit includes a feature extraction processor, a resource mapping processor and a parameter feedback processor, wherein the feature extraction processor is used to extract feature information of the subtask, the resource mapping processor obtains the resource type based on the feature information and rule information mapping, and the parameter feedback processor generates parameters of the corresponding type of resource based on the feature information and feeds back to the task management processor;

[0069] The priority evaluation unit includes a time-sensitive evaluation processor, a resource occupancy evaluation processor and a priority calculation processor, wherein the time-sensitive evaluation processor is used to evaluate the time sensitivity of the subtask, the resource occupancy evaluation processor is used to evaluate the resource occupancy of the subtask, and the priority calculation processor calculates the priority information of the subtask based on the evaluation result;

[0070] The priority calculation processor calculates the priority P of the subtask according to the following formula:

[0071] ;

[0072] Among them, T is the time-sensitive value, R is the resource occupancy value, and w 1 is the time coefficient, w 2 is the resource coefficient;

[0073] The task management processor divides the subtasks into multiple batches based on the task codes, and then sorts the subtasks in the same batch based on the priority, sends the subtasks to the adaptive scheduling module in order, and processes the subtasks of the next batch after processing one batch;

[0074] The dynamic resource allocation unit includes a policy management processor, a node selection processor and a task allocation processor, wherein the policy management processor is used to receive the scheduling policy, the node selection processor is used to select the corresponding resource node for the subtask in each scheduling policy, and the task allocation processor is used to sort the task data and send it to the corresponding resource node;

[0075] The process of selecting a resource node by the node selection processor comprises the following steps:

[0076] S1. Filter out nodes whose resource amount is greater than the actual resource allocation amount in the scheduling strategy, which are called candidate nodes;

[0077] S2. Calculate the data dependency index do of the node to be selected according to the following formula:

[0078] ;

[0079] Among them, ch is the remaining resource amount of the node, cd is the actual resource allocation amount of the subtask, dh is the necessary data amount of the node, and da is the total necessary data amount of the subtask;

[0080] The necessary data of a subtask is the data generated by the previous subtask, which is distributed on different nodes and needs to be synchronized to the node where the subtask is located before the subtask can be executed;

[0081] S3, select the node with the largest data dependency index as the node to allocate resources to execute the subtask;

[0082] The scheduling strategy generation unit includes a state evaluation processor, a resource adjustment processor and a strategy output processor, wherein the state evaluation processor is used to evaluate the current resource state, the resource adjustment processor adjusts the demand parameter based on the resource state evaluation result to obtain the actual resource allocation amount, and the strategy output processor is used to send the strategy information based on the actual resource allocation amount to the dynamic resource allocation unit;

[0083] The resource adjustment processor calculates the actual resource allocation amount cd according to the following formula:

[0084] ;

[0085] Among them, s0 is the standard state, rs is the resource state evaluation value, and pn is the demand parameter;

[0086] The resources in the node have two labels, one is the project label and the other is the task label. The project label indicates that the resource is used for the corresponding training project, and the task label indicates that the resource is assigned to the corresponding subtask. Resources with project labels but not task labels are called idle resources, and resources with both project labels and task labels are called special resources.

[0087] The adaptive optimization unit includes an optimization detection processor, a resource change processor and an idle management processor. The optimization detection processor determines whether to optimize based on the usage of special resources by subtasks, the resource change processor adjusts and changes the task label of the resource based on the optimization information, and the idle management processor is used to manage the use of idle resources by subtasks;

[0088] When the usage rate of a subtask for a special resource exceeds the first threshold for a period of time T, if there are idle resources at this time, additional resources will be optimized; when the usage rate of a subtask for a special resource is lower than the second threshold for a period of time T, resource reduction will be optimized, where T is the optimization judgment duration;

[0089] The resource change processor calculates the additional resource amount cd+ according to the following formula:

[0090] ;

[0091] in, As the basic ratio, is the average usage rate within T time, y 1 is the first threshold;

[0092] The resource change processor calculates the amount of deleted resources cd- according to the following formula:

[0093] ;

[0094] Among them, y 2 is the second threshold.

[0095] When the utilization rate of the special resource by the subtask temporarily reaches 100%, the idle management processor calls the idle resource to the subtask for use, and the idle resource will be released immediately after the subtask uses up the idle resource;

[0096] The data synchronization unit includes a synchronization retrieval processor and a data sending processor, wherein the synchronization retrieval processor is used to retrieve nodes whose data needs to be synchronized, and the data sending processor is used to sort out the data that needs to be synchronized and send it to the corresponding node;

[0097] Part of the code of this system is described as follows:

[0098] # Computing power management module

[0099] class ComputeManagementModule:

[0100] def __init__(self):

[0101] self.resource_monitor = ResourceMonitor()

[0102] self.compute_pool_manager = ComputePoolManager()

[0103] self.resource_predictor = ResourcePredictor()

[0104] def get_resource_state(self):

[0105] # Return resource status

[0106] return self.resource_monitor.monitor_resources()

[0107] # Computing resource monitoring unit

[0108] class ResourceMonitor:

[0109] def monitor_resources(self):

[0110] # Check the load and availability of computing resources

[0111] return {"CPU": 70, "GPU": 80, "Memory": 60}

[0112] # Hash Pool Management Unit

[0113] class ComputePoolManager:

[0114] def manage_pools(self):

[0115] # Divide resources into different levels of resource pools

[0116] print("Managing compute pools...")

[0117] # Estimated resource units

[0118] class ResourcePredictor:

[0119] def predict_resources(self, task_characteristics):

[0120] # Predict overall computing power requirements

[0121] return {"required_CPU": 100, "required_GPU": 200}

[0122] # Task Analysis Module

[0123] class TaskAnalysisModule:

[0124] def __init__(self):

[0125] self.task_splitter = TaskSplitter()

[0126] self.demand_analyzer = DemandAnalyzer()

[0127] self.priority_evaluator = PriorityEvaluator()

[0128] def analyze_tasks(self):

[0129] # Analyze task requirements and characteristics

[0130] tasks = self.task_splitter.split_task("Main Training Task")

[0131] for task in tasks:

[0132] task["demands"] = self.demand_analyzer.analyze_demands(task)

[0133] task["priority"] = self.priority_evaluator.evaluate_priority(task)

[0134] return tasks

[0135] # Task decomposition unit

[0136] class TaskSplitter:

[0137] def split_task(self, task):

[0138] # Break down the task into subtasks

[0139] return [{"name": "Subtask1"}, {"name": "Subtask2"}]

[0140] # Requirements Analysis Unit

[0141] class DemandAnalyzer:

[0142] def analyze_demands(self, task):

[0143] # Generate computing power requirement parameters

[0144] return {"CPU": 50, "GPU": 80}

[0145] # Priority Assessment Units

[0146] class PriorityEvaluator:

[0147] def evaluate_priority(self, task):

[0148] # Calculate task priority

[0149] return 1

[0150] # Adaptive Scheduling Module

[0151] class AdaptiveSchedulingModule:

[0152] def __init__(self):

[0153] self.dynamic_allocator = DynamicAllocator()

[0154] self.strategy_generator = StrategyGenerator()

[0155] self.optimizer = AdaptiveOptimizer()

[0156] def generate_schedule(self, tasks, resource_state):

[0157] # Generate scheduling strategies based on task and resource status

[0158] return self.strategy_generator.generate_strategy(tasks,resource_state)

[0159] def allocate_resources(self, schedule):

[0160] # Dynamically allocate resources

[0161] self.dynamic_allocator.allocate(schedule)

[0162] # Dynamic resource allocation unit

[0163] class DynamicAllocator:

[0164] def allocate(self, schedule):

[0165] # Allocate computing resources

[0166] print(f"Allocating resources based on schedule: {schedule}")

[0167] # Scheduling strategy generation unit

[0168] class StrategyGenerator:

[0169] def generate_strategy(self, tasks, resource_state):

[0170] # Generate scheduling strategy

[0171] return {"task_allocation": tasks, "resource_state": resource_state}

[0172] # Adaptive Optimization Unit

[0173] class AdaptiveOptimizer:

[0174] def optimize(self, task_state, resource_state):

[0175] # Dynamically adjust computing power allocation

[0176] print("Optimizing resource allocation...")

[0177] # Data interaction module

[0178] class DataInteractionModule:

[0179] def __init__(self):

[0180] self.data_cache = DataCache()

[0181] self.data_synchronizer = DataSynchronizer()

[0182] self.transfer_optimizer = TransferOptimizer().

[0183] There are 10 training samples. The training process is performed by this system and the common system respectively. The training time is obtained by testing. Figure 6 Comparison chart shown.

[0184] The contents disclosed above are only preferred feasible embodiments of the present invention, and do not limit the protection scope of the present invention. Therefore, all equivalent technical changes made using the contents of the present invention specification and drawings are included in the protection scope of the present invention. In addition, the elements therein can be updated as technology develops.

Claims

1. An adaptive computing power scheduling system for large model training, characterized in that: It includes computing power management module, task analysis module, adaptive scheduling module and data interaction module; The computing power management module is responsible for the management and monitoring of the overall computing power resources, the task analysis module is responsible for analyzing the requirements and characteristics of the training tasks, the adaptive scheduling module performs computing power scheduling according to the real-time resource status and task requirements, and the data interaction module is used to manage the data interaction between different computing power nodes; The computing power management module includes a computing power resource monitoring unit, a computing power pool management unit and an estimated resource unit. The computing power resource monitoring unit is used to detect the load and available status of computing power resources. The computing power pool management unit is used to divide the available computing power into resource pools of different levels. The estimated resource unit predicts the overall computing power resources required based on the large model training features. The task analysis module includes a task decomposition unit, a demand analysis unit and a priority evaluation unit. The task decomposition unit is used to decompose the large model training task into multiple subtasks and identify the dependencies of the tasks. The demand analysis unit is used to analyze the requirements of each subtask and generate computing power requirement parameters. The priority evaluation unit is used to assign priorities to subtasks. The adaptive scheduling module includes a dynamic resource allocation unit, a scheduling strategy generation unit and an adaptive optimization unit. The dynamic resource allocation unit allocates appropriate computing resources to each subtask based on the scheduling strategy. The scheduling strategy generation unit generates a scheduling strategy based on the priority, demand parameters and resource status of the task. The adaptive optimization unit is used to adjust the computing power allocation during the task execution process. The data interaction module includes a data cache unit, a data synchronization unit and a transmission optimization unit. The data cache unit is used to cache common data and intermediate results. The data synchronization unit is used to ensure the consistency and synchronization of data between various nodes. The transmission optimization unit is used to select the optimal data transmission path and method. The dynamic resource allocation unit includes a policy management processor, a node selection processor and a task allocation processor, wherein the policy management processor is used to receive the scheduling policy, the node selection processor is used to select the corresponding resource node for the subtask in each scheduling policy, and the task allocation processor is used to sort the task data and send it to the corresponding resource node; The process of selecting a resource node by the node selection processor comprises the following steps: S1. Filter out nodes whose resource amount is greater than the actual resource allocation amount in the scheduling strategy, which are called candidate nodes; S2. Calculate the data dependency index do of the node to be selected according to the following formula: ； Among them, ch is the remaining resource amount of the node, cd is the actual resource allocation amount of the subtask, dh is the necessary data amount of the node, and da is the total necessary data amount of the subtask; The necessary data of a subtask is the data generated by the previous subtask, which is distributed on different nodes and needs to be synchronized to the node where the subtask is located before the subtask can be executed; S3. Select the node with the largest data dependency index as the node to allocate resources to execute the subtask.

2. An adaptive computing power scheduling system for large model training as claimed in claim 1, characterized in that: The adaptive optimization unit includes an optimization detection processor, a resource change processor and an idle management processor. The optimization detection processor determines whether to optimize based on the usage of special resources by subtasks. The resource change processor adjusts and changes the task label of the resource based on the optimization information. The idle management processor is used to manage the use of idle resources by subtasks.

3. An adaptive computing power scheduling system for large model training as claimed in claim 2, characterized in that: When the usage rate of a subtask for a special resource exceeds the first threshold for a period of time T, if there are idle resources at this time, additional resources will be optimized; when the usage rate of a subtask for a special resource is lower than the second threshold for a period of time T, resource reduction will be optimized, where T is the optimization judgment duration; The resource change processor calculates the additional resource amount cd+ according to the following formula: ； in, As the basic ratio, is the average usage rate within T time, y1 is the first threshold; The resource change processor calculates the amount of deleted resources cd- according to the following formula: ； Wherein, y2 is the second threshold.

Images