A large model training method and system based on green distributed computing center
By building a model dependency graph on a green distributed computing power center and introducing a graph partitioning algorithm with a global communication cost matrix, combined with green-aware computing power center modeling and distributed reinforcement learning, the problems of high energy consumption and serious carbon emissions in large model training are solved, and efficient and low-carbon distributed large model training is achieved.
Patent Information
- Application Number
- CN202510947761.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-10
- Publication Date
- 2025-09-19
- Estimated Expiration
- 2045-07-10
AI Technical Summary
Existing large-model training methods are unable to fully utilize the resource scheduling capabilities and energy efficiency advantages in green distributed computing centers, resulting in high energy consumption and serious carbon emissions, and it is difficult to achieve the sustainability and green goals of large-model training.
By constructing a model dependency graph and introducing a graph partitioning algorithm with a global communication cost matrix, combined with a green-aware computing center modeling mechanism and distributed reinforcement learning, we can achieve split training and dynamic scheduling of large models, optimize resource utilization, and reduce energy consumption and carbon emissions.
It significantly improves resource scheduling flexibility and parallel training efficiency, reduces redundant energy consumption and carbon emissions during the computing process, and realizes low-carbon distributed large-model training.
Smart Images
Figure CN120450088B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of large model training, and specifically relates to a large model training method and system based on a green distributed computing center. Background Art
[0002] In recent years, with the rapid development of artificial intelligence (AI), large-scale pre-trained models have achieved significant breakthroughs in fields such as natural language processing, computer vision, and multimodal understanding. These large models often contain billions or even hundreds of billions of parameters, which dramatically increases the demand for computing resources. The training process consumes enormous amounts of energy and computing power. Traditional data centers often face challenges supporting such large-scale model training, including high energy consumption, severe carbon emissions, and inefficient resource scheduling, making them unable to meet the technological requirements of green and sustainable development.
[0003] To address the imbalance between computing power supply and demand and the challenges of energy efficiency, green distributed computing centers have emerged. These centers, based on optimized geographic locations, diversified power mixes (such as wind, photovoltaic, and hydropower), and energy-efficient hardware, offer higher energy efficiency and lower carbon emissions. Furthermore, their distributed architecture enables multiple computing nodes to collaborate in parallel, enabling elastic scalability, remote disaster recovery, and task load balancing, providing stable and reliable computing power. However, most current large-model training methods are still based on centralized architectures, which hinders the full utilization of the resource scheduling capabilities and energy efficiency advantages of green distributed computing centers. Therefore, how to achieve the sustainability and greenness of large-model training by splitting large-model training based on green distributed computing centers while ensuring training accuracy and convergence efficiency while improving energy efficiency and reducing carbon emissions is an urgent issue. Summary of the Invention
[0004] In order to solve the above problems existing in the prior art, the present invention provides a large model training method and system based on a green distributed computing power center.
[0005] The purpose of the present invention can be achieved through the following technical solutions:
[0006] A large-scale model training method based on a green distributed computing center, including:
[0007] S1: Acquire the structural data of a large model to be trained, construct a model dependency graph based on the structural data of the large model to be trained, and split the structural data of the large model to be trained into encapsulated submodules using a model splitting strategy;
[0008] S2: Build a green distributed computing center model and obtain node information, and build a training topology of the encapsulation submodule based on the green distributed computing center model and the node information through a dynamic topology mapping strategy;
[0009] S3: Large model training is achieved by performing green distributed scheduling training on the training topology to train the encapsulated submodule.
[0010] Preferably, the process of constructing the model dependency graph in step S1 is: using the structure data of the large model to be trained as nodes and the data flow dependency relationship between network layers as edges to construct a model directed graph; the edges of the model directed graph carry data transmission volume and timestamp labels, and the node calculation cost is marked to form the model dependency graph.
[0011] Preferably, the splitting process of the model splitting strategy in step S1 is:
[0012] S101: Obtain dependency node pairs of the model dependency graph, and calculate the communication costs of the dependency node pairs;
[0013] S102: traversing the edge set of the model dependency graph, and obtaining a global communication cost matrix by calculating the communication cost;
[0014] S103: splitting the model dependency graph and outputting a model subgraph by a minimum cut algorithm based on the global communication cost matrix;
[0015] S104: Generate an input interface and an output interface, and encapsulate them in combination with the model subgraph to obtain an encapsulated model submodule;
[0016] S105: Optimize the encapsulation model submodule by expanding the equilibrium constraint to output the encapsulation submodule.
[0017] Preferably, the constraint process of expanding the equilibrium constraint in step S105 is:
[0018] S105-1: defining a total load metric of a single submodule of the encapsulated submodule, calculating a maximum submodule load by a maximum value based on the total load metric of the single submodule, and calculating an average submodule load by an average value;
[0019] S105-2: Constructing a load balancing loss function based on the maximum load of the submodule and the average load of the submodule;
[0020] S105-3: Obtain a distributed node resource vector, and define an adaptation ratio based on the distributed node resource vector and the load balancing loss function;
[0021] S105-4: Preset an adaptation constraint. If the adaptation ratio is smaller than the adaptation constraint, the encapsulation model submodule is the encapsulation submodule; if the adaptation ratio is larger than the adaptation constraint, re-segmentation is performed.
[0022] Preferably, the model construction of the green distributed computing power center model in step S2 includes a computing power center, which contains computing resources and storage resources. The computing resources include total computing power resources and occupied computing power resources. The storage resources include total storage resources and occupied storage resources. The computing power center model is fully connected, and the input data is the computing resource amount and storage resource amount of the encapsulated sub-module.
[0023] Preferably, the training process of the training topology in step S2 is:
[0024] An optimization objective function is constructed, and based on the optimization objective function, the encapsulated submodule is mapped to the node of the computing power center through a graph mapping algorithm to output the training topology.
[0025] Preferably, the expression of the optimization objective function is:
[0026] ,
[0027] in, minutes Represents a minimize operation, v i represents the i-th computing power center node, v j represents the jth computing power center node, E Represents the computing power center node, C i,j (N i ,N j ) represents the communication cost after mapping the i-th and j-th submodules, l is the energy consumption penalty weight, used to adjust the green priority, k represents the kth computing power center node, E(N k ) Indicates the unit computing energy consumption of the kth computing power center node, W k is the workload assigned to the kth computing power center node.
[0028] Preferably, the training process of the green distributed scheduling training in step S3 is:
[0029] S301: Using the training topology as a training environment, defining a state space, an action space, and a reward function;
[0030] S302: Based on the training environment, the state space, the action space, and the reward function are trained in a distributed collaborative manner on the encapsulation submodule.
[0031] Preferably, the distributed collaborative training in step S302 includes:
[0032] Obtain the local state, deploy the local intelligent agent on the green distributed computing center node, execute the local strategy based on the local state, and upload the execution data; the central coordinator aggregates the execution data, and updates the global policy network through the policy gradient algorithm to obtain the updated information, outputs the updated information through asynchronous gradient feedback to accumulate the green scheduling strategy, and migrates the accumulated green scheduling strategy to the new node through the migration algorithm to realize large model training.
[0033] A large model training system based on a green distributed computing center includes a model splitting module, a topology construction module, and a distributed training module, including:
[0034] The model splitting module is used to obtain the structural data of the large model to be trained, build a model dependency graph based on the structural data of the large model to be trained, and split the structural data of the large model to be trained into encapsulated sub-modules through the model splitting strategy;
[0035] The topology construction module is used to construct a green distributed computing power center model and obtain node information, and to construct a training topology of the encapsulation submodule based on the green distributed computing power center model and the node information through a dynamic topology mapping strategy;
[0036] The distributed training module is used to implement large model training by performing green distributed scheduling training on the training topology to train the encapsulation submodule.
[0037] The beneficial effects of the present invention are:
[0038] (1) By constructing a model dependency graph and introducing a graph partitioning algorithm based on a global communication cost matrix, we can effectively identify highly coupled data dependencies in large models and utilize communication edge cuts and load balancing constraints to achieve reasonable model splitting. The encapsulation submodule reduces communication overhead while maintaining computational consistency, allowing the model to be flexibly deployed on heterogeneous green computing resources, significantly improving resource scheduling flexibility and parallel training efficiency.
[0039] (2) By introducing a green-aware computing center modeling mechanism in the topology construction stage, combining the node energy consumption model, load status and resource occupancy, a green-oriented optimization objective function is constructed, and the dynamic scheduling of packaged sub-modules to edge / center nodes is realized through the graph mapping algorithm, effectively reducing redundant energy consumption and carbon emissions in the computing process, and realizing low-carbon distributed large-scale model training.
[0040] (3) Through distributed reinforcement learning, a training and scheduling agent network is constructed. Through the mechanism of local perception-central aggregation-asynchronous update, the system load and network status are perceived in real time, and the optimal scheduling strategy is learned autonomously. It can dynamically adapt to heterogeneous computing resources and task fluctuations, improve the overall training convergence speed and scheduling intelligence level of the system, and have green knowledge transfer capabilities to adapt to multi-scenario deployment. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] To facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.
[0042] Figure 1 A flow chart of a large model training method based on a green distributed computing center according to the present invention;
[0043] Figure 2 This is a structural block diagram of the green distributed computing power center model in the present invention. DETAILED DESCRIPTION
[0044] See also Figure 1 , a large model training method based on a green distributed computing center, including:
[0045] S1: Acquire the structural data of a large model to be trained, construct a model dependency graph based on the structural data of the large model to be trained, and split the structural data of the large model to be trained into encapsulated submodules using a model splitting strategy;
[0046] S2: Build a green distributed computing center model and obtain node information, and build a training topology of the encapsulation submodule based on the green distributed computing center model and the node information through a dynamic topology mapping strategy;
[0047] S3: Large model training is achieved by performing green distributed scheduling training on the training topology to train the encapsulated submodule.
[0048] Specifically, the process of constructing the model dependency graph in step S1 is as follows:
[0049] The structure data of the large model to be trained is used as nodes, and the data flow dependency between network layers is used as edges to construct a model directed graph; the edges of the model directed graph carry data transmission volume and timestamp labels, and the node calculation cost is marked to form the model dependency graph.
[0050] Specifically, the structural data of the large model to be trained includes network layers, input and output tensor shapes, parameter scale, execution order and parallel strategy.
[0051] Specifically, the splitting process of the model splitting strategy in step S1 is:
[0052] S101: Obtain dependency node pairs of the model dependency graph, and calculate the communication costs of the dependency node pairs;
[0053] The calculation expression of the communication cost is:
[0054] ,
[0055] in, C i,j is the communication cost, D i,j represents the amount of data transmitted from node i to node j, B link represents the communication bandwidth between deployed nodes, L latency Indicates the delay of the communication link;
[0056] S102: traversing the edge set of the model dependency graph, and obtaining a global communication cost matrix by calculating the communication cost;
[0057] S103: splitting the model dependency graph and outputting a model subgraph by a minimum cut algorithm based on the global communication cost matrix;
[0058] S104: Generate an input interface and an output interface, and encapsulate them in combination with the model subgraph to obtain an encapsulated model submodule;
[0059] S105: Optimize the encapsulation model submodule by expanding the equilibrium constraint to output the encapsulation submodule.
[0060] Specifically, the constraint process of expanding the balance constraint in step S105 is:
[0061] S105-1: defining a total load metric of a single submodule of the encapsulated submodule, calculating a maximum submodule load by a maximum value based on the total load metric of the single submodule, and calculating an average submodule load by an average value;
[0062] S105-2: Constructing a load balancing loss function based on the maximum load of the submodule and the average load of the submodule;
[0063] The expression of the load balancing loss function is:
[0064] ,
[0065] in, L balance represents the load balancing loss function, c 、 d is the constraint importance parameter, L maxis the maximum load of the submodule, is the average load of the submodule, represents the variance of all submodule loads;
[0066] S105-3: Obtain a distributed node resource vector, and define an adaptation ratio based on the distributed node resource vector and the load balancing loss function;
[0067] The definition expression of the adaptation ratio is:
[0068] ,
[0069] in, A i,j is the adaptation ratio, L(P i ) Represents the total load measurement of a single module, α 、 β is an adjustable parameter, r j C For computing resources, r j M For video memory capacity;
[0070] S105-4: Preset an adaptation constraint. If the adaptation ratio is smaller than the adaptation constraint, the encapsulation model submodule is the encapsulation submodule; if the adaptation ratio is larger than the adaptation constraint, re-segmentation is performed.
[0071] Specifically, the distributed node resource vector in step S105 - 3 includes computing resources and video memory capabilities.
[0072] For details, please refer to Figure 2 The model construction of the green distributed computing power center model in step S2 includes a computing power center, which contains computing resources and storage resources. The computing resources include total computing power resources and occupied computing power resources. The storage resources include total storage resources and occupied storage resources. The computing power center model is fully connected, and t represents the encapsulation sub-module, and d represents the input data required by the encapsulation sub-module. The input data includes the amount of computing resources and the amount of storage resources.
[0073] Specifically, the node information in step S2 includes computing power, available memory, network bandwidth, current energy consumption index, power source type, and real-time load status.
[0074] Specifically, the training process of the training topology in step S2 is: constructing an optimization objective function, mapping the encapsulated submodule to the node of the computing power center through a graph mapping algorithm based on the optimization objective function, and outputting the training topology;
[0075] The expression of the optimization objective function is:
[0076] ,
[0077] in, minutes Represents the minimize operation, v i represents the i-th computing power center node, v j represents the jth computing power center node, E Represents the computing power center node, C i,j (N i ,N j ) represents the communication cost after mapping the i-th and j-th submodules, l is the energy consumption penalty weight, used to adjust the green priority, k represents the kth computing power center node, E(N k ) Indicates the unit computing energy consumption of the kth computing power center node, W k is the workload assigned to the kth computing power center node.
[0078] Specifically, the training process of the green distributed scheduling training in step S3 is:
[0079] S301: Using the training topology as a training environment, defining a state space, an action space, and a reward function;
[0080] S302: Based on the training environment, the state space, the action space, and the reward function are trained in a distributed collaborative manner on the encapsulation submodule.
[0081] Specifically, the state space in step S301 is the current load of the computing node, remaining memory, expected power consumption, task queue length, and network delay; the action space is the allocation of encapsulated submodule tasks to nodes, adjustment of communication paths, and adjustment of execution timing;
[0082] The expression of the reward function is:
[0083] ,
[0084] in, Rt represents the reward function, l i is the weight factor, Acc t Indicates the model training accuracy per unit time, Energy t represents the total energy consumption of the system, Delay t represents scheduling delay and communication delay, LoadImb t Indicates the system load imbalance. CO2 t is the carbon emission coefficient.
[0085] Specifically, the distributed collaborative training in step S302 includes: obtaining local status, deploying local intelligent agents on green distributed computing center nodes, executing local strategies based on local status, and uploading execution data; the central coordinator aggregates the execution data, and updates the global policy network through the policy gradient algorithm to obtain update information, outputs the updated information through asynchronous gradient feedback to accumulate green scheduling strategies, and migrates the accumulated green scheduling strategies to new nodes through the migration algorithm to realize large model training.
[0086] In this embodiment, a large model training system based on a green distributed computing center includes a model splitting module, a topology construction module, and a distributed training module, including:
[0087] The model splitting module is used to obtain the structural data of the large model to be trained, build a model dependency graph based on the structural data of the large model to be trained, and split the structural data of the large model to be trained into encapsulated sub-modules through the model splitting strategy;
[0088] The topology construction module is used to construct a green distributed computing power center model and obtain node information, and to construct a training topology of the encapsulation submodule based on the green distributed computing power center model and the node information through a dynamic topology mapping strategy;
[0089] The distributed training module is used to train the encapsulated submodule to achieve large model training by performing green distributed scheduling on the training topology.
[0090] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any person skilled in the art can make some changes or modifications to equivalent embodiments using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.
Claims
1. A large model training method based on a green distributed computing center, characterized in that: The following steps are involved: S1: Acquire the structural data of a large model to be trained, construct a model dependency graph based on the structural data of the large model to be trained, and split the structural data of the large model to be trained into encapsulated submodules using a model splitting strategy; S2: Build a green distributed computing center model and obtain node information, and build a training topology of the encapsulation submodule based on the green distributed computing center model and the node information through a dynamic topology mapping strategy; S3: Implementing large model training by performing green distributed scheduling training on the training topology for the encapsulated submodule; The process of constructing the model dependency graph in step S1 is as follows: the structure data of the large model to be trained is used as nodes, and the data flow dependency relationship between network layers is used as edges to construct a model directed graph; the edges of the model directed graph carry data transmission volume and timestamp labels, and the node calculation cost is marked to form the model dependency graph; The splitting process of the model splitting strategy in step S1 is: S101: Obtain dependency node pairs of the model dependency graph, and calculate the communication costs of the dependency node pairs; S102: traversing the edge set of the model dependency graph, and obtaining a global communication cost matrix by calculating the communication cost; S103: splitting the model dependency graph and outputting a model subgraph by a minimum cut algorithm based on the global communication cost matrix; S104: Generate an input interface and an output interface, and encapsulate them in combination with the model subgraph to obtain an encapsulated model submodule; S105: Optimize the encapsulation model submodule by expanding the equilibrium constraint to output the encapsulation submodule.
2. The large model training method based on the green distributed computing center according to claim 1 is characterized in that: The constraint process of expanding the equilibrium constraint in step S105 is as follows: S105-1: defining a total load metric of a single submodule of the encapsulated submodule, calculating a maximum submodule load by a maximum value based on the total load metric of the single submodule, and calculating an average submodule load by an average value; S105-2: Constructing a load balancing loss function based on the maximum load of the submodule and the average load of the submodule; S105-3: Obtain a distributed node resource vector, and define an adaptation ratio based on the distributed node resource vector and the load balancing loss function; S105-4: Preset an adaptation constraint. If the adaptation ratio is smaller than the adaptation constraint, the encapsulation model submodule is the encapsulation submodule; if the adaptation ratio is larger than the adaptation constraint, re-segmentation is performed.
3. The large model training method based on the green distributed computing center according to claim 1 is characterized in that: The model construction of the green distributed computing power center model in step S2 includes a computing power center, which contains computing resources and storage resources. The computing resources include total computing power resources and occupied computing power resources. The storage resources include total storage resources and occupied storage resources. The computing power center model is fully connected, and the input data is the computing resource amount and storage resource amount of the encapsulated sub-module.
4. The large model training method based on a green distributed computing center according to claim 2 is characterized in that: The training process of the training topology in step S2 is: An optimization objective function is constructed, and based on the optimization objective function, the encapsulated submodule is mapped to the node of the computing power center through a graph mapping algorithm to output the training topology.
5. The large model training method based on a green distributed computing center according to claim 4 is characterized in that: The expression of the optimization objective function is: ， in, min Represents a minimize operation, v i represents the i-th computing power center node, v j represents the jth computing power center node, E Represents the computing power center node, C i,j (N i ,N j ) represents the communication cost after mapping the i-th and j-th submodules, λ is the energy consumption penalty weight, used to adjust the green priority, k represents the kth computing power center node, E(N k ) Indicates the unit computing energy consumption of the kth computing power center node, W k is the workload assigned to the kth computing power center node.
6. The large model training method based on a green distributed computing center according to claim 1 is characterized in that: The training process of the green distributed scheduling training in step S3 is as follows: S301: Using the training topology as a training environment, defining a state space, an action space, and a reward function; S302: Based on the training environment, the state space, the action space, and the reward function are trained in a distributed collaborative manner on the encapsulation submodule.
7. The large model training method based on a green distributed computing center according to claim 6 is characterized in that: The distributed collaborative training in step S302 includes: Obtain the local state, deploy the local intelligent agent on the green distributed computing center node, execute the local strategy based on the local state, and upload the execution data; the central coordinator aggregates the execution data, and updates the global policy network through the policy gradient algorithm to obtain the updated information, outputs the updated information through asynchronous gradient feedback to accumulate the green scheduling strategy, and migrates the accumulated green scheduling strategy to the new node through the migration algorithm to realize large model training.
Citation Information
Patent Citations
Parallel strategy search method for efficient training of artificial intelligence large model
CN120012879A