Large model distributed training and system based on all-in-one machine hybrid cloud

Abstract

The invention discloses a large model distributed training method and system based on all-in-one machine hybrid cloud, and relates to the technical field of resource management. A hybrid cloud training architecture and secure communication are established between a local all-in-one machine cluster and a public cloud elastic cluster; dividing training data according to sensitivity, and realizing local storage of core sensitive data and cloud side storage of non-sensitive data; dividing the training task into a core task and an extension task, and scheduling the core task and the extension task to a local training node and a cloud side training node for execution; monitoring local resource load and task progress, and elastically applying and releasing cloud side training nodes according to training stages; and limiting cross-domain transmission as necessary information, updating model parameters through gradient aggregation, and synchronously entering the next iteration. Through the technical scheme of the invention, on the premise of ensuring that sensitive data is not out of a domain, on-demand expansion and recovery of cloud side computing power are realized, cross-domain communication overhead is reduced, distributed training throughput and stability are improved, a training period is shortened, and comprehensive cost is reduced.

Description

Technical Field

[0001] This invention relates to the field of resource management technology, and in particular to a large-scale model distributed training method and a large-scale model distributed training system based on an all-in-one machine and a hybrid cloud. Background Technology

[0002] As the scale of AI model parameters increases from tens of billions to hundreds of billions or even larger, large-scale model training places higher demands on computing resources, storage resources, and network transmission capabilities. To improve training throughput and shorten training cycles, the industry typically employs distributed training frameworks to organize data parallelism, model parallelism, or pipeline parallelism, enabling multiple nodes to collaboratively complete forward computation, backward gradient calculation, and parameter updates. However, in practical applications, training data often contains both core business data and user privacy data, and the training process exhibits distinct phased characteristics. The requirements for computing power and data access locations differ significantly at different stages, making it difficult for existing training deployment solutions to balance security compliance, computing power expansion, resource utilization, and network costs.

[0003] One approach is pure local all-in-one machine cluster training. This approach relies on a locally deployed AI all-in-one machine cluster to complete training data storage and computation locally, meeting compliance requirements for sensitive data not leaving the domain, and utilizing local high-speed interconnects to achieve low-latency parameter synchronization. However, the computing power and storage expansion capabilities of this approach are limited by existing hardware configurations. When facing rapid growth in training scale, additional procurement and deployment are often required, resulting in long expansion cycles and high one-time investments. At the same time, the computing power requirements for large model training are lower during data preprocessing, initialization, validation, and fine-tuning stages than during mid-term large-scale iteration stages. Local clusters are prone to resource idleness during off-peak periods, resulting in low overall utilization. In addition, local clusters need to continuously undertake hardware maintenance, system upgrades, and troubleshooting, and the complexity of maintenance and labor costs increase simultaneously with the expansion of scale.

[0004] Another approach is pure public cloud training. This approach relies on cloud GPU instances and object storage to provide on-demand computing power through elastic scaling, reducing upfront hardware investment and shortening resource acquisition cycles. However, in large-scale model training, uploading training data to the cloud introduces data privacy and security compliance risks, especially for industries constrained by data localization and compliance, where sensitive data leaving the domain faces significant obstacles. During training, parameters and gradients need frequent synchronization across nodes. The public cloud network environment is affected by fluctuations in public network bandwidth and cross-regional transmission, easily leading to higher communication latency and reduced training efficiency. At the same time, large-scale data uploads and downloads and synchronization between nodes also incur significant bandwidth costs. Furthermore, the shared nature of cloud resources may lead to instance preemption, offline status, or fluctuations, making it difficult to guarantee the stability and continuity of training tasks, further increasing training costs and cycle uncertainty.

[0005] Against this backdrop, the industry has begun to explore hybrid training models that combine on-premises and public cloud resources, hoping to leverage the elastic computing power of the cloud while ensuring the security of sensitive data. However, existing hybrid models often face the following key contradictions: On the one hand, if the entire training data is copied to the cloud in exchange for elastic computing power, there are still risks of sensitive data leaving the domain and compliance issues; on the other hand, if only part of the computation is placed on the cloud without layering and organizing tasks and data, a large amount of data transmission and parameter synchronization overhead will still be generated in cross-domain communication, causing training efficiency to be constrained by network bottlenecks. Furthermore, the cloud resource scaling lacks a scheduling mechanism that matches the training stage and resource load, making it difficult to achieve a stable balance between cost and efficiency. Summary of the Invention

[0006] To address the aforementioned issues, this invention provides a large-scale distributed training method and system based on a hybrid cloud architecture using an all-in-one machine. By constructing a hybrid cloud training architecture between a local all-in-one machine cluster and a public cloud elastic cluster, training data is categorized by sensitivity, with core sensitive data stored locally and non-sensitive data stored on the cloud side. Training tasks are split into core tasks and extended tasks, executed on local and cloud-side training nodes respectively. While ensuring compliance by preventing sensitive data from leaving its domain, cloud-side computing power is elastically requested and released according to training stages, thereby improving resource utilization and reducing peak computing power input. By restricting cross-domain transmission as necessary information for collaborative training and employing a distributed collaborative mechanism to aggregate and update model parameters using local and cloud-side gradients, cross-domain communication is controlled, training synchronization overhead is reduced, and the throughput and stability of large-scale distributed training are improved. Furthermore, while ensuring secure isolation, the training cycle is shortened and overall costs are optimized.

[0007] To achieve the above objectives, this invention provides a large-scale model distributed training method based on an integrated machine and hybrid cloud, comprising: A hybrid cloud training architecture is established between a local all-in-one machine cluster and a public cloud elastic cluster. The local all-in-one machine cluster includes at least a training control node, a local training node, and local storage, while the public cloud elastic cluster includes at least a cloud-side training node and cloud-side storage. The local all-in-one machine cluster and the public cloud elastic cluster are interconnected through a secure communication channel. The training data is divided into core sensitive data and non-sensitive data according to sensitivity, and the core sensitive data is stored in the local storage, while the non-sensitive data is stored in the cloud storage. The large model training task is divided into core tasks and extended tasks. The core tasks are scheduled to be executed on the local training nodes of the local integrated machine cluster, and the extended tasks are scheduled to be executed on the cloud-side training nodes of the public cloud elastic cluster. The core tasks are executed based on the core sensitive data, and the extended tasks are executed based on the non-sensitive data. During training, the resource load and training task progress of the local integrated machine cluster are monitored, and the cloud-side training nodes are applied for or released according to the resource load and training task progress, so that the computing power of the public cloud elastic cluster can scale as the training stage changes. The transmission of necessary information for collaborative training is restricted between the local all-in-one cluster and the public cloud elastic cluster. The necessary information includes at least model parameter data, gradient data, and task status information. A distributed training collaboration mechanism is used to update parameters, wherein gradient data generated by the local training node and the cloud-side training node are aggregated to update model parameters, and the updated model parameters are used in the next training iteration.

[0008] In the above technical solution, preferably, the local training node is composed of a preset number of AI all-in-one machines, and each AI all-in-one machine is equipped with multiple GPUs, local SSD storage and interconnection network; The local storage includes distributed storage nodes, which are used to store the core sensitive data, model backbone parameters and intermediate training results. The training control node is used to perform task splitting and scheduling, resource management, and collaborative communication with the public cloud elastic cluster.

[0009] In the above technical solution, preferably, the cloud-side training node consists of a preset number of cloud GPU instances, and the cloud-side storage is cloud object storage; the secure communication channel is established based on a cloud private network to realize encrypted interconnection between the local all-in-one machine cluster and the public cloud elastic cluster.

[0010] In the above technical solution, preferably, the core tasks include sensitive data preprocessing, model backbone network training, and core parameter updating; The extended tasks include non-sensitive data augmentation, model branch network training, and gradient calculation assistance. The core sensitive data includes user privacy data and core business data, while the non-sensitive data includes public datasets and data generated through data augmentation.

[0011] In the above technical solution, preferably, the resource load includes at least GPU utilization, storage occupancy, and network bandwidth; The specific process for applying for or releasing cloud-side training nodes includes: When the GPU utilization reaches the first threshold or the training enters a large-scale iteration phase, apply for additional cloud-side training nodes to expand computing power. When the GPU utilization rate is lower than the second threshold or the training enters the data preprocessing stage or the model fine-tuning stage, the idle cloud-side training nodes are released. The first threshold is 85%, and the second threshold is 30%.

[0012] In the above technical solution, preferably, the necessary information undergoes transmission optimization processing before cross-domain transmission. The transmission optimization processing includes performing quantization compression and sparsification processing on the model parameter data and the gradient data, and adjusting the transmission rate of the necessary information based on public network bandwidth fluctuations. The distributed training collaboration mechanism includes parallel strategy fusion, wherein the local all-in-one machine cluster uses a hybrid parallel strategy of model parallelism and data parallelism to execute the core task, and the public cloud elastic cluster uses a data parallel strategy to execute the extended task.

[0013] This invention also proposes a large-scale model distributed training system based on an all-in-one machine and hybrid cloud, which applies the large-scale model distributed training method based on an all-in-one machine and hybrid cloud disclosed in any of the above technical solutions, including: a local all-in-one machine cluster, a public cloud elastic cluster, a secure communication channel, a data isolation module, a task hierarchical scheduling module, an elastic scaling module, a transmission control module, and a gradient aggregation module. The local integrated machine cluster includes local training nodes and local storage, the public cloud elastic cluster includes cloud-side training nodes and cloud-side storage, and the secure communication channel is used to realize the interconnection between the local integrated machine cluster and the public cloud elastic cluster. The data isolation module is used to divide the training data into core sensitive data and non-sensitive data according to their sensitivity, and to control the core sensitive data to be stored in the local storage and the non-sensitive data to be stored in the cloud storage. The task hierarchical scheduling module is used to split the large model training task into core tasks and extended tasks, and schedule the core tasks to be executed on the local training node and the extended tasks to be executed on the cloud-side training node. The elastic scaling module is used to apply for or release the cloud-side training nodes based on the resource load and training task progress of the local all-in-one machine cluster. The transmission control module is used to restrict the transmission of necessary collaborative training information across domains. The necessary information includes at least model parameter data, gradient data, and task status information. The gradient aggregation module is used to aggregate the gradient data generated by the local training node and the cloud-side training node to update the model parameters, and synchronize the updated model parameters to the local training node and the cloud-side training node.

[0014] In the above technical solution, preferably, the large model distributed training system based on integrated machine hybrid cloud also includes a training control node, which includes a resource monitoring unit and an instance lifecycle management unit; The resource monitoring unit is used to obtain GPU utilization, storage occupancy and network bandwidth. The instance lifecycle management unit is used to apply for or release the cloud-side elastic computing instance corresponding to the cloud-side training node based on the monitoring results of the resource monitoring unit and the training task progress.

[0015] In the above technical solution, preferably, the training control node includes a topology-aware scheduling unit, which is used for: Obtain training task parameters, which include at least the number of parameters and the batch size; Obtain computing node information, which includes at least computing power type, computing performance, topology location, and network bandwidth, wherein the topology location includes at least rack identifier and region identifier; The task-computing power matching degree is determined based on the training task parameters and the computing performance; the topology weight is determined based on the rack identifier and the region identifier; the bandwidth weight is determined based on the network bandwidth; and the node score is generated by combining the task-computing power matching degree, the topology weight, and the bandwidth weight. The node group with the highest score is selected to perform the hierarchical scheduling of the task.

[0016] In the above technical solution, preferably, the training control node includes a fault-tolerant unit, which is used to monitor the node status of the local training node and the cloud-side training node. When a failure is detected in the cloud GPU instance corresponding to the cloud training node, the corresponding training task is migrated to other normal cloud training nodes or local training nodes. When a failure is detected in a local training node, the core task will be migrated to other local training nodes in the local integrated machine cluster.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) By constructing a hybrid cloud training architecture between the local all-in-one machine cluster and the public cloud elastic cluster, and by classifying the training data according to sensitivity, core sensitive data is stored locally and non-sensitive data is stored on the cloud side. With the hierarchical scheduling of core tasks and extended tasks, the training is executed on the local training node and the cloud side training node respectively, so that sensitive data can participate in training with cloud computing power while not leaving the domain, thereby reducing data compliance and security risks and improving training scalability.

[0018] (2) By continuously monitoring the local resource load and task progress during the training process, the cloud training nodes are applied for and released elastically according to the training stage, so that the peak stage such as large-scale iteration can obtain incremental computing power, and the idle resources can be released during the non-peak stage such as preprocessing or fine-tuning, thereby improving the computing power utilization rate and reducing peak hardware investment and cloud resource costs.

[0019] (3) By restricting cross-domain transmission as necessary information for collaborative training, and by aggregating and updating model parameters based on gradients generated locally and in the cloud, and synchronously entering the next iteration, the cross-domain communication volume and synchronization overhead are reduced, the throughput and convergence stability of distributed training are improved, thereby shortening the training cycle and improving training continuity. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a distributed training method for large models based on an integrated machine and hybrid cloud, as disclosed in one embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] The present invention will now be described in further detail with reference to the accompanying drawings: like Figure 1 As shown, a large-scale model distributed training method based on an integrated machine and hybrid cloud, according to the present invention, includes: A hybrid cloud training architecture is established between the local all-in-one machine cluster and the public cloud elastic cluster. The local all-in-one machine cluster includes at least a training control node, a local training node, and local storage, while the public cloud elastic cluster includes at least a cloud-side training node and cloud-side storage. The interconnection between the local all-in-one machine cluster and the public cloud elastic cluster is achieved through a secure communication channel. The training data is divided into core sensitive data and non-sensitive data according to their sensitivity. The core sensitive data is stored in local storage, and the non-sensitive data is stored in cloud storage. The large model training task is divided into core tasks and extended tasks. The core tasks are scheduled to be executed on the local training nodes of the local integrated machine cluster, and the extended tasks are scheduled to be executed on the cloud-side training nodes of the public cloud elastic cluster. Among them, the core tasks are executed based on core sensitive data, and the extended tasks are executed based on non-sensitive data. During training, the resource load and training task progress of the local integrated machine cluster are monitored, and cloud-side training nodes are applied for or released according to the resource load and training task progress, so that the computing power of the public cloud elastic cluster can scale up and down with the training stage. Restrict the transmission of necessary information for collaborative training between the local all-in-one cluster and the public cloud elastic cluster. The necessary information includes at least model parameter data, gradient data, and task status information. A distributed training collaboration mechanism is adopted to complete the parameter update. The gradient data generated by the local training node and the cloud-side training node are aggregated to update the model parameters, and the updated model parameters are used for the next training iteration.

[0023] Specifically, the training control node establishes a hybrid cloud training architecture between the local all-in-one cluster and the public cloud elastic cluster, and achieves interconnection between the two sides through a secure communication channel.

[0024] Before training begins, the training data is classified according to sensitivity. Core sensitive data is stored locally with access control implemented locally, while non-sensitive data is stored in cloud storage for easy access by cloud-based training nodes. The training task is then divided into core tasks and extended tasks: core tasks are executed on local training nodes and perform backbone training and critical updates solely based on core sensitive data; extended tasks are executed on cloud-based training nodes and handle incremental computing power solely based on non-sensitive data.

[0025] During training, the training control node continuously collects local resource load and training progress, and applies for or releases cloud-side training nodes according to the training stage to match the computing power requirements of peak iteration and off-peak stages. When cross-domain collaboration is performed, the transmitted content is limited to the necessary information for collaborative training, including at least model parameters, gradients, and task status. The gradients produced by the local training node and the cloud-side training node are aggregated to update the parameters, and the updated model parameters are synchronized to both training nodes to enter the next round of iteration.

[0026] This implementation combines the security of a local integrated machine cluster with the elastic computing power of a public cloud, achieving dynamic collaboration between the two types of resources through a hierarchical scheduling mechanism, thus overcoming the limitations of purely local or purely cloud solutions. Privacy and security are ensured through categorized data storage, while core sensitive data is kept in a local closed-loop processing environment, reducing compliance and leakage risks associated with data leaving the domain. Based on the characteristics of training task stages and resource load status, cloud-side computing power is scaled up and down in stages, enabling automatic application and release of cloud resources, improving overall resource utilization and reducing peak workload. Cross-domain transmission of only necessary information reduces communication volume and latency, improves distributed training throughput and training continuity, and shortens the overall training cycle.

[0027] In the above implementation, preferably, the local training node consists of N AI all-in-one machines. Each AI all-in-one machine is equipped with multiple GPUs, local SSD storage and interconnection network, and forms a low-latency, high-bandwidth cluster interconnection through a high-speed interconnection network (such as InfiniBand).

[0028] Local storage includes distributed storage nodes, which are used to store core sensitive data, model backbone parameters, and intermediate training results, so that data reading and writing and parameter access for core tasks can be completed within the local domain.

[0029] The training control node collaborates with local training nodes: The training control node is responsible for splitting core tasks to specific local training nodes, maintaining training progress and resource views, and communicating with the public cloud side to trigger cloud-side resource scaling. It is used to perform task splitting and scheduling, resource management, and collaborative communication with the public cloud elastic cluster.

[0030] In this implementation, the local side uses the computing power of the AI ​​all-in-one machine's GPU and high-speed interconnection to support low-latency synchronization and high-throughput computing for backbone training. Distributed storage ensures that sensitive data and key intermediate results do not leave the domain. The unified orchestration of training control nodes reduces the complexity of local operation and maintenance, and improves the controllability of task scheduling and the stability of execution.

[0031] In the above implementation, preferably, the public cloud elastic cluster consists of M cloud GPU instances forming cloud-side training nodes, and the cloud-side storage is cloud object storage, carrying non-sensitive training data and auxiliary parameters to meet the elastic data supply requirements of extended tasks. A secure communication channel is established based on a cloud private network VPN to achieve encrypted interconnection between the local integrated machine cluster and the public cloud elastic cluster, used for cross-domain transmission of necessary collaborative training information and carrying task status feedback and scheduling control commands.

[0032] In this implementation, cloud GPU instances are dynamically added or removed on demand, shortening the computing power acquisition cycle and increasing the expansion limit; cloud object storage provides highly available data supply for non-sensitive data; and encrypted interconnection of private networks reduces the security risks of cross-domain transmission and improves link availability.

[0033] In the above implementation, preferably, the core tasks include sensitive data preprocessing, model backbone network training, and core parameter updating, so that the model can complete the key training loop around the core sensitive data on the local training node.

[0034] The extended tasks include non-sensitive data augmentation, model branch network training, and gradient calculation assistance, enabling cloud-side training nodes to perform scalable parallel computation around non-sensitive data.

[0035] Core sensitive data includes user privacy data and core business data, serving as input for core tasks and sources of key intermediate results. Non-sensitive data includes public datasets and data augmentation-generated data, serving as input for extended tasks and used to improve training coverage and computational efficiency. Both types of tasks achieve unified model parameter convergence through gradient collaboration during the parameter update phase.

[0036] In this implementation, core tasks are bound to sensitive data, enabling training paths where sensitive data does not leave the domain; extended tasks utilize non-sensitive data to carry out scalable computing, relieving pressure on the local cluster; and layered collaboration improves training scale capacity and training efficiency while ensuring security isolation.

[0037] In the above implementation, preferably, the training control node uses GPU utilization, storage occupancy, and network bandwidth as resource load indicators, and identifies the training stage in combination with the training progress.

[0038] The specific process for applying for or releasing cloud-based training nodes includes: When the GPU utilization reaches the first threshold or the training enters a large-scale iteration phase, the training control node triggers an application for a new cloud-side training node to increase the parallelism of the extended task and shorten the iteration time. When GPU utilization falls below the second threshold, or when training enters the data preprocessing or model fine-tuning stage, the training control node releases idle cloud-side training nodes to match cloud-side resource usage with the phased computing power demand.

[0039] The thresholds are set to a first threshold of 85% and a second threshold of 30% to form stable automatic scaling trigger conditions and avoid frequent jitter.

[0040] In this implementation, incremental computing power on the cloud side is quickly obtained during peak periods, and the expansion response is changed from the hardware procurement cycle to the instance application cycle; idle resources are automatically reclaimed during off-peak periods, reducing cloud resource costs; overall resource utilization is improved and the total training cost is reduced, while maintaining the continuity of the training process.

[0041] In the above implementation, preferably, the necessary information is subjected to transmission optimization processing before cross-domain transmission: the model parameters and gradients are quantized and compressed to reduce the numerical bit width occupation, the gradients are sparsified to reduce the number of effective transmission elements, and the transmission rate of the necessary information is dynamically adjusted based on the fluctuation of public network bandwidth, so as to prioritize the accessibility of training key synchronization information when bandwidth is limited.

[0042] In terms of parallel strategies, the local all-in-one cluster adopts a hybrid parallel strategy of model parallelism and data parallelism to improve the throughput and memory utilization of the backbone training, while the cloud elastic cluster adopts a data parallel strategy to improve the horizontal scaling capability of extended tasks; the gradients produced by both sides are merged by a gradient aggregator and used for unified parameter updates.

[0043] Specifically, in one embodiment, the local side hybrid parallelism is implemented using DualPipe pipeline parallelism to improve GPU utilization and throughput. The training engine divides the model into num_stages stages by layer and binds each stage to the corresponding device; at the same time, the global batch is divided into num_microbatches to form a pipelined input sequence.

[0044] During training, the process is advanced by time slots, with the number of time slots being the combined length of num_microbatches and num_stages, causing the pipeline to go through three stages: filling, steady state, and emptying.

[0045] Within each time slice, forward scheduling is performed first: for each stage_idx, the index mb_idx of the micro-batch being processed is calculated. When mb_idx falls within the valid range, forward calculation for this stage is performed on the stage device. The first stage directly uses the micro-batch data as input, while the remaining stages use the forward output of the corresponding micro-batch from the previous stage as input, and cache the forward output of each stage for subsequent reverse use.

[0046] Then, reverse scheduling is performed: the stage is traversed in reverse order, the corresponding micro-batch index mb_idx is calculated, and when the forward propagation of the micro-batch is completed and the output of the final stage is valid, the final stage calculates the loss based on the predicted output and label and performs reverse propagation to obtain the gradient; the gradients returned by the previous stage are used as inputs for the remaining stages, and the backpropagation is performed stage by stage to obtain the gradients of each stage.

[0047] After all time slices are completed, cross-node aggregation (corresponding to distributed gradient synchronization) is performed on the resulting gradient set, and the aggregated gradients are used for parameter updates.

[0048] In this implementation, quantization compression and sparsity reduce cross-domain communication volume, decreasing the amount of transmitted data by 60%-80% and mitigating the impact of network latency on training throughput. Bandwidth adaptation reduces synchronization blockage caused by congestion. Hybrid parallelism and cloud-side data parallelism work together to improve overall training throughput and convergence stability, enhancing training efficiency and approaching the synchronous performance of local low-latency training. Micro-batch partitioning and forward / backward interleaving reduce pipeline cavitation and improve equipment utilization at each stage. Staged caching and unified gradient aggregation ensure consistency between pipeline parallelism and distributed synchronization, improving training throughput and shortening iteration time.

[0049] This invention also proposes a large-scale model distributed training system based on an all-in-one machine and hybrid cloud, which applies the large-scale model distributed training method based on an all-in-one machine and hybrid cloud disclosed in any of the above embodiments, including: a local all-in-one machine cluster, a public cloud elastic cluster, a secure communication channel, a data isolation module, a task hierarchical scheduling module, an elastic scaling module, a transmission control module, and a gradient aggregation module. The local all-in-one cluster includes local training nodes and local storage, while the public cloud elastic cluster includes cloud-side training nodes and cloud-side storage. A secure communication channel is used to enable interconnection between the local all-in-one cluster and the public cloud elastic cluster. The data isolation module is used to divide training data into core sensitive data and non-sensitive data according to their sensitivity, and to control the storage of core sensitive data in local storage and non-sensitive data in cloud storage. The task hierarchical scheduling module is used to split large model training tasks into core tasks and extended tasks, and schedule core tasks to local training nodes for execution and extended tasks to cloud-side training nodes for execution. The elastic scaling module is used to apply for or release cloud-side training nodes based on the resource load and training task progress of the local all-in-one machine cluster. The transmission control module is used to restrict the transmission of necessary collaborative training information across domains. The necessary information includes at least model parameter data, gradient data, and task state information. The gradient aggregation module is used to aggregate gradient data generated by local training nodes and cloud-side training nodes to update model parameters, and synchronize the updated model parameters to local training nodes and cloud-side training nodes.

[0050] Specifically, the data isolation module classifies training data according to sensitivity and controls core sensitive data to only enter local storage, while non-sensitive data enters cloud storage; the task hierarchical scheduling module splits training tasks into core tasks and extended tasks, and distributes them to local training nodes and cloud training nodes respectively; the elastic scaling module drives the addition or removal of cloud training nodes according to local resource load and training progress; the transmission control module restricts cross-domain interaction to be necessary information for collaborative training and maintains the consistency of task status; the gradient aggregation module aggregates gradients from both sides to update model parameters and synchronizes the updated parameters back to both training nodes.

[0051] In this implementation, the systematic modular division of labor forms a reusable standard training foundation by separating data, layering tasks, elastic scaling, controlled transmission, and unified updates. This reduces the deployment and operation complexity of hybrid cloud distributed training and improves training scalability and stability under the premise of secure isolation.

[0052] In the above embodiments, preferably, the large-scale model distributed training system based on the all-in-one machine hybrid cloud further includes a training control node, which includes a resource monitoring unit and an instance lifecycle management unit.

[0053] The resource monitoring unit acquires GPU utilization, storage usage, and network bandwidth, and generates resource load snapshots. It also correlates these with training task progress to output the state variables required for scaling decisions. The instance lifecycle management unit, upon receiving output from the resource monitoring unit, performs lifecycle operations on the cloud-side elastic computing instances corresponding to the cloud-side training nodes—including application, initialization into the pool, health checks, and release from the pool—based on the monitoring results and training task progress. During scaling, it synchronously updates the available node set of the task hierarchical scheduling module to avoid scheduling unavailable instances.

[0054] In this implementation, monitoring and lifecycle management form a closed loop, enabling cloud computing power scaling to shift from manual application to state-driven automated orchestration, reducing the impact of resource fluctuations on training continuity, and reducing idle and ineffective use of cloud resources.

[0055] In the above embodiments, preferably, the training control node includes a topology-aware scheduling unit, which is used for: Obtain the training task parameters, which should include at least the number of parameters L and the batch size B; Obtain computing node information, which includes at least computing power type, computing performance, topology location, and network bandwidth. The topology location includes at least rack identifier and region identifier. The task-computing power matching degree is determined based on training task parameters and computing performance, the topology weight is determined based on rack identifier and region identifier, the bandwidth weight is determined based on network bandwidth, and the node score is generated by combining the task-computing power matching degree, topology weight and bandwidth weight. Select the node group with the highest score to perform hierarchical scheduling of tasks.

[0056] During scheduling, the task-computing power matching degree M is first calculated, using L×B and computing performance F to form a matching degree metric. Then, a topology weight is generated based on the topology location, assigning higher weights to nodes within the same rack, medium weights to nodes across racks, and lower weights to nodes across regions. The node bandwidth is then normalized to form a bandwidth weight. Subsequently, the matching degree, topology weight, and bandwidth weight are weighted to obtain a node score. Node groups that meet the required number of nodes are selected from high to low scores and used as priority landing points for core or extended tasks, thereby achieving better topology and performance matching when GPU and NPU computing power nodes coexist.

[0057] Specifically, in one embodiment, the topology-aware scheduling unit takes training task parameters and computing node information as input and outputs a list of node groups for executing the training task. The training task parameters include at least the number of parameters L, the batch size B, and the required number of nodes (required_nodes); the computing node information includes at least the node identifier (node_id), computing power type, computing performance F, rack identifier (rack_id), region identifier (region_id), and network bandwidth (bandwidth).

[0058] During scheduling, a node score is calculated for each candidate computing power node. First, a task-computing power matching index M is calculated based on the task size and node computing performance. M is related to the task size L×B and the computing performance F, and this matching index is converted into a matching score (match_score) to characterize the node's capacity to handle the current task. Second, a topology weight is determined based on the topology location: 1.0 when the node is in the same rack as the local node; 0.7 when the node is in the same region but in a different rack; and 0.3 when the node spans different regions. Third, the node bandwidth is normalized to the maximum bandwidth of the candidate nodes to obtain a bandwidth weight (bandwidth_weight). Then, the match_score, topology weight, and bandwidth weight are combined using preset weight coefficients to obtain the total score (total_score), where the weight coefficients for match_score, topology weight, and bandwidth weight are 0.5, 0.3, and 0.2, respectively. Finally, all candidate nodes are sorted by total_score, and the nodes with the highest scores (required_nodes) are selected to form a node group, which is then used as the landing point for hierarchical task scheduling.

[0059] In this implementation, by incorporating task size, node performance, and physical topology into the scoring, node groups with better bandwidth conditions in the same rack or region are prioritized. This reduces the synchronization overhead caused by cross-rack and cross-region communication, improves the network consistency and parameter synchronization efficiency of node groups, reduces training jitter caused by topology mismatch, and improves the overall training throughput.

[0060] In the above embodiments, preferably, the training control node includes a fault-tolerant unit, which is used to monitor the node status of the local training node and the cloud-side training node, and to establish a task-node mapping relationship.

[0061] When a failure is detected in the cloud GPU instance corresponding to the cloud training node, the fault tolerance unit migrates the training task responsible for the instance to other normal cloud training nodes, or migrates it back to the local training node when cloud resources are insufficient to keep training going. When a failure is detected in a local training node, the core tasks will be migrated to other local training nodes in the local integrated machine cluster to keep sensitive data-related training processes out of the domain and avoid training interruptions.

[0062] In this implementation, training interruption rate is reduced and training continuity is improved by fault detection and task migration. In scenarios with fluctuating cloud instances, duplicate training and wasted computing power are reduced, while ensuring the security boundary and stable execution of core tasks on the local side.

[0063] According to the large-scale model distributed training method based on integrated machine hybrid cloud disclosed in the above embodiments, in one embodiment, when the training model includes a MoE structure, the training engine adopts the EPLB load balancing mechanism to reduce expert hotspots and queuing congestion. EPLB maintains the number of experts (num_experts), the number of nodes (num_nodes), and the single expert capacity (expert_capacity), and initializes the mapping relationship from experts to nodes (expert_node_map). The initialization strategy is to allocate experts by taking the modulo of their node numbers, thereby ensuring a balanced initial distribution of experts across nodes. EPLB also maintains an expert load count (expert_load) to record the number of tokens allocated to each expert during the current scheduling period.

[0064] During the token distribution phase, EPLB selects the top_k candidate experts for each token's routing weights (routing_weights). For each candidate expert, it first checks if their load has reached the capacity threshold: if not, they are selected into the allocation set; if so, they are replaced by the least loaded candidate expert within the same node; when there are no available candidates on the same node, the expert with the lowest current load is selected globally as a fallback allocation. After selection, the load count of the selected expert is updated, and the token allocation to the expert set and corresponding node is output. During the scheduling cycle, EPLB performs replica expansion for high-load experts whose load occupancy reaches the threshold (load_threshold): a replica expert identifier is added to the expert and mapped to the same node, enabling high-load experts to have higher parallel carrying capacity; the load count is reset after the scheduling cycle ends, and statistics are recalculated for the next cycle.

[0065] By using capacity constraints, alternative replacements for the same node, and global fallback, token allocation is prevented from being concentrated in a small number of experts, thus avoiding congestion. By expanding the replicas of high-load experts, the load balancing and throughput stability in the MoE scenario are improved, and training fluctuations caused by expert hotspots are reduced.

[0066] In one embodiment, to adapt to domestic computing power types, the training control node loads the corresponding distributed training configuration and initializes the training engine according to the device_type during the training startup phase. If the device_type is a domestic NPU, the NPU runtime library is loaded and the NPU device is configured, internal format optimization is enabled and the compilation optimization level is set. At the same time, a training configuration containing the training batch size train_batch_size, gradient accumulation steps gradient_accumulation_steps, mixed precision fp16 switch, and ZeRO optimization parameters is generated. The ZeRO optimization stage is set to stage 3, and the optimizer state and model parameter unloading target are configured to the NPU side to reduce main memory pressure.

[0067] If device_type is a domestic computing power, the corresponding runtime library is loaded and the device type is set, generating a training configuration that includes train_batch_size, gradient_accumulation_steps, fp16, and ZeRO optimization stage 2. If device_type is a regular GPU, the default training configuration is loaded.

[0068] After obtaining the configuration, the training entry point calls the training engine initialization interface to complete the binding of the model, optimizer and data loader. In the training loop, it performs forward calculation to obtain the loss, backpropagation to accumulate gradients and parameter update steps for each batch, and outputs the training round and loss metric for operation monitoring.

[0069] By loading differentiated configurations according to computing power type and unifying the training entry process, the engineering adaptation of domestic computing power is achieved, reducing the cost of switching training environments and improving deployment consistency and training availability in hybrid cloud heterogeneous computing power scenarios.

[0070] The large-scale model distributed training method and system based on integrated machine hybrid cloud disclosed in the above embodiments have the following technical effects during implementation: (1) Significantly improved computing power scalability: The coordinated scheduling of local clusters and cloud clusters enables computing power to be flexibly expanded according to training needs, supporting the training of large models with trillions of parameters. Compared with the pure local solution, the expansion cycle is shortened from several weeks to several hours, and the upper limit of computing power expansion is increased by 5-10 times. (2) Resource utilization and cost optimization: By dynamically releasing idle cloud resources, the local cluster resource utilization rate is increased from less than 40% to 60%-70%, and the overall training cost is reduced by 30%-40% compared with the pure cloud solution and by 20%-30% compared with the pure local solution; (3) Balancing data security and training efficiency: Core sensitive data is stored locally, meeting data privacy compliance requirements and reducing the risk of data leakage by more than 90%; at the same time, through data transmission optimization and low-latency communication channels, training efficiency is improved by 40%-60% compared to pure cloud solutions, and is on par with pure local solutions. (4) Enhanced stability and applicability: The fault tolerance mechanism reduces the training task interruption rate from 15%-20% in the pure cloud solution to less than 5%, and supports large model training in different scenarios such as finance, government affairs, and scientific research, adapting to diverse training needs.

[0071] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A distributed training method for large models based on an integrated machine and hybrid cloud, characterized in that, include: A hybrid cloud training architecture is established between a local all-in-one machine cluster and a public cloud elastic cluster. The local all-in-one machine cluster includes at least a training control node, a local training node, and local storage, while the public cloud elastic cluster includes at least a cloud-side training node and cloud-side storage. The local all-in-one machine cluster and the public cloud elastic cluster are interconnected through a secure communication channel. The training data is divided into core sensitive data and non-sensitive data according to sensitivity, and the core sensitive data is stored in the local storage, while the non-sensitive data is stored in the cloud storage. The large model training task is divided into core tasks and extended tasks. The core tasks are scheduled to be executed on the local training nodes of the local integrated machine cluster, and the extended tasks are scheduled to be executed on the cloud-side training nodes of the public cloud elastic cluster. The core tasks are executed based on the core sensitive data, and the extended tasks are executed based on the non-sensitive data. During training, the resource load and training task progress of the local integrated machine cluster are monitored, and the cloud-side training nodes are applied for or released according to the resource load and training task progress, so that the computing power of the public cloud elastic cluster can scale as the training stage changes. The transmission of necessary information for collaborative training is restricted between the local all-in-one cluster and the public cloud elastic cluster. The necessary information includes at least model parameter data, gradient data, and task status information. A distributed training collaboration mechanism is used to update parameters, wherein gradient data generated by the local training node and the cloud-side training node are aggregated to update model parameters, and the updated model parameters are used in the next training iteration.

2. The large-scale model distributed training method based on integrated machine hybrid cloud according to claim 1, characterized in that, The local training node consists of a preset number of AI all-in-one machines, each of which is equipped with multiple GPUs, local SSD storage, and an interconnection network. The local storage includes distributed storage nodes, which are used to store the core sensitive data, model backbone parameters and intermediate training results. The training control node is used to perform task splitting and scheduling, resource management, and collaborative communication with the public cloud elastic cluster.

3. The large-scale model distributed training method based on integrated machine hybrid cloud according to claim 1, characterized in that, The cloud-side training nodes consist of a preset number of cloud GPU instances, and the cloud-side storage is cloud object storage; the secure communication channel is established based on a cloud private network to achieve encrypted interconnection between the local all-in-one machine cluster and the public cloud elastic cluster.

4. The large-scale model distributed training method based on integrated machine hybrid cloud according to claim 1, characterized in that, The core tasks include sensitive data preprocessing, model backbone network training, and core parameter updating. The extended tasks include non-sensitive data augmentation, model branch network training, and gradient calculation assistance. The core sensitive data includes user privacy data and core business data, while the non-sensitive data includes public datasets and data generated through data augmentation.

5. The large-scale model distributed training method based on integrated machine hybrid cloud according to claim 1, characterized in that, The resource load includes at least GPU utilization, storage occupancy, and network bandwidth. The specific process for applying for or releasing cloud-side training nodes includes: When the GPU utilization reaches the first threshold or the training enters a large-scale iteration phase, apply for additional cloud-side training nodes to expand computing power. When the GPU utilization rate is lower than the second threshold or training enters the data preprocessing stage or model fine-tuning stage, the idle cloud-side training nodes are released. The first threshold is 85%, and the second threshold is 30%.

6. The large-scale model distributed training method based on integrated machine hybrid cloud according to claim 1, characterized in that, Before cross-domain transmission, the necessary information undergoes transmission optimization processing. The transmission optimization processing includes performing quantization compression and sparsification processing on the model parameter data and the gradient data, and adjusting the transmission rate of the necessary information based on public network bandwidth fluctuations. The distributed training collaboration mechanism includes parallel strategy fusion, wherein the local integrated machine cluster uses a hybrid parallel strategy of model parallelism and data parallelism to execute the core task, and the public cloud elastic cluster uses a data parallel strategy to execute the extended task.

7. A large-scale distributed training system for models based on an all-in-one machine and hybrid cloud, characterized in that, The large-scale model distributed training method based on hybrid cloud using an all-in-one machine, as described in any one of claims 1 to 6, includes: a local all-in-one machine cluster, a public cloud elastic cluster, a secure communication channel, a data isolation module, a task hierarchical scheduling module, an elastic scaling module, a transmission control module, and a gradient aggregation module; The local integrated machine cluster includes local training nodes and local storage, the public cloud elastic cluster includes cloud-side training nodes and cloud-side storage, and the secure communication channel is used to realize the interconnection between the local integrated machine cluster and the public cloud elastic cluster. The data isolation module is used to divide the training data into core sensitive data and non-sensitive data according to their sensitivity, and to control the core sensitive data to be stored in the local storage and the non-sensitive data to be stored in the cloud storage. The task hierarchical scheduling module is used to split the large model training task into core tasks and extended tasks, and schedule the core tasks to be executed on the local training node and the extended tasks to be executed on the cloud-side training node. The elastic scaling module is used to apply for or release the cloud-side training nodes based on the resource load and training task progress of the local all-in-one machine cluster. The transmission control module is used to restrict the transmission of necessary collaborative training information across domains. The necessary information includes at least model parameter data, gradient data, and task status information. The gradient aggregation module is used to aggregate the gradient data generated by the local training node and the cloud-side training node to update the model parameters, and synchronize the updated model parameters to the local training node and the cloud-side training node.

8. The large-scale model distributed training system based on integrated machine hybrid cloud as described in claim 7, characterized in that, It also includes a training control node, which comprises a resource monitoring unit and an instance lifecycle management unit; The resource monitoring unit is used to obtain GPU utilization, storage occupancy and network bandwidth. The instance lifecycle management unit is used to apply for or release the cloud-side elastic computing instance corresponding to the cloud-side training node based on the monitoring results of the resource monitoring unit and the training task progress.

9. The large-scale model distributed training system based on an all-in-one machine and hybrid cloud as described in claim 7, characterized in that, The training control node includes a topology-aware scheduling unit, which is used for: Obtain training task parameters, which include at least the number of parameters and the batch size; Obtain computing node information, which includes at least computing power type, computing performance, topology location, and network bandwidth, wherein the topology location includes at least rack identifier and region identifier; The task-computing power matching degree is determined based on the training task parameters and the computing performance; the topology weight is determined based on the rack identifier and the region identifier; the bandwidth weight is determined based on the network bandwidth; and the node score is generated by combining the task-computing power matching degree, the topology weight, and the bandwidth weight. The node group with the highest score is selected to perform the hierarchical scheduling of the task.

10. The large-scale model distributed training system based on an all-in-one machine and hybrid cloud as described in claim 7, characterized in that, The training control node includes a fault-tolerant unit, which is used to monitor the node status of the local training node and the cloud-side training node. When a failure is detected in the cloud GPU instance corresponding to the cloud training node, the corresponding training task is migrated to other normal cloud training nodes or local training nodes. When a failure is detected in a local training node, the core task will be migrated to other local training nodes in the local integrated machine cluster.

Images