Hybrid parallel training-oriented cross-data center calculation and optical network joint scheduling method and device

Abstract

The invention provides a hybrid parallel training-oriented cross-data center calculation and optical network joint scheduling method and device, and the method comprises the steps: carrying out the hybrid parallel configuration and assembly line stage division according to a model structure of a training task, a time model formula is adopted to determine data parallel gradient synchronous communication traffic, calculation time of each assembly line stage on single-micro batch, tensor parallel communication traffic and assembly line parallel communication traffic between adjacent stages; and with minimization of the total time of single iteration as a target, under the constraints of calculation and an optical network, performing joint optimization to determine a data center deployment position, calculation resources and an optical network path and bandwidth for cross-data center communication in each assembly line stage so as to perform deployment and optical channel establishment. According to the invention, the problem of low training efficiency caused by mutual independence of calculation scheduling and optical network resource allocation and lack of collaboration in cross-data center training can be solved, collaborative optimization of calculation and optical network resources can be realized, the training time delay is effectively reduced, and the resource utilization efficiency is improved.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and deep learning technology, and in particular to a method and device for joint scheduling of cross-data center computing and optical networks for hybrid parallel training. Background Technology

[0002] With the development of artificial intelligence and deep learning technologies, the parameter scale and training data scale of large models continue to grow. Model training has gradually expanded from a single server or a single data center (DC) to collaborative training across multiple geographically dispersed data centers. Cross-data center training can fully utilize computing resources distributed in different regions, alleviating to some extent the limitations of a single data center in terms of computing power, energy consumption, and physical space, and has become an important development trend for large model training. In cross-data center training scenarios, training tasks typically employ a combination of various parallelism methods, such as data parallelism, pipelined parallelism, and tensor parallelism. While these parallelism methods improve computational efficiency, they also introduce complex communication requirements such as parameter gradient synchronization and inter-stage activation data transmission, which need to be completed through wide area networks. At the same time, the interconnection network between data centers is gradually evolving towards optical networks with characteristics such as high bandwidth, low latency, and strong reconfigurability, providing a high-performance communication foundation for cross-data center training.

[0003] Currently, scheduling schemes for hybrid parallel training of large models are primarily designed for use within a single data center. In these schemes, training tasks are deployed within the same data center, and scheduling decisions are mainly determined based on the state of computing resources within the data center (such as the number of available GPUs and node computing load). Data exchange during training is completed through the high-speed interconnect network within the data center, and communication resources are considered fixed or sufficient by default; there is little separate modeling and control of communication resources during scheduling. Furthermore, the training scheduling system is independent of the underlying network resource management system, and the scheduling decision-making process does not dynamically perceive or jointly optimize network link status, bandwidth allocation, or communication latency.

[0004] However, the aforementioned existing technical solutions have significant drawbacks in cross-data center training scenarios: because computation scheduling and network resource allocation are independent and lack a unified coordination mechanism, it is difficult for computation resource scheduling and optical network resource allocation to work effectively together. Specifically, scheduling decisions are based solely on the local computation resource status, without explicitly modeling the communication dependencies between training subtasks; underlying network resources are treated as uncontrollable or static resources, failing to incorporate optical network bandwidth, path, and latency into the scheduling decision-making process; and in scenarios with multiple concurrent training tasks, situations easily arise where some data center computation resources are idle while cross-data center links are congested, resulting in low overall resource utilization efficiency. Therefore, there is an urgent need for a technical solution that can comprehensively consider the computational and communication characteristics of training tasks and coordinate scheduling with the optical network resource status in cross-data center large-scale model training scenarios. Summary of the Invention

[0005] In view of this, embodiments of this application provide a method and apparatus for joint scheduling of cross-data center computing and optical networks for hybrid parallel training, so as to eliminate or improve one or more defects existing in the prior art.

[0006] One aspect of this application provides a cross-datacenter computing and optical network joint scheduling method for hybrid parallel training, including: Based on the model structure, hybrid parallel configuration, and pipeline stage division scheme of the hybrid parallel training task to be scheduled, the amount of data parallel gradient synchronization communication data, the computation time of each pipeline stage on a single microbatch, the amount of tensor parallel communication data within each pipeline stage, and the amount of pipeline parallel communication data between adjacent pipeline stages are determined using a preset time model formula. Based on the computation time, tensor parallel communication data volume, pipeline parallel communication data volume, and data parallel gradient synchronization communication data volume, with the goal of minimizing the total time of a single iteration of the hybrid parallel training task, and under the condition of satisfying the pre-acquired computational resource constraints and optical network resource constraints, the joint optimization decision result corresponding to the hybrid parallel training task is determined. The joint optimization decision result includes: the data center deployment location of each pipeline stage, the computational resources allocated to each pipeline stage, and the optical network path and bandwidth resources allocated for cross-data center communication. Based on the joint optimization decision results, corresponding pipeline stages are deployed and computing resources are allocated in each data center, and corresponding optical channels are established in the optical network for cross-data center communication.

[0007] In some embodiments of this application, the hybrid parallel training task includes multiple concurrently executed copies of the training task; Correspondingly, based on the computation time, the amount of tensor parallel communication data, the amount of pipelined parallel communication data, and the amount of data parallel gradient synchronization communication data, with the objective of minimizing the total time of a single iteration of the hybrid parallel training task, and under the condition of satisfying the pre-acquired computational resource constraints and optical network resource constraints, the joint optimization decision result corresponding to the hybrid parallel training task is determined, including: Based on the computation time, tensor parallel communication data volume, pipeline parallel communication data volume, and data parallel gradient synchronization communication data volume corresponding to each of the training task replicas, an objective function is constructed for each of the training task replicas based on a preset single-iteration total time function form; wherein, the objective function is used to represent the data center deployment location of the pipeline stage of a training task replica, the computational resources allocated to each pipeline stage of the training task replica, and the optical network path and bandwidth resources allocated for cross-data center communication of the training task replica; The objective is to minimize the weighted sum of the total time of a single iteration for each of the training task replicas, wherein the weights of the weighted sum are determined based on the preset priority or preset resource requirements of each of the training task replicas. Under the constraints of computing resources and optical network resources, the data center deployment location, computing resource allocation, and optical network path and bandwidth resources for cross-data center communication in the pipeline stage of each training task replica are uniformly optimized to determine the joint optimization decision result corresponding to each training task replica.

[0008] In some embodiments of this application, the step of constructing the objective function for each of the training task replicas based on a preset single-iteration total time function form, according to the computation time, tensor parallel communication data volume, pipeline parallel communication data volume, and data parallel gradient synchronization communication data volume corresponding to each of the training task replicas, includes: For each pipeline stage in each of the training task replicas, a first functional relationship is established between the computation execution time of the pipeline stage and the computation resources allocated to the pipeline stage, based on the computation time corresponding to the pipeline stage. For each pipeline stage in each of the training task replicas, the tensor parallel communication time of that pipeline stage is determined based on the amount of tensor parallel communication data corresponding to that pipeline stage. For each pair of adjacent pipeline stages in each of the training task replicas, a second functional relationship is established between the pipeline parallel communication time between adjacent pipeline stages and the data center deployment location of the adjacent pipeline stages, as well as the optical network path and bandwidth resources allocated for cross-data center communication of the training task replicas, based on the amount of pipeline parallel communication data between adjacent pipeline stages. Furthermore, for each of the training task replicas, based on the amount of data parallel gradient synchronization communication data corresponding to the training task replica, a third functional relationship is established between the data parallel gradient synchronization communication time of the training task replica and the optical network path and bandwidth resources allocated for the cross-data center communication of the training task replica. For each training task replica, the objective function of the training task replica is constructed based on the first functional relationship, the second functional relationship, the third functional relationship, and the tensor parallel communication time, combined with the preset pipeline parallelism of the training task replica.

[0009] In some embodiments of this application, it also includes: During the training process of the model corresponding to the hybrid parallel training task, the actual execution time of each pipeline stage and the cross-data center communication status are monitored in real time. If the deviation between the currently monitored actual execution time and the calculation time exceeds a preset threshold, and / or if the currently monitored cross-data center communication status meets a preset trigger condition, then the calculation time is corrected according to the actual execution time, the availability information of the optical network resources is updated according to the cross-data center communication status, and based on the corrected calculation time and the updated availability information of the optical network resources, at least one of the following is adjusted: the data center deployment location of the currently executing pipeline stage, the computing resources allocated to each pipeline stage, and the optical network path and bandwidth resources allocated for cross-data center communication, so as to redetermine the corresponding joint optimization decision result; Based on the redefined joint optimization decision, at least one of the following is adjusted: the data center deployment location of the pipeline stage of the currently executing hybrid parallel training task, the computing resources allocated to each pipeline stage, and the optical network path and bandwidth resources allocated for cross-data center communication. Training of the model corresponding to the hybrid parallel training task continues.

[0010] In some embodiments of this application, before determining the amount of data-parallel gradient synchronization communication data corresponding to the hybrid parallel training task, the computation time of each pipeline stage on a single microbatch, the amount of tensor-parallel communication data within each pipeline stage, and the amount of pipeline-parallel communication data between adjacent pipeline stages, the following steps are further included: Obtain a training request for a hybrid parallel training task for a model to be scheduled; wherein the training request includes the model structure and hybrid parallel configuration of the hybrid parallel training task; With the goal of minimizing the maximum computation time of each pipeline stage on a single microbatch, the model is divided into multiple consecutive pipeline stages based on the pipeline parallelism in the hybrid parallel configuration, to obtain the pipeline stage partitioning scheme.

[0011] In some embodiments of this application, before determining the joint optimization decision result corresponding to the hybrid parallel training task, the method further includes: The system acquires the current computing resource status and optical network resource status across data center environments. The computing resource status includes the number of available GPUs and available storage capacity in each data center. The optical network resource status includes the available bandwidth capacity of each link. The computing resource constraints are determined based on the computing resource status, wherein the computing resource constraints include: the total number of GPUs allocated to all scheduled hybrid parallel training tasks in each data center does not exceed the number of available GPUs in that data center, and the storage resources allocated to each pipeline stage do not exceed the available storage capacity of that data center. Furthermore, optical network resource constraints are determined based on the optical network resource status, wherein the optical network resource constraints include: the frequency slots allocated for cross-data center communication do not exceed the total frequency slot capacity of each link, and the frequency slots occupied by cross-data center communication of the same hybrid parallel training task meet preset spectral continuity constraints, spectral consistency constraints, and guard band isolation constraints.

[0012] In some embodiments of this application, the time model formula includes: total computation formula, computation time formula, data parallel communication volume formula, tensor parallel communication volume formula, and pipeline parallel communication volume formula. Correspondingly, the step of determining the amount of data parallel gradient synchronization communication data, the computation time of each pipeline stage on a single microbatch, the amount of tensor parallel communication data within each pipeline stage, and the amount of pipeline parallel communication data between adjacent pipeline stages, based on the model structure, hybrid parallel configuration, and pipeline stage division scheme of the hybrid parallel training task to be scheduled, using a preset time model formula, includes: Based on the computational amount of each computational layer in the model structure and the pipeline stage division scheme, the total computational amount of each pipeline stage is determined based on the total computational amount formula; and the total computational amount, the tensor parallelism in the hybrid parallel configuration, and the preset single GPU computing power are substituted into the computational time formula to determine the computational time of each pipeline stage on a single microbatch. Based on the parameter sizes of each computational layer in the model structure, the total parameter size of the hybrid parallel training task is determined, and based on the data parallelism in the hybrid parallel configuration, the amount of data parallel gradient synchronization communication data is determined according to the data parallel communication volume formula. Based on the tensor parallel communication data volume of each computational layer in the model structure and the pipeline stage division scheme, the tensor parallel communication data volume within each pipeline stage is determined based on the tensor parallel communication volume formula. Furthermore, based on the activation size of each computational layer in the model structure and the pipeline stage partitioning scheme, the pipeline parallel communication data volume between adjacent pipeline stages is determined according to the pipeline parallel communication volume formula.

[0013] In some embodiments of this application, the cross-data center communication status satisfies preset triggering conditions, including: The actual bandwidth of cross-data center communication is lower than a preset bandwidth threshold, the communication latency exceeds a preset latency threshold, and optical network link congestion is detected, at least one of the following:

[0014] Another aspect of this application provides a cross-data center computing and optical network joint scheduling system for hybrid parallel training, comprising: The time model parameter calculation module is used to determine, based on the model structure, hybrid parallel configuration, and pipeline stage division scheme of the hybrid parallel training task to be scheduled, the amount of data parallel gradient synchronization communication corresponding to the hybrid parallel training task, the computation time of each pipeline stage on a single microbatch, the amount of tensor parallel communication data within each pipeline stage, and the amount of pipeline parallel communication data between adjacent pipeline stages, using a preset time model formula. The joint optimization decision module is used to determine the joint optimization decision result corresponding to the hybrid parallel training task based on the computation time, the tensor parallel communication data volume, the pipeline parallel communication data volume, and the data parallel gradient synchronization communication data volume, with the goal of minimizing the total time of a single iteration of the hybrid parallel training task, under the condition of satisfying the pre-acquired computational resource constraints and optical network resource constraints. The joint optimization decision result includes: the data center deployment location of each pipeline stage, the computational resources allocated to each pipeline stage, and the optical network path and bandwidth resources allocated for cross-data center communication. The deployment execution module is used to deploy the corresponding pipeline stages and allocate computing resources in each data center based on the joint optimization decision results, and to establish corresponding optical channels for cross-data center communication in the optical network.

[0015] A third aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the aforementioned method for joint scheduling of cross-data center computing and optical networks for hybrid parallel training.

[0016] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the described cross-data center computing and optical network joint scheduling method for hybrid parallel training.

[0017] The fifth aspect of this application provides a computer program product comprising a computer program that, when executed by a processor, implements the described cross-data center computing and optical network joint scheduling method for hybrid parallel training.

[0018] The cross-data center computing and optical network joint scheduling method for hybrid parallel training provided in this application determines the amount of data parallel gradient synchronization communication data, the computation time of each pipeline stage on a single microbatch, the amount of tensor parallel communication data within each pipeline stage, and the amount of pipeline parallel communication data between adjacent pipeline stages by using a preset time model formula. This quantifies and models the three communication types (tensor parallel, pipeline parallel, and data parallel) with different latency sensitivities and bandwidth requirements in hybrid parallel training, enabling subsequent scheduling decisions to specifically meet the needs of different communication types. It transforms abstract communication dependencies into specific data volume parameters, providing precise input for optical network bandwidth allocation and avoiding resource misallocation caused by experience or static configuration in existing technologies. Furthermore, it refines the computation time to each pipeline stage and each microbatch, allowing subsequent optimization to accurately identify performance bottleneck stages and achieve finer-grained resource allocation. The method further considers the computation time, tensor parallel communication data volume, pipeline parallel communication data volume, and data parallel gradient synchronization communication data volume. With the objective of minimizing the total time of a single iteration of the hybrid parallel training task, and under the condition of satisfying pre-acquired computational resource constraints and optical network resource constraints, it determines the joint optimization decision result corresponding to the hybrid parallel training task. By placing computational resource allocation and optical network resource allocation under the same objective function and constraints for joint decision-making, it can fundamentally solve the decision-making conflict caused by the independent operation of computation scheduling and network resource management in existing technologies. Using the total time of a single iteration as the optimization objective directly links the optimization process to the core indicators of training efficiency, avoiding the problem of sacrificing overall performance by separately optimizing local computational efficiency and local network efficiency in existing technologies. Optimization is performed under the condition of satisfying computational resource constraints (number of GPUs, storage capacity) and optical network resource constraints (slot capacity, spectrum continuity, etc.), ensuring the feasibility of the decision result and avoiding resource over-allocation or allocation conflicts.The method further optimizes the decision-making process by designing the following: the data center deployment location of each pipeline stage, the computing resources allocated to each pipeline stage, and the optical network path and bandwidth resources allocated for cross-data center communication. By deploying adjacent pipeline stages with high communication volume in the same or adjacent data centers, cross-data center communication traffic can be significantly reduced, thus mitigating the impact of communication latency on training efficiency. Dynamic allocation of GPU resources based on the computational load differences of each pipeline stage avoids performance bottlenecks due to insufficient resources in some stages, while also preventing resource idleness and waste. Allocating explicit optical network paths and bandwidth resources for cross-data center communication avoids congestion caused by multiple tasks competing for the same link, improving the utilization rate of optical network resources. The three decision variables are coupled and jointly optimized under the objective function, avoiding the side effects of single-dimensional optimization (e.g., optimizing only the deployment location may lead to unbalanced computing resources, and optimizing only the bandwidth allocation may lead to inappropriate path selection). In addition, based on the joint optimization decision results, the method deploys corresponding pipeline stages and allocates computing resources in each data center, and establishes corresponding optical channels for cross-data center communication in the optical network, transforming the optimization results into actual deployment operations and resource allocation, which can ensure that theoretical optimization can be implemented into actual performance improvement; by establishing clear optical channels in the optical network, it can provide definite bandwidth and paths for cross-data center communication, avoiding performance fluctuations caused by statistical multiplexing in traditional IP networks.

[0019] Additional advantages, objectives, and features of this application will be set forth in part in the description which follows, and will in part become apparent to those skilled in the art upon review of the following description, or may be learned by practice of the application. The objectives and other advantages of this application can be realized and obtained by means of the structures specifically pointed out in the specification and drawings.

[0020] Those skilled in the art will understand that the purposes and advantages that can be achieved with this application are not limited to those specifically described above, and that the above and other purposes that this application can achieve will be more clearly understood from the following detailed description. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, do not constitute a limitation thereof. The components in the drawings are not drawn to scale but are merely for illustrating the principles of this application. For ease of illustration and description of certain parts of this application, corresponding portions in the drawings may be enlarged, i.e., may appear larger relative to other components in an exemplary system actually manufactured according to this application. In the drawings: Figure 1 This is a diagram comparing training within a single data center and training across data centers.

[0022] Figure 2 This is a schematic diagram of the first process of a cross-data center computing and optical network joint scheduling method for hybrid parallel training in one embodiment of this application.

[0023] Figure 3 This is a schematic diagram illustrating an example of a hybrid parallel training task in one embodiment of this application.

[0024] Figure 4 This is a schematic diagram of the second process of a cross-data center computing and optical network joint scheduling method for hybrid parallel training in one embodiment of this application.

[0025] Figure 5 This is a schematic diagram of the framework of a joint scheduling scheme in one embodiment of this application.

[0026] Figure 6 This is a schematic diagram of the structure of a cross-data center computing and optical network joint scheduling system for hybrid parallel training according to an embodiment of this application.

[0027] Figure 7 This is a schematic diagram of the execution flow of the cross-data center computing and optical network joint scheduling method for hybrid parallel training in an application example of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and their descriptions are used to explain this application, but are not intended to limit it.

[0029] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the structures and / or processing steps closely related to the solution according to this application are shown in the accompanying drawings, while other details that are not closely related to this application are omitted.

[0030] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0031] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0032] In the following description, embodiments of the present application will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0033] First, it should be noted that training tasks typically employ a combination of various parallelism methods, such as data parallelism (DP), pipeline parallelism (PP), and tensor parallelism (TP).

[0034] Existing training scheduling schemes and optical network resource management schemes are typically completed independently by different systems, lacking a unified coordination mechanism. This leads to difficulties in effectively coordinating computational resource scheduling and optical network resource allocation, resulting in low overall resource utilization efficiency. For example, one type of existing technical solution related to this application mainly targets the hybrid parallel training scheduling of large models within a single data center. In this type of solution, training tasks are usually deployed within the same data center, and the computational resources used include multiple computing nodes or GPU servers within the same server room. To improve training efficiency, the training process generally adopts hybrid parallel methods such as data parallelism, pipelined parallelism, and tensor parallelism to divide and coordinate the model computation process. Correspondingly, the scheduling decision for training subtasks is mainly determined based on the state of computational resources within the data center, such as the number of available GPUs, node computational load, and internal interconnection topology.

[0035] In the aforementioned technical solutions, data exchange during training is primarily accomplished through high-speed interconnect networks within the data center. Communication paths and bandwidth are typically considered fixed resources or are assumed to be sufficient, with minimal separate modeling and control of communication resources during scheduling. Furthermore, the training scheduling system and the underlying network resource management system are usually independent, and the scheduling decision-making process does not dynamically perceive or jointly optimize network link status, bandwidth allocation, or communication latency. Because these technical solutions are primarily designed for single-data center environments, they can improve training efficiency to some extent in this application scenario and are therefore widely used in existing large-scale model training systems.

[0036] Because the aforementioned existing technologies employ the aforementioned technical features in their technical solutions, they inevitably suffer from the following drawbacks in cross-data center training scenarios: 1. Since training tasks are deployed within a single data center by default and lack a cross-data center scheduling mechanism, existing technologies cannot effectively scale to multi-data center collaborative training scenarios when the training scale exceeds the computing power limit of a single data center. 2. Since the scheduling decision is based only on the local computing resource status and does not explicitly model the communication dependencies between training subtasks, once the training subtasks are distributed in different data centers, a large amount of critical communication will pass through the cross-data center network, thus significantly increasing the training latency. 3. Because existing technologies treat underlying network resources as uncontrollable or static resources and do not incorporate optical network bandwidth, path and latency into the scheduling decision process, they cannot allocate network resources in a differentiated manner based on training communication characteristics, which can easily lead to communication congestion or resource waste. 4. Since computation scheduling and network resource allocation are independent of each other and lack a unified coordination mechanism, in scenarios with multiple concurrent training tasks, it is easy for some data center computing resources to be idle while cross-data center links are congested, resulting in low overall system resource utilization efficiency.

[0037] Specifically, such as Figure 1 As shown, the time series comparison between training within a single data center and training across data centers in a hybrid parallel mode is presented. Figure 1 The numbers 1 to 8 in the boxes represent different micro-batch numbers. Figure 1 The horizontal axis represents the progression of training time, and the vertical axis represents the different pipeline stages (Stage 0 to Stage 3) in pipeline parallelism. In a single-datacenter training scenario, each pipeline stage is deployed within the same datacenter. Computational nodes communicate through low-latency, high-bandwidth internal interconnects, allowing activation values ​​and gradients generated during forward and backward propagation to be rapidly transmitted between adjacent pipeline stages. Due to the low communication latency, this communication process can highly overlap with the computation process. After the pipeline is filled, it can maintain a relatively continuous and compact execution rhythm. There is only a small amount of idle time caused by pipeline filling and emptying at the beginning and end of training. The impact of the data parallel synchronization stage on the overall training rhythm is relatively limited, thus keeping the total duration of a single training iteration at a low level.

[0038] In contrast, in cross-datacenter training scenarios, different stages of the pipeline are distributed across multiple data centers, and the communication of activation values ​​and gradients between adjacent stages requires transmission through cross-datacenter links. Due to the significantly increased latency and limited available bandwidth of cross-datacenter communication, communication operations that could otherwise be masked by computation cannot fully overlap. Communication latency is explicitly amplified at pipeline stage boundaries, resulting in continuous idle intervals in multiple stages while waiting for communication to complete. As training progresses, these idle times appear periodically in each pipeline stage, forming distinct pipeline bubbles. Simultaneously, data parallel synchronization operations are further prolonged due to the impact of cross-datacenter communication. Ultimately, the overall time series of cross-datacenter training exhibits alternating computation and communication, segmented by multiple communication waits, and its single-round training iteration time is significantly increased compared to the single-datacenter scenario. Figure 1 The time difference visually illustrates the performance difference between the two training modes. This comparison shows that cross-datacenter communication has become a key performance bottleneck in hybrid parallel large model training.

[0039] In other words, the root cause of the aforementioned defects in the existing technologies lies in their design for training within a single data center and their use of scheduling techniques that decouple computing and networks, which makes it difficult to meet the requirements of cross-data center hybrid parallel training for collaborative scheduling of computing and optical networks.

[0040] To address the problem of independent and uncoordinated cross-data center computing scheduling and optical network resource allocation in existing technologies, this application provides a cross-data center computing and optical network joint scheduling method for hybrid parallel training, a cross-data center computing and optical network joint scheduling system for executing this method, an electronic device, a computer-readable storage medium, and a computer program product. It performs detailed time modeling for computing and communication, proposes a technical solution that can perceive the communication dependencies of hybrid parallel training tasks and achieve joint scheduling of computing and optical network resources in a cross-data center environment. This technical solution coordinates the deployment decisions of training tasks with the configuration of cross-data center communication resources, thereby effectively reducing training latency, alleviating communication bottlenecks, and improving the overall utilization efficiency of computing and optical network resources.

[0041] The following examples will provide a detailed description.

[0042] Based on this, embodiments of this application provide a cross-datacenter computing and optical network joint scheduling method for hybrid parallel training, which can be implemented by a cross-datacenter computing and optical network joint scheduling system for hybrid parallel training. See [link to relevant documentation]. Figure 2 The cross-data center computing and optical network joint scheduling method for hybrid parallel training specifically includes the following: Step 100: Based on the model structure, hybrid parallel configuration, and pipeline stage division scheme of the hybrid parallel training task to be scheduled, determine the amount of data parallel gradient synchronization communication data, the computation time of each pipeline stage on a single microbatch, the amount of tensor parallel communication data within each pipeline stage, and the amount of pipeline parallel communication data between adjacent pipeline stages using a preset time model formula.

[0043] In one or more embodiments of this application, a hybrid parallel training task refers to a model training task that simultaneously employs multiple parallel methods such as data parallelism, pipeline parallelism, and tensor parallelism, such as the training task of neural network models like the Transformer model.

[0044] The model structure refers to the specific composition of the model, including information such as the number of computational layers (e.g., Transformer Blocks), the type of each layer, the computational cost per layer, parameter size, and activation size. Hybrid parallel configuration includes specific values ​​for data parallelism, pipeline parallelism, and tensor parallelism. The pipeline stage (PP block or Pipeline stage) partitioning scheme is a method of dividing the model into multiple consecutive stages based on layers.

[0045] For example, consider a large model training task consisting of multiple Transformer modules, each layer including a self-attention module and a feedforward network module. The feedforward network typically involves large-scale matrix operations and constitutes the most computationally intensive part of the model training process. To efficiently complete training on large-scale GPU clusters, existing technologies often employ a hybrid parallel training approach combining data parallelism, tensor parallelism, and pipelined parallelism. Figure 3 As shown, in one example, the hybrid parallel training task consists of two data parallel groups, which are deployed in two data centers respectively. Each data parallel group uses four pipeline stages to vertically divide the large model. Within each pipeline stage, the model parameters and computational load are further divided across multiple GPUs for collaborative execution through four-way tensor parallelism.

[0046] In data parallelism (DP), data parallel groups in different data centers process different mini-batch training data. These parallel groups synchronize gradients or parameters through cross-data center communication, with the communication latency denoted as . This demonstrates the dependence of cross-domain training on the performance of wide-area networks.

[0047] In pipelined parallelism (PP), the complete large model is divided into multiple consecutive network stages (Stage 0 to Stage 3). Each stage is executed sequentially according to the forward and backward propagation order. Inter-stage communication of activation values ​​and gradients is required between adjacent stages, and the communication overhead is denoted as... This parallel approach improves the scalability of the model, but introduces significant inter-stage temporal dependencies.

[0048] In Tensor Parallelism (TP), the computation within each pipeline stage is further broken down into multiple sub-operators, which are completed in parallel by multiple GPUs. High-frequency, low-latency operator-level communication is required between GPUs. Therefore, Tensor Parallelism is usually limited to a single data center or high-speed interconnect domain to avoid performance degradation caused by cross-domain communication.

[0049] The time model formula in step 100 refers to the pre-established mathematical model used to quantify computation and communication time, also known as the computation-communication time model. It is a unified, cross-DC mathematical model constructed by this application to characterize the performance differences of cross-data center hybrid parallel training under different task partitioning and deployment schemes.

[0050] It's worth noting that the execution time of a pipeline stage (PP block or pipeline stage) consists of the serial computation of TP across all its internal layers and the collective communication of TP, with no overlap between the two. The activation / gradient propagation time between pipeline stages further affects the execution latency of that stage. Consider a training task. Its model is composed of It consists of several computational layers (such as Transformer Blocks), and the layer numbers are as follows: . No. The set of computational layers contained in each pipeline stage is as follows: Then there is In one example, the parameters for task configuration can be expressed as shown in Table 1.

[0051] Table 1 - Relevant Parameters and Their Meanings

[0052] The data-parallel gradient synchronization communication data volume refers to the total amount of gradient data that each model replica needs to synchronize after each iteration in data parallelism, determined by the total number of model parameters and the degree of data parallelism. A single micro-batch refers to dividing a global batch into multiple smaller micro-batches to improve pipeline parallelism efficiency; for example, each global batch contains 8 micro-batches. The computation time refers to the computation time required for a pipeline stage to process a single micro-batch, influenced by the total computational load of that stage, tensor parallelism, and GPU computing power. The tensor parallel communication data volume refers to the total amount of data transmitted during collective communication (such as AllGather or ReduceScatter) between multiple GPUs within the same stage in tensor parallelism, obtained by summing the TP communication volume of each layer within that stage. The pipeline parallel communication data volume is the amount of data required for transmitting activation values ​​during forward propagation and gradients during backward propagation between adjacent pipeline stages, determined by the activation size of each layer at the stage boundary.

[0053] Step 200: Based on the computation time, the tensor parallel communication data volume, the pipeline parallel communication data volume, and the data parallel gradient synchronization communication data volume, with the objective of minimizing the total time of a single iteration of the hybrid parallel training task, and under the condition of satisfying the pre-acquired computational resource constraints and optical network resource constraints, determine the joint optimization decision result corresponding to the hybrid parallel training task, wherein the joint optimization decision result includes: the data center deployment location of each pipeline stage, the computational resources allocated to each pipeline stage, and the optical network path and bandwidth resources allocated for cross-data center communication.

[0054] Understandably, the total time for a single iteration is the total time required to complete a global parameter update, including forward and backward propagation of all micro-batches, communication overhead, and pipeline bubble overhead. Minimizing this time is the objective. Computational resource constraints stipulate that the total number of GPUs allocated to all tasks within each data center must not exceed the number of available GPUs in that data center, and storage resources must not exceed available capacity. Optical network resource constraints stipulate that the total number of frequency slots allocated on each link must not exceed the total frequency slot capacity of the link, and spectrum allocation must satisfy continuity, consistency, and guard band isolation constraints. The joint optimization decision result includes three parts: which data center each pipeline stage is deployed in, how many GPUs are allocated to each stage, and which optical path and bandwidth are used for cross-data center communication. Specific values ​​can be obtained by solving the optimization problem. The data center deployment location refers to the physical data center where each pipeline stage is located, determining whether PP communication occurs within or across data centers. Computational resources mainly refer to the number of GPUs, but may also include resources such as storage, such as the number of GPUs allocated to each stage. Optical network path and bandwidth resources refer to the specific routing and allocated bandwidth (expressed in the number of frequency slots) of the optical channels established for cross-data center communication.

[0055] Specifically, the total time for a single iteration is determined by the following: the execution time of each pipeline stage on a single microbatch consists of its computation time and TP communication time (which are sequential); then, the PP boundary communication time depends on whether adjacent stages cross data centers; and the DP gradient synchronization communication time is determined by cross-data center bandwidth and latency. Ultimately, the total time for a single iteration is determined by the execution time of the slowest pipeline stage, the number of microbatches, and pipeline bubble overhead.

[0056] Specifically, in step 200, the system pre-acquires the computing resource status and optical network resource status across the data center environment. An example of the computing resource status is: assuming there are two available data centers (DC1 and DC2), each with 64 available GPUs and 64 available GPUs respectively, and sufficient available storage capacity. An example of the optical network resource status is: there is a direct optical link between DC1 and DC2, with 100 available frequency slots and a propagation delay of 10ms; there are no other paths in the optical network topology.

[0057] Based on this information, computational resource constraints and optical network resource constraints are then determined. Computational resource constraints include: the total number of GPUs allocated to each data center cannot exceed its available number. For example, if a phase is deployed in DC1, the number of GPUs allocated to it cannot exceed the available number in DC1; at the same time, the total number of GPUs allocated to all phases deployed in DC1 cannot exceed 64. Optical network resource constraints include: the number of frequency slots allocated for cross-data center communication cannot exceed the total link frequency slot capacity (100 slots), and spectrum allocation must meet requirements such as continuity (the same optical channel occupies consecutive frequency slots), consistency (frequency slots are consistent along the path), and guard band isolation (gaps are left between different tasks).

[0058] Under this constraint, the following joint decision variables can be solved using optimization algorithms (such as integer programming): (1) Data center deployment location for each pipeline stage: For example, pipeline stages Stage 0 and Stage 1 may be deployed in data center DC1, and pipeline stages Stage 2 and Stage 3 may be deployed in data center DC2 to reduce the number of cross-data center communications.

[0059] (2) Computing resources allocated to each pipeline stage: For example, based on the principle of balancing computing time in each stage, 32 GPUs are allocated to pipeline stage 0, 32 GPUs are allocated to pipeline stage 1 (both in DC1), and 32 GPUs are allocated to pipeline stages 2 and 3 (both in DC2).

[0060] (3) Optical network paths and bandwidth resources allocated for cross-data center communication: In this example, cross-data center communication includes PP communication (bidirectional) between Stage 1 and Stage 2 of the pipeline, as well as DP gradient synchronization (between two DCs). The system needs to allocate optical network paths (unique paths) and bandwidth resources (corresponding number of frequency slots) for these communications to ensure that the bandwidth meets the communication requirements without exceeding the capacity. For example, allocate 20 frequency slots (corresponding bandwidth) for PP communication and 30 frequency slots for DP synchronization, with a total usage not exceeding 100 frequency slots.

[0061] The optimization objective is to minimize the total time of a single iteration. By adjusting the decision variables mentioned above, the overall training efficiency is maximized.

[0062] Step 300: Based on the joint optimization decision results, deploy the corresponding pipeline stages and allocate computing resources in each data center, and establish corresponding optical channels for cross-data center communication in the optical network.

[0063] It should be noted that deploying the corresponding pipeline stage can start the training process of the corresponding model shard on the selected data center, loading model parameters and data. For example, start the training process of pipeline stage 0 and pipeline stage 1 on data center DC1, loading the model parameters of the corresponding layers (layers 1-12); start the training process of pipeline stage 2 and pipeline stage 3 on data center DC2, loading the parameters of layers 13-24.

[0064] Allocating computing resources involves distributing the number of GPUs needed for decision-making to each training process, which can be achieved through cluster management tools (such as Kubernetes or Slurm). For example, based on the decision results, a specified number of GPUs can be allocated to each stage: Data center DC1 allocates 32 GPUs to stage 0 and 32 GPUs to stage 1; DC2 allocates 32 GPUs to stages 2 and 3. Resource binding and task startup are then accomplished using cluster management tools (such as Kubernetes).

[0065] Establishing corresponding optical channels can be done through the optical network control plane (such as OpenFlow or ONOS) to create end-to-end optical connections for cross-data center communication, configuring bandwidth (slots) and paths. For example, based on the decision, the network controller can establish two optical channels on the optical link between data centers DC1 and DC2: one for PP communication from Stage 1 to Stage 2 of the pipeline, allocated with 20 slots; and the other for DP gradient synchronization, allocated with 30 slots. The configuration process must meet the requirements of spectrum continuity, consistency, and guard band isolation to ensure stable establishment of the optical channels.

[0066] After the above deployment is completed, the training task can be executed according to the optimized plan. Each pipeline stage will work together to propagate forward and backward, and cross-data center communication will transmit data through the established optical channels.

[0067] As described above, the cross-data center computing and optical network joint scheduling method for hybrid parallel training provided in this application accurately models the computation time of hybrid parallel training tasks and the three types of communication data volumes (tensor parallelism, pipeline parallelism, and data parallelism) by incorporating them into a preset time model formula. With the goal of minimizing the total time of a single iteration, it jointly optimizes the data center deployment location in the pipeline stage, the allocation of computing resources at each stage, and the optical network path and bandwidth resources for cross-data center communication under constraints of computing and optical network resources. Finally, based on the optimization results, it performs actual deployment and establishes optical channels. This fundamentally solves the problem of independent and uncoordinated computing scheduling and optical network resource allocation in existing technologies. This technical solution achieves deep coupling optimization of computing and communication, accurately perceives the communication needs of different parallel modes, rationally allocates computing and network resources across data centers, effectively reduces training latency introduced by cross-data center communication, avoids pipeline bubbles and resource idleness, and significantly improves the overall training efficiency and system resource utilization in multi-task concurrent scenarios.

[0068] To further address the problem that existing model partitioning methods often use a fixed number of layers, ignoring the differences in computational load between layers, leading to uneven computational load across pipeline stages, significant pipeline bubble overhead, and reduced training efficiency, this application provides a cross-data center computing and optical network joint scheduling method for hybrid parallel training, see [link to relevant documentation]. Figure 4 The method for joint scheduling of cross-data center computing and optical networks for hybrid parallel training also includes the following content before step 100: Step 010: Obtain the training request for a hybrid parallel training task for a model to be scheduled; wherein the training request includes the model structure and hybrid parallel configuration of the hybrid parallel training task.

[0069] Specifically, when receiving a training request, the system not only obtains the model structure and parallel configuration parameters, but also instantiates the parameters of the corresponding computation-communication time model based on the Transformer (a deep neural network structure composed of multiple layers of self-attention modules and feedforward network modules) hierarchical information, including but not limited to: (1) Computational load of each computational layer (Transformer Block) Activation scale With parameter size ; (2) User-specified or system-recommended data parallelism pipeline parallelism and tensor parallelism ; (3) Micro-batch number With the number of training iterations .

[0070] The above parameters are used to construct the computation time term, TP communication time term, PP boundary communication time term, and DP synchronization communication time term for each pipeline stage, providing quantifiable time cost inputs for subsequent joint optimization based on the above relevant model formulas.

[0071] Step 020: With the goal of minimizing the maximum computation time of each pipeline stage on a single microbatch, the model is divided into multiple consecutive pipeline stages according to the pipeline parallelism in the hybrid parallel configuration to obtain the pipeline stage partitioning scheme.

[0072] Specifically, the system first receives a training request for a hybrid parallel training task to be scheduled. Assume this task is the training of a large language model based on the Transformer architecture, containing 24 Transformer layers, with the computational cost of each layer known. The training request also includes a hybrid parallel configuration: data parallelism of 2, pipeline parallelism of 4, and tensor parallelism of 2.

[0073] The system then aims to minimize the maximum computation time of each pipeline stage on a single microbatch. Based on pipeline parallelism, the 24 layers of the model are divided into four consecutive pipeline stages. Since the specific deployment locations and the number of GPUs allocated to each stage are not yet determined, and each stage will use the same tensor parallelism, the computation time of each stage can be approximated as proportional to its total computational cost. Therefore, the optimization objective can be transformed into: balancing the total computational cost of each stage as much as possible, thereby minimizing the computation time of the stage with the largest computational cost. The system can use a dynamic programming algorithm to solve for the optimal partitioning.

[0074] For example: Assume the computational cost of each layer is as follows (unit: TFLOPs): (1) Floors 1-6: 15 per floor; (2) Floors 7-12: 20 per floor; (3) Floors 13-18: 10 per floor; (4) Floors 19-24: 5 per floor.

[0075] If a uniform partitioning method is used (each group consists of 6 layers), the computational costs for each stage are 90, 120, 60, and 30, respectively, with the maximum value of 120 becoming the bottleneck. Through dynamic programming, the following partitioning scheme is obtained: (1) Pipeline stage 0: Layers 1-5 (5 layers), computational cost 5×15 = 75; (2) Pipeline stage 1: Layers 6-11 (6 layers), calculation amount 1×15 + 5×20 = 115; (3) Pipeline stage 2: Layers 12-18 (7 layers), calculation amount 1×20 + 6×10 = 80; (4) Pipeline stage 3: Layers 19-24 (6 layers), calculation amount 6×5 = 30.

[0076] At this point, the maximum value is 115, which is lower than the 120 for a uniform partition, and the computational load is more balanced across stages. This partitioning scheme serves as the input for subsequent joint optimization decisions.

[0077] As can be seen from the above description, the cross-data center computing and optical network joint scheduling method for hybrid parallel training provided in this application partitions the model with the goal of minimizing the maximum stage computing time, so that the computing load of each pipeline stage is as balanced as possible, reducing the pipeline cycle time from the source, reducing pipeline bubbles, laying a good foundation for subsequent joint scheduling of computing and optical networks, and further improving the overall training efficiency.

[0078] To further address the problem in existing technologies where scheduling decisions lack awareness of real-time resource status, leading to a disconnect between resource allocation and actual conditions and resulting in resource over-allocation or allocation conflicts, this application provides a cross-data center computing and optical network joint scheduling method for hybrid parallel training, see [link to relevant documentation]. Figure 4 The method for joint scheduling of cross-data center computing and optical networks for hybrid parallel training also includes the following content before step 200: Step 030: Obtain the current computing resource status and optical network resource status across the data center environment, wherein the computing resource status includes the number of available GPUs and available storage capacity in each data center; and the optical network resource status includes the available bandwidth capacity of each link.

[0079] The computing resource status refers to the total amount of computing resources currently available for allocation in each data center, including the number of GPUs and storage capacity. The number of available GPUs is the number of idle GPUs in the data center that are available for training tasks. Available storage capacity is the storage space in the data center available for training tasks, including RAM and video memory. The optical network resource status refers to the communication resources available for allocation in the cross-data center optical network, including the available bandwidth capacity of each link. Available bandwidth capacity is the number of unused bandwidths on the optical link. A bandwidth is the basic unit of bandwidth allocation in an optical network, with each bandwidth corresponding to a certain bandwidth (e.g., 12.5 GHz).

[0080] Specifically, in step 030, the system can obtain the current computing resource status and optical network resource status of the cross-datacenter environment through the resource monitoring module. Assume the cross-datacenter environment includes two data centers: DC1 and DC2. The monitoring module collects the following information: (1) Computational resource status: DC1: 64 GPUs are available, and 10 TB of storage capacity is available (e.g., it can be used to store model parameters, activation values, etc.). DC2: 64 GPUs are available, and 8 TB of storage is available.

[0081] (2) Optical network resource status: There is a direct optical link L12 between DC1 and DC2, with a total bandwidth of 100 slots, all of which are currently available (no other tasks are using them). The optical network control plane can provide information on the slot usage of this link.

[0082] Furthermore, there are no other paths in the optical network topology, and all cross-data center communication must pass through link L12.

[0083] Step 040: Determine computing resource constraints based on the computing resource status, wherein the computing resource constraints include: the total number of GPUs allocated to all scheduled hybrid parallel training tasks in each data center does not exceed the number of available GPUs in that data center, and the storage resources allocated to each pipeline stage do not exceed the available storage capacity of that data center.

[0084] Specifically, the system determines the computational resource constraints that must be met for this scheduling based on the acquired computational resource status. These constraints will serve as boundary conditions for subsequent joint optimization decisions. Among these, the GPU quantity constraint includes: for each data center, the total number of GPUs allocated to all training tasks to be scheduled (including the task itself and other concurrent tasks) must not exceed the number of available GPUs in that data center. Mathematically, this is expressed as: for DC1: ∑(number of GPUs allocated to each stage deployed in DC1) ≤ 64, and for DC2: ∑(number of GPUs allocated to each stage deployed in DC2) ≤ 64. The storage capacity constraint includes: for each data center, the storage resources allocated to each pipeline stage (such as those used to store model parameters, activation cache, etc.) must not exceed the available storage capacity of that data center. Specifically, the storage required for each stage can be estimated based on the model parameter size and activation size, with the constraint that the sum of the storage requirements for each stage does not exceed the available capacity.

[0085] For example, in this task, the total number of model parameters is P. totalFor each byte, if a stage is deployed in DC1, its required storage is approximately several times the number of parameters of that stage (considering optimizer status, etc.). The system must ensure that the total storage requirement of all stages deployed in DC1 is ≤10TB, and similarly for DC2, it is ≤8TB.

[0086] And, step 050: determine optical network resource constraints based on the optical network resource status, wherein the optical network resource constraints include: the frequency slots allocated for cross-data center communication do not exceed the total frequency slot capacity of each link, and the frequency slots occupied by cross-data center communication of the same hybrid parallel training task meet the preset spectrum continuity constraints, spectrum consistency constraints and guard band isolation constraints.

[0087] Optical network resource constraints are optimization restrictions derived from the state of optical network resources, including slot capacity constraints and spectrum allocation specifications. Total slot capacity is the maximum number of slots a single optical link can provide; for example, link L12 has 100 slots. Spectrum continuity constraints require that the slots occupied by the same optical channel be continuous along a link. For example, the 20 slots for PP communication must be continuous. Spectrum consistency constraints require that the same optical channel occupy the same slots on each link in the path. Guard band isolation constraints require that a certain amount of free slots be reserved between different optical channels to avoid interference. For example, a guard band is required between PP communication and DP synchronization.

[0088] Specifically, in one example, the system determines optical network resource constraints based on the acquired optical network resource status to ensure the feasibility of cross-data center communication. Wherein: (1) Frequency slot capacity constraint: The total number of frequency slots allocated for cross-data center communication shall not exceed the total frequency slot capacity of the link on each link. For link L12, the sum of frequency slots occupied by all cross-data center communication (including PP communication, DP synchronization, etc. of this task) shall be ≤ 100.

[0089] (2) Spectrum continuity constraint: For optical channels established for communication across the same data center, the frequency slots occupied on a single link must be continuous. For example, if 20 frequency slots are allocated for PP communication of this task, these 20 frequency slots must be numbered consecutively on L12 (e.g., frequency slots 10-29) and cannot be scattered across multiple discontinuous regions.

[0090] (3) Spectrum consistency constraint: For optical channels established for communication across the same data center, the frequency slots occupied on each link along the path must be in the same position. In this example, the path only has one link, L12, so the consistency is automatically satisfied; if the path contains multiple links, then the frequency slots with the same number are required to be used on each link.

[0091] (4) Guard Band Isolation Constraints: Between optical channels of different training tasks, or between different optical channels of the same task, a certain amount of free frequency slots need to be reserved as guard bands to avoid signal interference. For example, if PP communication of this task occupies frequency slots 10-29 and DP synchronization occupies frequency slots 40-69, then at least one free frequency slot (such as reserving at least one free slot in frequency slots 30-39) needs to be reserved between them as a guard band. The specific size of the guard band can be preset according to the optical network specification (such as 1 frequency slot).

[0092] As can be seen from the above description, the cross-data center computing and optical network joint scheduling method for hybrid parallel training provided in this application obtains the computing and optical network resource status in real time and quantifies it into clear constraints, ensuring the feasibility and accuracy of subsequent optimization decisions, avoiding resource over-allocation and allocation conflicts, and improving the reliability and resource utilization of the scheduling scheme.

[0093] To further address the lack of precise quantification methods for computation time and three types of communication in hybrid parallel training, which leads to a lack of scientific quantitative basis for scheduling decisions and makes it difficult to achieve refined resource allocation, this application provides a cross-data center computing and optical network joint scheduling method for hybrid parallel training. The time model formula includes: a formula for total computation time, a formula for data parallel communication, a formula for tensor parallel communication, and a formula for pipelined parallel communication. (See also...) Figure 4 Step 100 in the cross-data center computing and optical network joint scheduling method for hybrid parallel training specifically includes the following: Step 110: Based on the computational amount of each computational layer in the model structure and the pipeline stage division scheme, determine the total computational amount of each pipeline stage based on the total computational amount formula; and substitute the total computational amount, the tensor parallelism in the hybrid parallel configuration, and the preset single GPU computing power into the computational time formula to determine the computational time of each pipeline stage on a single microbatch.

[0094] In this context, single-GPU computing power refers to the effective floating-point operation speed that each graphics processing unit (GPU) can achieve in actual training, usually measured in TFLOPS (trillion floating-point operations per second). For example, it could be 312 TFLOPS.

[0095] In the entire hybrid parallel training process, TP computation is the only computational operation that requires detailed modeling. PP and DP are communication-driven mechanisms and do not directly participate in the definition of computational load. For the... Each pipeline stage (including the set of computational layers contained in the s-th pipeline stage) (i.e., the pipeline stage partitioning scheme) using tensor parallelism. Then, the formula for calculating the total amount is: (1) in, This represents the total computation of the s-th pipeline stage, which is also the amount of computation undertaken by the s-th pipeline stage on each GPU. Indicates the first l The total computational cost required for each computational layer to complete forward and backward propagation on a microbatch, i.e., the computational cost of each computational layer; For tensor parallelism.

[0096] Let the first Each stage of the pipeline is deployed in the data center. The number of GPUs available in the data center to execute this pipeline stage is [number missing]. This refers to the computing resources allocated to each stage of the pipeline, with the effective computing power of each GPU (i.e., the computing power of a single GPU) being... (Unit: FLOPs / s), taking into account factors such as GPU peak computing power, utilization, and TP scheduling overhead, the calculation time formula is as follows: (2) in, This represents the computation time (in seconds) of the s-th pipeline stage on a single microbatch.

[0097] Step 120: Determine the total parameter size of the hybrid parallel training task based on the parameter size of each computational layer in the model structure, and determine the amount of data parallel gradient synchronization communication data based on the data parallel communication volume formula according to the data parallelism in the hybrid parallel configuration.

[0098] Specifically, the system calculates the total parameter size of the entire model based on the parameter size (in bytes) of each computational layer in the model structure. Assuming the parameter size of each layer is known (e.g., approximately 0.5GB per layer), the total parameter size for 24 layers is 24 × 0.5GB = 12GB. Then, based on the data parallelism degree D = 2, the system substitutes this into the data parallel communication volume formula to determine the total amount of gradient data that needs to be synchronized after each iteration. That is, the data parallel communication volume formula is: (3) in, This represents the total parameter size of the entire model (in bytes), i.e., the total parameter size of the hybrid parallel training task; It is the first l The parameter size of each computational layer It represents the total number of layers in the model.

[0099] In DP parallel mode, maintain Each model has multiple copies, and after each iteration, the parameter gradients need to be synchronized to ensure consistency across copies. The amount of data for global gradient synchronization is approximately [amount missing]. The global DP AllReduce communication time is given by formula (4): (4) in, It is the data parallel gradient synchronization communication time (unit: seconds), i.e., the data parallel gradient synchronization communication time; It refers to data parallelism, i.e., the number of model replicas. It is the average propagation delay between data centers. Indicates the total number of parameters in the model; This represents the optical network bandwidth between data center u and data center v.

[0100] Step 130: Based on the tensor parallel communication data volume of each computational layer in the model structure and the pipeline stage division scheme, determine the tensor parallel communication data volume within each pipeline stage based on the tensor parallel communication volume formula.

[0101] Specifically, the system calculates the amount of tensor parallel communication data (in bytes) within each computational layer of the model structure, combined with the pipeline stage partitioning scheme. The amount of tensor parallel communication data refers to the total amount of data that the layer needs to transmit for tensor parallel collective communication (such as AllGather and ReduceScatter) within a microbatch.

[0102] For stage 0 (including layers 1-5), assuming the tensor parallel communication data size for each layer is 0.1 GB, the tensor parallel communication data size for stage 0 is 5 × 0.1 GB = 0.5 GB. The tensor parallel communication data size for other stages can be calculated similarly, where the tensor parallel communication data size for the s-th pipeline stage is denoted as... That is, the total amount of data (in bytes) of collective communication among all layers TP in a microbatch of the s-th pipeline stage.

[0103] In TP parallel mode, after completing a portion of matrix multiplication, each layer needs to perform collective communication such as AllGather / ReduceScatter before proceeding to the next computation. These TP communications and their corresponding computational operations have a strict sequential dependency and cannot overlap with the computation of the current layer. The internal network bandwidth of data center u (implicitly contained in computing resources) is... At that time, its TP communication time can be abstracted as: (5) in, This represents the total amount of TP collective communication data (in bytes) across all layers within a microbatch in the s-th pipeline stage, i.e., the amount of tensor parallel communication data within each pipeline stage. This represents the tensor parallel communication time (in seconds) of the s-th pipeline stage.

[0104] And, step 140: Based on the activation size of each computational layer in the model structure and the pipeline stage division scheme, determine the pipeline parallel communication data volume between adjacent pipeline stages based on the pipeline parallel communication volume formula.

[0105] Specifically, the system determines the amount of pipeline parallel communication data at the boundary between adjacent stages based on the activation size (in bytes) of each computational layer in the model structure and the pipeline stage partitioning scheme. For pipeline stage 0 and pipeline stage 1, the boundary lies between layers 5 and 6. During forward propagation, pipeline stage 0 needs to pass the output activation of layer 5 to pipeline stage 1; this activation size is A5. During backward propagation, pipeline stage 1 needs to pass the gradient back to pipeline stage 0; the data size is also A5. Therefore, the total data size for pipeline parallel communication 4 between pipeline stage 0 and pipeline stage 1 is A5 bytes.

[0106] Assuming the activation size of layer 5 is 0.2 GB, then the PP communication data volume between pipeline stage 0 and pipeline stage 1 is A0 = 0.2 GB. The communication data volume between other adjacent stages can be calculated similarly.

[0107] In PP parallel mode, during forward propagation, the s-th pipeline stage will activate and send the s-th stage to the s-th stage. The pipeline stage; in backpropagation, the gradient changes from the first... Each pipeline stage sends data back to the s-th pipeline stage. For each microbatch, the data size crossing this boundary is... However, its transmission time depends on whether the boundary is within a data center or across data centers: (6) in, This represents the pipeline parallel communication time (in seconds) between the s-th pipeline stage and the next pipeline stage. This represents the activation size (in bytes) from the output of the s-th stage to the next stage, which is the amount of pipeline parallel communication data between adjacent pipeline stages. This indicates the internal network bandwidth of the data center (bytes per second). This represents the optical network bandwidth (bytes / second) allocated between data center u and data center v for this task, i.e., the optical network bandwidth resources allocated for cross-data center communication; This represents the optical network propagation delay (in seconds) between data center u and v.

[0108] As can be seen from the above description, the cross-data center computing and optical network joint scheduling method for hybrid parallel training provided in this application, by decomposing the time model into specific calculation formulas, achieves accurate quantification of computing time and three types of communication traffic, providing accurate input parameters for subsequent joint optimization, and can effectively improve the scientificity and effectiveness of scheduling decisions.

[0109] To further address the problem that existing technologies lack a unified coordination mechanism in scenarios with multiple concurrent training tasks, easily leading to idle computing resources in some data centers while cross-data center links become congested, resulting in low overall resource utilization efficiency, this application provides a cross-data center computing and optical network joint scheduling method for hybrid parallel training. In this method, the hybrid parallel training task includes multiple concurrently executed training task replicas; see also... Figure 4 Step 200 of the cross-data center computing and optical network joint scheduling method for hybrid parallel training specifically includes the following: Step 210: Based on the computation time, tensor parallel communication data volume, pipeline parallel communication data volume, and data parallel gradient synchronization communication data volume corresponding to each of the training task replicas, construct the objective function for each of the training task replicas based on a preset single-iteration total time function form; wherein, the objective function is used to represent the data center deployment location of the pipeline stage of a training task replica, the computational resources allocated to each pipeline stage of the training task replica, and the optical network path and bandwidth resources allocated for cross-data center communication of the training task replica.

[0110] Specifically, one embodiment of step 210 includes: for each pipeline stage in each of the training task replicas, establishing a first functional relationship between the computation execution time of the pipeline stage and the computational resources allocated to the pipeline stage based on the computation time corresponding to the pipeline stage; for each pipeline stage in each of the training task replicas, determining the tensor parallel communication time of the pipeline stage based on the tensor parallel communication data volume corresponding to the pipeline stage; for each pair of adjacent pipeline stages in each of the training task replicas, establishing a pipeline parallel communication time relationship between adjacent pipeline stages and the pipeline parallel communication time between adjacent pipeline stages based on the pipeline parallel communication data volume between adjacent pipeline stages. The system establishes a second functional relationship between the data center deployment location of each training task replica and the optical network path and bandwidth resources allocated for cross-data center communication of that training task replica, based on the amount of data parallel gradient synchronization communication data corresponding to that training task replica; and for each training task replica, it establishes a third functional relationship between the data parallel gradient synchronization communication time of that training task replica and the optical network path and bandwidth resources allocated for cross-data center communication of that training task replica, based on the first functional relationship, the second functional relationship, the third functional relationship, and the tensor parallel communication time, combined with the preset pipeline parallelism of the training task replica, to construct the objective function of that training task replica.

[0111] Among them, the first functional relationship, the second functional relationship, and the third functional relationship are respectively formula (2), formula (6) and formula (4).

[0112] It should be noted that a training task replica refers to an independent instance of a large model training task. In this embodiment, task A and task B are two concurrently executed replicas. Decision variables are parameters that can be adjusted during the optimization process, including deployment location, computing resource allocation, and bandwidth allocation.

[0113] Specifically, for each training task replica, the system constructs its objective function based on a preset single-iteration total time function. This function is approximated by formula (7): (7) in, The value represents the total time of a single training iteration (in seconds), i.e., the total time of a single iteration; m represents the number of micro-batches, i.e., the number of micro-batches in each global batch. Indicates the degree of parallelism in the pipeline; This represents the longest execution time among all pipeline stages. This reflects that the slowest stage of the production line determines the production line cycle time; coefficient This characterizes the bubble overhead generated by the pipeline in each iteration, which increases with the number of micro-batches. As the number of steps increases, the coefficient gradually approaches 1; the execution time of each pipeline stage. From computation time and TP communication time Composition of PP communication time That is, formula (8): (8) in, This represents the total execution time (in seconds) of the s-th pipeline stage on a single microbatch, including computation and communication. As determined in step 100, its relationship with the allocated computing resources is given by formula (2). Tensor parallel communication time Calculated using formula (5); Calculated using formula (6); It is calculated using formula (4).

[0114] If the training task needs to be executed The total training time for the entire training task is then... for: (9) In this application, the number of training iterations Generally, the task objectives and optimization strategies are predetermined, while This is the key area of ​​optimization in this application: it is task partitioning. The deployment scheme from the pipeline stage to the data center and the joint function of optical network bandwidth / spectrum resource allocation. Different partitioning and data center placement will change whether the data is transmitted within the data center (intra-DC) or across data centers (inter-DC), and change the amount of computation and available computing power carried by each pipeline stage, resulting in significantly different iteration time and total training time.

[0115] Building upon the aforementioned computation-communication time model for cross-data center hybrid parallel training, this application further proposes a joint optimization framework for cross-data center computation and optical networks aimed at minimizing training iteration time, such as... Figure 5 As shown, its core idea is to explicitly model the computational load, inter-stage communication, and cross-datacenter communication introduced by different parallel dimensions in hybrid parallel training, and to integrate them into the same time cost function for joint decision-making. Figure 5In this system, multiple Large Language Model (LLM) training tasks (Task A, Task B, and Task C) are used as inputs. Each task is configured with a hybrid parallelism approach, including tensor parallelism (TP), data parallelism (DP), and pipelined parallelism (PP). Upon receiving a task request, the core scheduling module first divides each task model into multiple pipeline stages using the training subtask partitioning unit. Then, through the joint computing and network resource allocation unit, it performs joint optimization decisions on computing and optical network resources, combining cross-data center (Inter-DC) computing and communication dependency configurations with resource status information provided by the intra-data center (Intra-DC) storage modules. The scheduling results are distributed to cross-data center and intra-data center environments for execution. Simultaneously, the monitoring and feedback unit monitors the execution status in real time and sends feedback information back to the core scheduling module for online adjustment of the optimization scheme.

[0116] Step 220: The objective is to minimize the weighted sum of the total time of a single iteration for each of the training task replicas, wherein the weights of the weighted sum are determined based on the preset priority or preset resource requirements of each of the training task replicas.

[0117] Among them, cross-DC optical networks often simultaneously carry multiple large model training tasks. Different tasks differ in model size, hybrid parallel configuration, and priority, but share the same set of cross-DC optical links and intra-DC GPU resources. Therefore, it is necessary to jointly optimize the computation and communication resources of multiple tasks under a unified system model to simultaneously improve the overall training throughput and increase the utilization rate of optical network and computing resources. This application aims to minimize the iteration time of the total tasks, as shown in formula (10): (10) in, Indicates the training task index; This represents the set of all hybrid parallel training tasks to be scheduled; This represents the total time of a single iteration of task r, i.e., the total time of a single iteration.

[0118] The weights are determined based on the preset priorities of each task. For example, task A has a high priority and a weight of 2, while task B has a low priority and a weight of 1. This weighted sum reflects the system's emphasis on the training efficiency of different tasks. Tasks with higher priority contribute more to the overall goal and are thus given priority in resource competition.

[0119] Step 230: Under the constraints of computing resources and optical network resources, perform unified optimization on the data center deployment location, computing resource allocation, and optical network path and bandwidth resources for cross-data center communication in the pipeline stage of each training task replica, so as to determine the joint optimization decision result corresponding to each training task replica.

[0120] Specifically, to ensure the feasibility of the scheduling results, the joint optimization model must satisfy the following constraints.

[0121] First, at the computing resource level, each training subtask can only be deployed in one data center at any given time, as per formula (11), and the number of GPUs and storage resources allocated to all training tasks in each data center must not exceed its physical capacity limit, as per formulas (12) and (13): (11) (12) (13) in, This is a binary variable. If the pipeline stage s of task r is deployed in data center u, it takes the value 1; otherwise, it takes the value 0. This is the decision variable for the data center deployment location of each pipeline stage. Represents a collection of data centers; This represents the set of pipeline stages for task r; This represents the number of GPUs required for each pipeline stage s of task r, i.e., the number of GPUs in the computing resources allocated to each pipeline stage. This represents the total number of available GPUs in data center u, i.e., the number of available GPUs in each data center. This is a binary variable indicating whether task r has deployed any pipeline stages in data center u (1 for yes, 0 for no). Indicates task r Total storage requirements (in bytes), which is the sum of storage resources allocated to each stage of the pipeline; For data centers u The available storage capacity refers to the available storage capacity of each data center.

[0122] Secondly, at the optical network level, cross-data center communication requires establishing end-to-end optical channels along available links in the optical network, and the total amount of spectrum resources allocated on each optical link must not exceed its capacity limit: (14) in, This is a binary variable indicating whether task r occupies the optical link. The f-th frequency slot, i.e., if task r occupies the link The f If there are one frequency slot, the value is 1; otherwise, it is 0. This represents the total bandwidth capacity of each optical link, i.e., the available bandwidth capacity of each link or the total bandwidth capacity. This represents a set of optical links.

[0123] Meanwhile, the spectrum allocation for different training tasks on the same optical link must meet the constraints of continuity, consistency, and guard band isolation to avoid spectrum conflicts and ensure stable establishment of the optical channel.

[0124] Under the constraints of the above objective function, the joint optimization decision simultaneously determines: (1) Data center deployment location for each pipeline stage; (2) The number of GPUs and computing resources that can be allocated to each training subtask; (3) Optical network path and effective bandwidth allocation used in cross-data center PP / DP communication.

[0125] This problem is formally presented as a coupled optimization problem of computational placement and optical network resource allocation, in which the decision on either side directly affects the objective function value through the time model.

[0126] As can be seen from the above description, the cross-data center computing and optical network joint scheduling method for hybrid parallel training provided in this application aims to minimize the weighted sum of the total time of a single iteration of multiple tasks and performs unified optimization under shared resource constraints. This enables the coordinated allocation of computing and optical network resources among multiple tasks, effectively alleviates resource competition, and improves overall training throughput and resource utilization.

[0127] To further address the problem that existing static scheduling schemes cannot adapt to dynamic environmental factors such as fluctuations in computational performance and changes in network bandwidth during training, leading to deviations in actual training efficiency from expectations, this application provides a cross-data center computing and optical network joint scheduling method for hybrid parallel training, see [link to relevant documentation] Figure 4 The method for joint scheduling of cross-data center computing and optical networks for hybrid parallel training further includes the following content after step 300: Step 400: During the training process of the model corresponding to the hybrid parallel training task, monitor the actual execution time of each pipeline stage and the cross-data center communication status in real time.

[0128] Step 500: If the deviation between the currently monitored actual execution time and the calculation time exceeds a preset threshold, and / or if the currently monitored cross-data center communication status meets a preset trigger condition, then the calculation time is corrected according to the actual execution time, the availability information of the optical network resources is updated according to the cross-data center communication status, and based on the corrected calculation time and the updated availability information of the optical network resources, at least one of the following is adjusted: the data center deployment location of the currently executing pipeline stage, the computing resources allocated to each pipeline stage, and the optical network path and bandwidth resources allocated for cross-data center communication, so as to redetermine the corresponding joint optimization decision result.

[0129] Step 600: Based on the redefined joint optimization decision results, adjust at least one of the following: the data center deployment location of the pipeline stage of the currently executing hybrid parallel training task, the computing resources allocated to each pipeline stage, and the optical network path and bandwidth resources allocated for cross-data center communication, and continue to execute the training of the model corresponding to the hybrid parallel training task.

[0130] Specifically, during the training process, the system continuously monitors: (1) The deviation between the actual stage execution time and the model prediction time; (2) Fluctuations in optical network link bandwidth and changes in latency; (3) The actual proportion of pipeline bubbles.

[0131] When the deviation exceeds a preset threshold, the time model parameters are corrected, and online adjustments are made to stage placement or cross-DC bandwidth allocation to maintain the stability of training efficiency. In one implementation, the cross-data center communication status in step 410 meeting the preset triggering conditions specifically includes: (1) The actual bandwidth of cross-data center communication is lower than the preset bandwidth threshold (e.g., lower than 50% of the allocated bandwidth or lower than 1 Gbps). (2) The communication delay exceeds the preset delay threshold (e.g., more than 10 ms); (3) Detection of optical network link congestion (e.g., the interface packet loss rate exceeds 1% or the frequency slot occupancy rate exceeds 90% as monitored by SNMP).

[0132] The above conditions can be flexibly set according to the actual network environment and task requirements.

[0133] As can be seen from the above description, the cross-data center computing and optical network joint scheduling method for hybrid parallel training provided in this application can correct model parameters and re-optimize resource allocation according to the actual execution situation through real-time monitoring and dynamic adjustment mechanisms, so that the scheduling scheme can adapt to environmental changes and always maintain high training efficiency.

[0134] From a software perspective, this application also provides a cross-datacenter computing and optical network joint scheduling system for executing all or part of the cross-datacenter computing and optical network joint scheduling method for hybrid parallel training, see [link to relevant documentation]. Figure 6 The cross-data center computing and optical network joint scheduling system for hybrid parallel training specifically includes the following components: The time model parameter calculation module 10 is used to determine, based on the model structure, hybrid parallel configuration, and pipeline stage division scheme of the hybrid parallel training task to be scheduled, the amount of data parallel gradient synchronization communication corresponding to the hybrid parallel training task, the computation time of each pipeline stage on a single microbatch, the amount of tensor parallel communication data within each pipeline stage, and the amount of pipeline parallel communication data between adjacent pipeline stages, using a preset time model formula.

[0135] The joint optimization decision module 20 is used to determine the joint optimization decision result corresponding to the hybrid parallel training task based on the computation time, the tensor parallel communication data volume, the pipeline parallel communication data volume, and the data parallel gradient synchronization communication data volume, with the goal of minimizing the total time of a single iteration of the hybrid parallel training task, under the condition of satisfying the pre-acquired computational resource constraints and optical network resource constraints. The joint optimization decision result includes: the data center deployment location of each pipeline stage, the computational resources allocated to each pipeline stage, and the optical network path and bandwidth resources allocated for cross-data center communication.

[0136] The deployment execution module 30 is used to deploy corresponding pipeline stages and allocate computing resources in each data center based on the joint optimization decision results, and to establish corresponding optical channels for cross-data center communication in the optical network.

[0137] The embodiments of the cross-data center computing and optical network joint scheduling system for hybrid parallel training provided in this application can be used to execute the processing flow of the embodiments of the cross-data center computing and optical network joint scheduling method for hybrid parallel training in the above embodiments. Its functions will not be repeated here, but can be referred to the detailed description of the embodiments of the cross-data center computing and optical network joint scheduling method for hybrid parallel training.

[0138] The cross-datacenter computing and optical network joint scheduling system for hybrid parallel training described herein can perform the joint scheduling of cross-datacenter computing and optical networks for hybrid parallel training either on a server or in a client device. The specific choice depends on the processing power of the client device and the limitations of the user's usage scenario. This application does not impose any limitations in this regard. If all operations are performed in the client device, the client device may further include a processor for specific processing of the cross-datacenter computing and optical network joint scheduling for hybrid parallel training.

[0139] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed system server structure.

[0140] The server and the client device can communicate using any suitable network protocol, including those not yet developed as of the date of this application. Such network protocols may include, for example, TCP / IP, UDP / IP, HTTP, HTTPS, etc. Furthermore, such network protocols may also include RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer Protocol) protocols used on top of the aforementioned protocols.

[0141] As described above, the cross-data center computing and optical network joint scheduling system for hybrid parallel training provided in this application accurately models the computation time of hybrid parallel training tasks and three types of communication data volumes (tensor parallelism, pipeline parallelism, and data parallelism) by incorporating them into a preset time model formula. With the goal of minimizing the total time of a single iteration, it jointly optimizes the data center deployment location in the pipeline stage, the allocation of computing resources at each stage, and the optical network path and bandwidth resources for cross-data center communication under constraints of computing and optical network resources. Finally, based on the optimization results, it performs actual deployment and establishes optical channels. This fundamentally solves the problem of independent and uncoordinated computing scheduling and optical network resource allocation in existing technologies. This technical solution achieves deep coupling optimization of computing and communication, accurately perceives the communication needs of different parallel modes, rationally allocates computing and network resources across data centers, effectively reduces training latency introduced by cross-data center communication, avoids pipeline bubbles and resource idleness, and significantly improves overall training efficiency and system resource utilization in multi-task concurrent scenarios.

[0142] To further illustrate the above embodiments, this application also provides a specific application example of a cross-data center computing and optical network joint scheduling method for hybrid parallel training executed by a cross-data center computing and optical network joint scheduling system for hybrid parallel training. See [link to relevant documentation]. Figure 7 This application example illustrates a cross-datacenter hybrid parallel training scenario in which multiple training tasks are executed concurrently.

[0143] In this application example, the system simultaneously receives multiple large model training task requests, denoted as . These different training tasks differ in model size, number of Transformer layers, hybrid parallelism configuration parameters (data parallelism, pipeline parallelism, tensor parallelism), number of micro-batches, and number of training iterations, but share the same cross-datacenter computing resource pool and optical network resource pool. During execution, each training task will generate cross-datacenter communication needs of varying scale and intensity.

[0144] The system first integrates multiple training tasks through a training request access module, and then parses the model structure information, parallel configuration parameters, and training objectives for each training task. Unlike existing scheduling methods that only identify resource requirements, this application instance instantiates the corresponding computation-communication time model parameters based on the model structure information during the task access phase. This includes the computational load, activation scale, parameter scale of each Transformer layer, and inter-stage communication requirements introduced by the hybrid parallel approach, providing a quantifiable time cost basis for subsequent joint scheduling.

[0145] Subsequently, the system performs model partitioning for each training task. Specifically, based on the pipeline parallelism of the training task, the system divides the model into multiple pipeline stages. Each pipeline stage corresponds to a set of consecutive Transformer layers, forming a corresponding training subtask. Unlike fixed-rule partitioning, this application example comprehensively considers the computation time of each pipeline stage, the tensor parallel communication time within each stage, and the potential cross-datacenter communication time during the partitioning process. This ensures that the execution time of different pipeline stages is as balanced as possible, thereby reducing the risk of the pipeline tick being dominated by a single stage.

[0146] Based on this, the system acquires the computing resource status of multiple data centers, as well as the topology, available bandwidth, and propagation latency information of cross-data center optical networks through a resource awareness module. In scenarios with multiple concurrent training tasks, the system not only perceives the resource requirements of a single training task but also continuously maintains the global resource occupancy status, characterizing the competitive relationship between different training tasks on computing and optical network resources over time.

[0147] During the joint scheduling phase, the system aims to minimize the overall training time of multiple training tasks. Based on the computation-communication time model of each training task, it makes unified computation and optical network joint scheduling decisions for multiple training tasks. Specifically, when selecting data centers for deployment of each training sub-task, the system considers not only the communication dependency strength within a single training task but also comprehensively evaluates resource conflicts between different training tasks, avoiding multiple communication-intensive training tasks from simultaneously occupying the same cross-data center link or the same data center's computational resource bottleneck. Simultaneously, the system allocates corresponding optical network communication paths and bandwidth resources to training sub-tasks with cross-data center communication dependencies, ensuring that inter-stage communication and gradient synchronization communication can still meet the bandwidth and latency constraints required by the time model while sharing optical network resources.

[0148] During training, the system continuously monitors the pipeline stage execution time of each training task, cross-data center communication latency, and optical network link load. When a significant deviation is detected between the actual execution time and the time model prediction, or when the iteration time of some training tasks increases significantly due to competition among multiple training tasks, the system dynamically adjusts the joint scheduling decision based on real-time monitoring results. This includes reallocating some training sub-tasks to different data centers or adjusting the bandwidth allocation scheme of the cross-data center optical network to maintain the stable operation of the overall training system and high resource utilization efficiency.

[0149] Through the above implementation methods, this application can uniformly model the time characteristics of computation and communication during hybrid parallel training in cross-data center scenarios where multiple training tasks are executed concurrently, and thereby realize the coordinated scheduling of computing resources and optical network resources, effectively reducing the negative impact of cross-data center communication on training efficiency and improving the overall training performance and resource utilization efficiency of the system.

[0150] In other words, this application provides a hybrid parallel training structure that divides large model training tasks into pipeline-stage training subtasks and explicitly constructs the computational requirements and communication dependencies between training subtasks. It also provides a computation-communication time model based on cross-data center hybrid parallel training to evaluate and optimize different training subtask partitioning schemes, deployment schemes, and optical network resource allocation schemes. In a cross-data center environment, it also integrates the computational resource deployment decisions of training subtasks with the path selection and bandwidth allocation of cross-data center optical networks into a joint scheduling process to avoid performance bottlenecks caused by the independence of computational scheduling and communication resource allocation.

[0151] Based on this, compared with existing technologies, the cross-data center computing and optical network joint scheduling method for hybrid parallel training provided in this application example has the following beneficial effects: (1) To address the long-standing problem of the separation between computation scheduling and communication resource allocation in cross-data center hybrid parallel training scenarios, a computation-optical network joint scheduling mechanism for hybrid parallel training is introduced. This mechanism integrates the computation deployment decision of training subtasks with the cross-data center optical network path and bandwidth allocation into the same scheduling process, thereby avoiding performance bottlenecks caused by local optimization based solely on computing power or solely on the network at the system level.

[0152] (2) By explicitly constructing the communication dependencies between training subtasks and distinguishing between tensor parallel communication, pipeline inter-stage communication and data parallel gradient synchronization communication, the scheduling process can accurately perceive the sensitivity of different communication types to latency and bandwidth, thereby more reasonably determining the deployment location of training subtasks and their corresponding communication paths in cross-data center environments, and significantly reducing the training iteration overhead introduced by cross-data center communication latency amplification.

[0153] (3) In the joint scheduling process, the perception and modeling of cross-data center optical network link bandwidth and propagation delay are introduced, so that optical network resources are no longer passively carried, but are key factors for collaborative optimization with computing resources. This effectively alleviates link congestion and improves the overall utilization efficiency of optical network resources when multiple training tasks are executed concurrently, avoiding resource waste caused by uneven bandwidth allocation or improper path selection.

[0154] (4) By introducing a time-based dynamic feedback and adjustment mechanism during the training execution phase, the system can dynamically optimize the deployment scheme of training subtasks and the allocation scheme of communication resources according to changes in training progress and fluctuations in resource status. This enables the system to maintain stable operation and high training efficiency in scenarios with multiple concurrent training tasks and resource competition, and has good scalability and applicability.

[0155] This application also provides an electronic device, which may include a processor, a memory, a receiver, and a transmitter. The processor is used to execute the cross-data center computing and optical network joint scheduling method for hybrid parallel training mentioned in the above embodiments. The processor and memory can be connected via a bus or other means, taking a bus connection as an example. The receiver can be connected to the processor and memory via wired or wireless means.

[0156] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0157] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the cross-data center computing and optical network joint scheduling method for hybrid parallel training in the embodiments of this application. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the cross-data center computing and optical network joint scheduling method for hybrid parallel training in the above method embodiments.

[0158] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0159] The one or more modules are stored in the memory, and when executed by the processor, the cross-data center computing and optical network joint scheduling method for hybrid parallel training in the embodiment is executed.

[0160] In some embodiments of this application, the user equipment may include a processor, a memory, and a transceiver unit. The transceiver unit may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected via a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0161] As one implementation method, the functions of the receiver and transmitter in this application can be implemented by transceiver circuits or dedicated transceiver chips, and the processor can be implemented by dedicated processing chips, processing circuits or general-purpose chips.

[0162] As another implementation approach, the server provided in this application embodiment can be implemented using a general-purpose computer. That is, the program code implementing the processor, receiver, and transmitter functions is stored in memory, and the general-purpose processor implements the processor, receiver, and transmitter functions by executing the code in memory.

[0163] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned method for joint scheduling of cross-data center computing and optical networks for hybrid parallel training. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0164] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the aforementioned method for joint scheduling of cross-data center computing and optical networks for hybrid parallel training.

[0165] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave.

[0166] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0167] In this application, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0168] The above description is merely a preferred embodiment of this application and is not intended to limit this application. For those skilled in the art, various modifications and variations can be made to the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A cross-data center computing and optical network joint scheduling method for hybrid parallel training, characterized in that, include: Based on the model structure, hybrid parallel configuration, and pipeline stage division scheme of the hybrid parallel training task to be scheduled, the amount of data parallel gradient synchronization communication data, the computation time of each pipeline stage on a single microbatch, the amount of tensor parallel communication data within each pipeline stage, and the amount of pipeline parallel communication data between adjacent pipeline stages are determined using a preset time model formula. Based on the computation time, tensor parallel communication data volume, pipeline parallel communication data volume, and data parallel gradient synchronization communication data volume, with the goal of minimizing the total time of a single iteration of the hybrid parallel training task, and under the condition of satisfying the pre-acquired computational resource constraints and optical network resource constraints, the joint optimization decision result corresponding to the hybrid parallel training task is determined. The joint optimization decision result includes: the data center deployment location of each pipeline stage, the computational resources allocated to each pipeline stage, and the optical network path and bandwidth resources allocated for cross-data center communication. Based on the joint optimization decision results, corresponding pipeline stages are deployed and computing resources are allocated in each data center, and corresponding optical channels are established in the optical network for cross-data center communication.

2. The cross-data center computing and optical network joint scheduling method for hybrid parallel training according to claim 1, characterized in that, The hybrid parallel training task includes multiple concurrently executed copies of the training task; Correspondingly, based on the computation time, the amount of tensor parallel communication data, the amount of pipelined parallel communication data, and the amount of data parallel gradient synchronization communication data, with the objective of minimizing the total time of a single iteration of the hybrid parallel training task, and under the condition of satisfying the pre-acquired computational resource constraints and optical network resource constraints, the joint optimization decision result corresponding to the hybrid parallel training task is determined, including: Based on the computation time, tensor parallel communication data volume, pipeline parallel communication data volume, and data parallel gradient synchronization communication data volume corresponding to each of the training task replicas, an objective function is constructed for each of the training task replicas based on a preset single-iteration total time function form; wherein, the objective function is used to represent the data center deployment location of the pipeline stage of a training task replica, the computational resources allocated to each pipeline stage of the training task replica, and the optical network path and bandwidth resources allocated for cross-data center communication of the training task replica; The objective is to minimize the weighted sum of the total time of a single iteration for each of the training task replicas, wherein the weights of the weighted sum are determined based on the preset priority or preset resource requirements of each of the training task replicas. Under the constraints of computing resources and optical network resources, the data center deployment location, computing resource allocation, and optical network path and bandwidth resources for cross-data center communication in the pipeline stage of each training task replica are uniformly optimized to determine the joint optimization decision result corresponding to each training task replica.

3. The cross-data center computing and optical network joint scheduling method for hybrid parallel training according to claim 2, characterized in that, The step of constructing the objective function for each training task replica based on the computation time, tensor parallel communication data volume, pipeline parallel communication data volume, and data parallel gradient synchronization communication data volume corresponding to each training task replica, and based on a preset single-iteration total time function form, includes: For each pipeline stage in each of the training task replicas, a first functional relationship is established between the computation execution time of the pipeline stage and the computation resources allocated to the pipeline stage, based on the computation time corresponding to the pipeline stage. For each pipeline stage in each of the training task replicas, the tensor parallel communication time of that pipeline stage is determined based on the amount of tensor parallel communication data corresponding to that pipeline stage. For each pair of adjacent pipeline stages in each of the training task replicas, a second functional relationship is established between the pipeline parallel communication time between adjacent pipeline stages and the data center deployment location of the adjacent pipeline stages, as well as the optical network path and bandwidth resources allocated for cross-data center communication of the training task replicas, based on the amount of pipeline parallel communication data between adjacent pipeline stages. Furthermore, for each of the training task replicas, based on the amount of data parallel gradient synchronization communication data corresponding to the training task replica, a third functional relationship is established between the data parallel gradient synchronization communication time of the training task replica and the optical network path and bandwidth resources allocated for the cross-data center communication of the training task replica. For each training task replica, the objective function of the training task replica is constructed based on the first functional relationship, the second functional relationship, the third functional relationship, and the tensor parallel communication time, combined with the preset pipeline parallelism of the training task replica.

4. The cross-data center computing and optical network joint scheduling method for hybrid parallel training according to claim 1, characterized in that, Also includes: During the training process of the model corresponding to the hybrid parallel training task, the actual execution time of each pipeline stage and the cross-data center communication status are monitored in real time. If the deviation between the currently monitored actual execution time and the calculation time exceeds a preset threshold, and / or if the currently monitored cross-data center communication status meets a preset trigger condition, then the calculation time is corrected according to the actual execution time, the availability information of the optical network resources is updated according to the cross-data center communication status, and based on the corrected calculation time and the updated availability information of the optical network resources, at least one of the following is adjusted: the data center deployment location of the currently executing pipeline stage, the computing resources allocated to each pipeline stage, and the optical network path and bandwidth resources allocated for cross-data center communication, so as to redetermine the corresponding joint optimization decision result; Based on the redefined joint optimization decision, at least one of the following is adjusted: the data center deployment location of the pipeline stage of the currently executing hybrid parallel training task, the computing resources allocated to each pipeline stage, and the optical network path and bandwidth resources allocated for cross-data center communication. Training of the model corresponding to the hybrid parallel training task continues.

5. The cross-data center computing and optical network joint scheduling method for hybrid parallel training according to claim 1, characterized in that, Before determining the amount of data-parallel gradient synchronization communication data corresponding to the hybrid parallel training task, the computation time of each pipeline stage on a single microbatch, the amount of tensor-parallel communication data within each pipeline stage, and the amount of pipeline-parallel communication data between adjacent pipeline stages, the method further includes: Obtain a training request for a hybrid parallel training task for a model to be scheduled; wherein the training request includes the model structure and hybrid parallel configuration of the hybrid parallel training task; With the goal of minimizing the maximum computation time of each pipeline stage on a single microbatch, the model is divided into multiple consecutive pipeline stages based on the pipeline parallelism in the hybrid parallel configuration, to obtain the pipeline stage partitioning scheme.

6. The cross-data center computing and optical network joint scheduling method for hybrid parallel training according to claim 1, characterized in that, Before determining the joint optimization decision result corresponding to the hybrid parallel training task, the method further includes: The system acquires the current computing resource status and optical network resource status across data center environments. The computing resource status includes the number of available GPUs and available storage capacity in each data center. The optical network resource status includes the available bandwidth capacity of each link. The computing resource constraints are determined based on the computing resource status, wherein the computing resource constraints include: the total number of GPUs allocated to all scheduled hybrid parallel training tasks in each data center does not exceed the number of available GPUs in that data center, and the storage resources allocated to each pipeline stage do not exceed the available storage capacity of that data center. Furthermore, optical network resource constraints are determined based on the optical network resource status, wherein the optical network resource constraints include: the frequency slots allocated for cross-data center communication do not exceed the total frequency slot capacity of each link, and the frequency slots occupied by cross-data center communication of the same hybrid parallel training task meet preset spectral continuity constraints, spectral consistency constraints, and guard band isolation constraints.

7. The cross-data center computing and optical network joint scheduling method for hybrid parallel training according to claim 1, characterized in that, The time model formulas include: total computation formula, computation time formula, data parallel communication volume formula, tensor parallel communication volume formula, and pipeline parallel communication volume formula. Correspondingly, the step of determining the amount of data parallel gradient synchronization communication data, the computation time of each pipeline stage on a single microbatch, the amount of tensor parallel communication data within each pipeline stage, and the amount of pipeline parallel communication data between adjacent pipeline stages, based on the model structure, hybrid parallel configuration, and pipeline stage division scheme of the hybrid parallel training task to be scheduled, using a preset time model formula, includes: Based on the computational amount of each computational layer in the model structure and the pipeline stage division scheme, the total computational amount of each pipeline stage is determined based on the total computational amount formula; and the total computational amount, the tensor parallelism in the hybrid parallel configuration, and the preset single GPU computing power are substituted into the computational time formula to determine the computational time of each pipeline stage on a single microbatch. Based on the parameter sizes of each computational layer in the model structure, the total parameter size of the hybrid parallel training task is determined, and based on the data parallelism in the hybrid parallel configuration, the amount of data parallel gradient synchronization communication data is determined according to the data parallel communication volume formula. Based on the tensor parallel communication data volume of each computational layer in the model structure and the pipeline stage division scheme, the tensor parallel communication data volume within each pipeline stage is determined based on the tensor parallel communication volume formula. Furthermore, based on the activation size of each computational layer in the model structure and the pipeline stage partitioning scheme, the pipeline parallel communication data volume between adjacent pipeline stages is determined according to the pipeline parallel communication volume formula.

8. The cross-data center computing and optical network joint scheduling method for hybrid parallel training according to claim 4, characterized in that, The cross-data center communication status meets the preset triggering conditions, including: The actual bandwidth of cross-data center communication is lower than a preset bandwidth threshold, the communication latency exceeds a preset latency threshold, and optical network link congestion is detected, at least one of the following:

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the cross-data center computing and optical network joint scheduling method for hybrid parallel training as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the cross-data center computing and optical network joint scheduling method for hybrid parallel training as described in any one of claims 1 to 8.

Images