AI large model cross-domain hybrid training and hybrid reasoning collaborative scheduling method and system

Abstract

The application provides an AI large model cross-domain hybrid training and hybrid reasoning cooperative scheduling method and system, belonging to the technical field of data processing. The method comprises: constructing a global computing power view and a scheduling control plane between an AI private cloud and one or more public clouds, the global computing power view being used to maintain global computing power resources, the global computing power resources comprising logical computing power units of all GPUs; the scheduling control plane schedules training tasks and reasoning tasks of AI large models based on the following steps: task graph modeling of the training tasks and the reasoning tasks; based on network index data, regional division of task graphs of the training tasks and task graphs of the reasoning tasks, to obtain a plurality of segmented subgraphs; and respectively assigning each segmented subgraph to a different logical computing power unit for execution. The application improves the cooperative scheduling efficiency and resource utilization rate of AI large model training and reasoning tasks in a multi-cloud heterogeneous environment.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and system for collaborative scheduling of cross-domain hybrid training and hybrid inference for large AI models. Background Technology

[0002] In recent years, the scale of large-scale artificial intelligence models has continued to expand, leading to an exponential increase in the demand for computing, storage, and network resources. A single data center or cloud environment often struggles to independently support complete training and inference workloads, especially in scenarios where enterprises need to balance data security (preferring private clouds) and elastic scaling (relying on public clouds). Therefore, collaborative execution of AI (Artificial Intelligence) tasks across multiple cloud environments has become an inevitable trend. Consequently, how to efficiently schedule the training and inference tasks of large models in geographically dispersed and architecturally heterogeneous multi-cloud environments has become a key issue in improving the performance of AI infrastructure.

[0003] Currently, mainstream cross-domain scheduling methods for large models typically treat training and inference as two independent tasks, deployed separately on centralized GPU (Graphics Processing Unit) clusters and edge nodes, and achieve weak coupling through periodic model synchronization. In terms of resource management, each cloud region is often treated as an isolated resource pool, lacking a unified view; task scheduling often employs static rules or simple load balancing strategies, failing to fully consider the actual network conditions of cross-domain links. Furthermore, GPUs from different vendors are often deployed in isolation due to differences in their software stacks, making mixed use difficult. This fragmented scheduling approach leads to low resource utilization and an inability to dynamically optimize deployment locations based on task characteristics and network conditions.

[0004] In summary, achieving efficient collaboration and unified resource scheduling for large model training and inference tasks in a multi-cloud heterogeneous computing environment has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides a method and system for collaborative scheduling of cross-domain hybrid training and hybrid inference for large AI models, in order to address the shortcomings of existing technologies in the lack of efficient collaborative and unified resource scheduling schemes for large model training and inference tasks in multi-cloud heterogeneous computing environments.

[0006] This application provides a method for cross-domain hybrid training and hybrid inference collaborative scheduling of large AI models, including the following steps: Construct a global computing power view and scheduling control plane between an AI private cloud and one or more public clouds. The global computing power view abstracts GPU resources from different cloud regions into logical computing power units. Each logical computing power unit records at least normalized computing power indicators, video memory capacity, region labels, communication capabilities, and software stack compatibility information. Task graphs are modeled for the training task and the inference task respectively, where nodes in the task graph represent independently schedulable computational subtasks, and edges represent data dependencies or control dependencies between subtasks. Based on network metrics such as latency, bandwidth, and packet loss rate of cross-domain links, and combined with constraints on inter-node communication volume, latency sensitivity, and resource compatibility in the task graph, the training task graph and inference task graph are divided into regions to obtain multiple segmented subgraphs, and target region labels are determined for each segmented subgraph. Based on the target region label of each segmented subgraph, logical computing power units that meet resource requirements and compatibility constraints are selected in the corresponding region, and each segmented subgraph is assigned to the selected logical computing power units for execution. During the execution of each segmented subgraph, the mapping relationship between the segmented subgraph and the logical computing unit, as well as the cross-domain communication strategy, are dynamically adjusted based on training throughput, inference latency, resource utilization, and network status. According to the AI ​​large-scale model cross-domain hybrid training and hybrid inference collaborative scheduling method provided in this application, the step of assigning each segmented subgraph to the selected logical computing unit for execution includes: The first type of segmented subgraph is assigned to the logical computing unit on the AI ​​private cloud side for execution; The second type of segmented subgraph is assigned to the logical computing unit on the public cloud side for execution; Specifically, the computational density of the first type of segmented subgraph is greater than that of the second type of segmented subgraph; the communication volume and frequency of the first type of segmented subgraph are less than those of the second type of segmented subgraph; the user relevance of the second type of segmented subgraph is greater than that of the first type of segmented subgraph; and the latency sensitivity of the second type of segmented subgraph is greater than that of the first type of segmented subgraph. The first type of segmented subgraph and the second type of segmented subgraph are determined according to a preset classification rule or classification model.

[0007] According to the AI ​​large-scale model cross-domain hybrid training and hybrid inference collaborative scheduling method provided in this application, the step of assigning each of the segmented subgraphs to the selected logical computing units for execution includes: Under the conditions of satisfying the service level agreement and resource utilization threshold constraints of the inference task, the execution region of each segmented subgraph is determined with training throughput, inference latency and execution cost as joint optimization objectives. For each segmented subgraph, a logical computing unit is selected and executed in the corresponding execution region; Determine the topology and frequency for cross-domain communication.

[0008] According to the AI ​​large-scale model cross-domain hybrid training and hybrid inference collaborative scheduling method provided in this application, the AI ​​large-scale model cross-domain hybrid training and hybrid inference collaborative scheduling method further includes: In the training task, gradients are layered according to importance. The gradients of the first layer are synchronized in each iteration step. The synchronization frequency of the gradient of the i-th layer is greater than that of the gradient of the (i+1)-th layer, where i is a positive integer.

[0009] According to the collaborative scheduling method for cross-domain hybrid training and hybrid inference of large AI models provided in this application, the synchronization frequency and gradient compression ratio of each layer gradient are updated in real time based on the network state.

[0010] According to the AI ​​large-scale model cross-domain hybrid training and hybrid inference collaborative scheduling method provided in this application, the AI ​​large-scale model cross-domain hybrid training and hybrid inference collaborative scheduling method further includes: The traffic from the training task is labeled as a first-class business flow; The traffic from inference tasks is labeled as a second-category business flow; The priority of the second type of service flow is higher than that of the first type of service flow.

[0011] This application also provides a collaborative scheduling system for cross-domain hybrid training and hybrid inference of large AI models, including: The global computing power view building module is used to abstract GPU resources in the AI ​​private cloud and one or more public clouds into logical computing power units, and maintain the region label, computing power, memory capacity, communication capabilities and software stack compatibility information of the logical computing power units. The task graph modeling module is used to model task graphs for training and inference tasks respectively. Nodes in the task graph represent independently schedulable computational subtasks, and edges represent data or control dependencies between subtasks. The region segmentation module is used to divide the task graph into regions based on cross-domain link network indicator data and combined with task graph features, to obtain multiple segmented sub-graphs and determine the target region label of each segmented sub-graph. The scheduling decision module is used to select the execution region and target logical computing unit for each segmented subgraph based on the target region label and the capability information of the logical computing unit. The cross-domain communication control module is used to determine the cross-domain communication topology, synchronization frequency, compression ratio, and service flow priority strategy during the segmented subgraph execution process; The execution feedback and optimization module is used to dynamically adjust scheduling parameters, communication parameters, and task mapping relationships based on training throughput, inference latency, resource utilization, and network status. This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the AI ​​large-scale model cross-domain hybrid training and hybrid inference collaborative scheduling method as described above.

[0012] This application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the AI ​​large model cross-domain hybrid training and hybrid inference collaborative scheduling method as described above.

[0013] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the AI ​​large model cross-domain hybrid training and hybrid inference collaborative scheduling method as described above.

[0014] The AI ​​large-scale model cross-domain hybrid training and hybrid inference collaborative scheduling method and system provided in this application breaks down resource barriers between different cloud environments and hardware platforms by establishing a global computing power view and abstracting all GPUs as logical computing power units. The scheduling control plane can allocate computing power in a unified resource pool, avoiding the problem of local overload and overall idleness caused by the isolated management of private and public cloud resources, and significantly improving the visibility and schedulability of global computing power resources. By performing unified task graph modeling for training and inference tasks, the two types of tasks can be collaboratively analyzed and optimized under the same scheduling framework. The system no longer regards training and inference as unrelated loads, but can identify their potential correlations in computing dependencies, data flows, and resource requirements, paving the way for subsequent joint training and inference. The scheduling lays the foundation; based on network indicator data, the task graph is divided into regions, matching the deployment location of task subgraphs with the underlying network conditions. For example, subgraphs with high communication overhead can be constrained to the same low-latency domain, while latency-insensitive subgraphs can be flexibly distributed in remote areas. This network-aware partitioning strategy effectively reduces the negative impact of cross-domain communication on task execution efficiency and is more adaptable to the dynamic characteristics of real wide area network environments. The partitioned segmented subgraphs are allocated to different logical computing units for execution, achieving fine-grained matching of tasks and resources. The scheduling control plane can map the subgraphs to the most suitable physical nodes based on the real-time load, type, and location of each logical computing unit, thereby improving overall resource utilization efficiency and task completion time while ensuring correct task execution. In summary, this application's solution significantly improves the collaborative scheduling efficiency and resource utilization of AI large-scale model training and inference tasks in multi-cloud heterogeneous environments through global resource abstraction, unified task modeling, network-aware partitioning, cross-domain communication collaborative control, and precise subgraph allocation. This application does not schedule training tasks and inference tasks as two independent business categories separately. Instead, it performs joint modeling, joint partitioning, joint resource allocation, and priority coordination of training tasks and inference tasks under a unified scheduling control plane. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating the collaborative scheduling method for cross-domain hybrid training and hybrid inference of large AI models provided in this application; Figure 2 This is a schematic diagram of the task diagram provided in this application; Figure 3This is a schematic diagram of the structure of the AI ​​large model cross-domain hybrid training and hybrid inference collaborative scheduling system provided in this application; Figure 4 This is a schematic diagram of the cross-domain network and white-box switch deployment structure provided in this application; Figure 5 This is a schematic diagram of the automatic parameter adjustment and AIOps optimization process provided in this application; Figure 6 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] It should be noted that in the description of the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The terms "upper," "lower," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Unless otherwise expressly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly, for example, they can be fixed connections, detachable connections, or integral connections; they can be mechanical connections or electrical connections; they can be direct connections or indirect connections through an intermediate medium; and they can be internal connections between two elements. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0019] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0020] The following is combined with Figures 1-6 This application describes the AI ​​large-scale model cross-domain hybrid training and hybrid inference collaborative scheduling method and system provided in the embodiments.

[0021] Figure 1 This is a flowchart illustrating the collaborative scheduling method for cross-domain hybrid training and hybrid inference of large AI models provided in this application. Figure 1 As shown, the method includes the following: S110, Construct a global computing power view and scheduling control plane between the AI ​​private cloud and one or more public clouds, wherein the global computing power view abstracts GPU resources from different cloud regions into logical computing power units, and the logical computing power units record at least normalized computing power indicators, video memory capacity, region labels, communication capabilities and software stack compatibility information. The scheduling control plane schedules the training and inference tasks of the large AI model based on the following steps: S120, Perform task graph modeling for the training task and the inference task respectively, wherein the nodes in the task graph represent independently schedulable computational subtasks, and the edges represent data dependencies or control dependencies between subtasks. S130, based on the network metrics of latency, bandwidth and packet loss rate of cross-domain links, and combined with the inter-node communication volume, latency sensitivity and resource compatibility constraints in the task graph, the training task graph and the inference task graph are divided into regions to obtain multiple segmented subgraphs, and a target region label is determined for each segmented subgraph. S140, based on the target region label of each segmented subgraph, select logical computing units that meet resource requirements and compatibility constraints within the corresponding region, and assign each segmented subgraph to the selected logical computing units for execution.

[0022] Here, during the execution of each segmented subgraph, the mapping relationship between the segmented subgraph and the logical computing unit, as well as the cross-domain communication strategy, are dynamically adjusted based on training throughput, inference latency, resource utilization, and network status.

[0023] It should be noted that the execution subject of the AI ​​large model cross-domain hybrid training and hybrid inference collaborative scheduling method provided in this application embodiment can be a server, computer equipment, etc.

[0024] In this embodiment, the AI ​​private cloud is a cloud computing environment built or exclusively hosted by the enterprise, typically deployed in a local data center, possessing high security and complete control over the data / model. The public cloud is an elastic computing resource pool provided by a third-party cloud service provider. Here, the public cloud can also be a carrier cloud, a cloud computing service provided by telecommunications operators to government and enterprise customers, industry users, and the public based on their network infrastructure, data center resources, and cloud service capabilities.

[0025] Here, the global computing power view is a centralized resource registration and status awareness system that aggregates GPU resource information from private clouds and multiple public clouds in real time, such as type, quantity, load, location, network reachability, etc., to form a unified resource map.

[0026] Here, each GPU or group of GPUs is abstracted as a logical computing unit (LCU). Each LCU contains at least the following attributes: normalized computing power metrics (performance weight values ​​calculated based on computing power such as FP16 / BF16 / FP8); memory capacity and currently available memory; local and cross-node communication bandwidth (such as NVLink, PCIe, IB, or Ethernet) and its latency characteristics; the region it is located in (AI private cloud / a certain carrier cloud region) and the network characteristics between this region and other regions (round-trip time RTT, bandwidth, packet loss rate, bandwidth-latency product BDP, etc.); supported data types (FP32, FP16, BF16, INT8, FP8, etc.), key operator optimization capabilities (such as whether it supports Tensor Cores, matrix instructions, specific operator fusion); and software stack information (driver version, CUDA / ROCm / Ascend / domestic GPU SDK version, communication library version, etc.).

[0027] Here, the training or inference process of a large AI model is modeled as a directed acyclic graph, where a node represents an independently schedulable computational subtask, such as a Transformer layer or a preprocessing module, and edges represent data dependencies or control dependencies.

[0028] Here, network metrics data are quantitative parameters that describe the quality of cross-domain links, including but not limited to RTT, bandwidth, packet loss rate, and BDP.

[0029] Here, the region division is based on the network topology and indicators, dividing the task graph into multiple connected subgraphs, i.e. segmented subgraphs, so that communication is dense within each subgraph and sparse across subgraphs, and the subgraphs are assigned to geographical regions that match the network conditions.

[0030] In S110, a scheduling control plane is started on the control node. Through the agent or cloud API, GPU resource information is collected from the AI ​​private cloud and various public clouds. All physical GPUs are abstracted into logical computing units (LCUs) and registered in the global computing power view. The cloud region to which they belong, hardware type and capability tags, real-time load, and network indicators with other regions are recorded to build a global computing power view that reflects global resources.

[0031] Here, the normalized computing power metric is a relative performance score based on benchmark tests, used to measure the computing throughput of a GPU under a specific AI load; the video memory capacity is the total amount of video memory available to the physical GPU or GPU slice represented by the LCU; the region label is a set of metadata describing the physical location and logical affiliation of computing resources; the communication capability describes the bandwidth, latency, and topology information for data exchange between the LCU and other units; and the software stack compatibility information describes the software environment, driver version, and acceleration library information pre-installed or supported by the LCU.

[0032] In S120, when a user submits a large model training job or inference service request, the scheduling control plane abstracts the training and inference process of the large model into a task graph: like Figure 2 As shown, training nodes include, but are not limited to, data loading and preprocessing, embedding calculation, several layers of forward calculation, backward calculation, gradient aggregation, parameter update, checkpointing, and saving. Inference nodes include, but are not limited to, request access, sequence processing, forward inference, cache management, post-processing, and result return. Edges represent data dependencies or control dependencies, with constraints such as latency, bandwidth, and reliability (e.g., whether asynchronous operations or consistency requirements are allowed).

[0033] In S130, telemetry data of the link is acquired periodically, such as queue length, latency, packet loss rate, and ECN (Explicit Congestion Notification) marking ratio. RTT measurement results and bandwidth utilization data are collected from the end side. The task graph is divided into multiple segmented subgraphs to meet the following requirements: the communication volume between nodes within a subgraph is large, so they should be deployed in the same low-latency domain; the communication volume between subgraphs is small, so cross-domain transmission can be tolerated; and the end of the inference is as close to the user side as possible. Finally, each segmented subgraph is bound to a target area label.

[0034] In S140, for each segmented subgraph, available LCUs in the corresponding region of the global computing power view are selected based on its region label. Matching and binding are performed based on the resource requirements of the subgraph and the LCU capability labels, and execution instructions are issued to the target region. Each subgraph executes in parallel or pipelined on its own LCU, exchanging intermediate data, such as activation values ​​and gradients, through a cross-domain network.

[0035] The AI ​​large-scale model cross-domain hybrid training and hybrid inference collaborative scheduling method provided in this application establishes a global computing power view and abstracts all GPUs as logical computing power units, breaking down resource barriers between different cloud environments and hardware platforms. The scheduling control plane can allocate computing power in a unified resource pool, avoiding the problem of local overload and overall idleness caused by the isolated management of private and public cloud resources, and significantly improving the visibility and schedulability of global computing power resources. By performing unified task graph modeling for training and inference tasks, the two types of tasks can be collaboratively analyzed and optimized under the same scheduling framework. The system no longer regards training and inference as unrelated loads, but can identify their potential correlations in computing dependencies, data flows, and resource requirements, paving the way for subsequent joint... The scheduling lays the foundation; based on network indicator data, the task graph is divided into regions, matching the deployment location of task subgraphs with the underlying network conditions. For example, subgraphs with high communication overhead can be constrained to the same low-latency domain, while latency-insensitive subgraphs can be flexibly distributed in remote areas. This network-aware partitioning strategy effectively reduces the negative impact of cross-domain communication on task execution efficiency and is more adaptable to the dynamic characteristics of real wide area network environments. The partitioned segmented subgraphs are allocated to different logical computing units for execution, achieving fine-grained matching of tasks and resources. The scheduling control plane can map the subgraphs to the most suitable physical nodes based on the real-time load, type, and location of each logical computing unit, thereby improving overall resource utilization efficiency and task completion time while ensuring correct task execution. In summary, this application's solution, through global resource abstraction, unified task modeling, network-aware partitioning, and precise subgraph allocation, significantly improves the collaborative scheduling efficiency and resource utilization of AI large-scale model training and inference tasks in multi-cloud heterogeneous environments without relying on additional communication optimization or hardware modifications.

[0036] In an optional embodiment, assigning each of the segmented subgraphs to the selected logical computing units for execution includes: The first type of segmented subgraph is assigned to the logical computing unit on the AI ​​private cloud side for execution; The second type of segmented subgraph is assigned to the logical computing unit on the public cloud side for execution; Specifically, the computational density of the first type of segmented subgraph is greater than that of the second type of segmented subgraph; the communication volume and frequency of the first type of segmented subgraph are less than those of the second type of segmented subgraph; the user relevance of the second type of segmented subgraph is greater than that of the first type of segmented subgraph; and the latency sensitivity of the second type of segmented subgraph is greater than that of the first type of segmented subgraph. The first type of segmented subgraph and the second type of segmented subgraph are determined according to a preset classification rule or classification model.

[0037] Optionally, the computational density is determined based on the ratio of the required number of floating-point operations to the amount of input / output data. The higher the value, the more computationally intensive the computation and the less data is transmitted, making it more suitable for deployment on central nodes with high computing power but high network latency.

[0038] Here, user relevance refers to whether the task is triggered by the user in real time and whether its output directly affects the user experience. This can be determined through the request source, task metadata, etc.

[0039] Here, latency sensitivity refers to the tolerance threshold of a task for end-to-end response time. It is usually determined by the Service Level Agreement (SLA) contract or the type of business; for example, the latency sensitivity of chatbots is higher than that of video analytics.

[0040] In this embodiment, the task graph is segmented, and computationally intensive and less communication-intensive stages are preferentially placed on the side with stronger heterogeneous GPU capabilities, such as the NVIDIA cluster on the private cloud side; while stages that are closely related to user interaction and are sensitive to latency are placed on the carrier / public cloud side, such as the second half of inference or lightweight inference models.

[0041] Specifically, the first type of segmented subgraphs exhibits high computational intensity, meaning the floating-point operations performed per unit time far exceed the data transmission volume; low communication volume and frequency, meaning the amount of data interacting with other subgraphs is small and the number of synchronizations is infrequent; weak user relevance, meaning they do not directly handle end-user requests and are typically located in the later stages of the model or the training backbone; and low latency sensitivity, tolerating relatively high response times, making them suitable for offline or batch processing scenarios. Examples include intermediate Transformer layers in large language models, expert computation modules in hybrid expert models, and deep convolutional blocks in visual models.

[0042] The second type of segmented subgraph has a relatively lightweight computational load and low computational intensity; it needs to frequently receive user input or transmit results downstream, resulting in high communication volume and frequency; it directly responds to end-user requests, such as generating replies and returning recommendation results, and has strong user relevance; it requires strict end-to-end response time constraints and is highly latency sensitive.

[0043] In the specific implementation process, the computational intensity, communication volume, communication frequency, user relevance, and latency sensitivity of each segmented subgraph are analyzed. Based on preset rules or machine learning models, the segmented subgraphs are divided into either Category I or Category II. Category I segmented subgraphs are assigned to the AI ​​private cloud-side LCU, utilizing its high-performance GPU to accelerate intensive computation. Due to less communication, even with high cross-domain link RTT, the impact is limited. Category II segmented subgraphs are assigned to the public cloud-side LCU, preferably using operator edge nodes, which are geographically close to users, reducing request round-trip latency. Even with slightly weaker GPU performance, the lightweight computation still meets the usage requirements. After the private cloud side completes the computation of the Category I segmented subgraphs, the intermediate results are sent to the public cloud side through a cross-domain data channel. Upon receiving the results, the public cloud side immediately executes the Category II segmented subgraphs and returns the final results to the user.

[0044] The AI ​​large-scale model cross-domain hybrid training and hybrid inference collaborative scheduling method provided in this application addresses the second type of subgraph, which has high user relevance and high latency sensitivity. Its execution location directly determines end-to-end latency. Since the public cloud is geographically closer to end users, deploying the second type of subgraph on the public cloud allows user requests to complete critical responses without traversing a remote private cloud, significantly reducing request latency and improving the SLA achievement rate of inference tasks. The first type of subgraph is computationally intensive and has low communication; its performance bottleneck lies in GPU computing power, not the network. AI private clouds are typically equipped with top-tier GPUs; centrally deploying this type of subgraph on the private cloud maximizes the utilization of the private cloud's GPUs. In summary, this embodiment can automatically distribute traffic based on subgraph characteristics, simultaneously supporting offline large-scale model training and online dialogue services, enhancing the system's adaptability to heterogeneous business loads.

[0045] In an optional embodiment, assigning each of the segmented subgraphs to the selected logical computing units for execution includes: Under the conditions of satisfying the service level agreement and resource utilization threshold constraints of the inference task, the execution region of each segmented subgraph is determined with training throughput, inference latency and execution cost as joint optimization objectives. For each segmented subgraph, a logical computing unit is selected and executed in the corresponding execution region; Determine the topology and frequency for cross-domain communication.

[0046] Here, training throughput is the number of training samples or gradient steps completed per unit time; inference latency is the end-to-end latency from user request to response; cost is the combination of private cloud depreciation and public cloud pay-as-you-go billing; resource utilization is the global GPU utilization; and meeting the preset SLA for inference tasks is a hard constraint.

[0047] Here, the objective function is a mathematical combination of multiple performance metrics that the scheduling decision aims to optimize, including but not limited to maximizing training throughput, minimizing inference latency, minimizing cost, ensuring resource utilization exceeds a preset threshold, and ensuring the inference task meets a preset SLA.

[0048] Here, the execution region refers to a geographically or logically divided cloud environment unit, namely AI private cloud, carrier cloud, and public cloud. Each region has different computing power costs, network RTT, bandwidth limits, and compliance attributes.

[0049] Here, the topology of cross-domain communication refers to the connection method of data transmission between segmented subgraphs, such as point-to-point, star aggregation, and hierarchical synchronization. Frequency refers to the number of times cross-domain communication occurs per unit time, such as once per training step or once per K steps.

[0050] In this embodiment, the objective functions are to maximize training throughput, minimize inference latency, and minimize cost, while comprehensively considering resource utilization and SLA requirements; the execution area, target LCU type, and topology and frequency of cross-domain communication for each task node are determined.

[0051] In the specific implementation process, the scheduling control plane pre-establishes or learns the following models online: computation time consumption model, communication time consumption model, cost model, and SLA violation probability model. Among them, the computation time consumption model evaluates the expected execution time of a subgraph on a certain LCU, the communication time consumption model evaluates cross-domain transmission delay, the cost model evaluates the unit time computing power cost of each region, and the SLA violation probability model predicts the SLA violation probability based on historical monitoring data.

[0052] Furthermore, for each segmented subgraph, considering factors such as hardware compatibility and network reachability, its deployable execution regions are enumerated to form a composite space. For example, segmented subgraphs... Figure 1 Options include private cloud and carrier edge cloud, segmented. Figure 2 The options are carrier edge cloud and public cloud, and the total number of candidate solutions is the product of the number of options in each segmented subgraph.

[0053] Furthermore, each candidate solution is substituted into the objective function to solve for the optimal solution set. From the optimal solution, the execution region of each segmented subgraph is extracted. Within this region, specific logical computing units are matched according to load and capacity. Based on the dependencies between subgraphs and the network model, the cross-domain communication topology and synchronization frequency are determined. Finally, the segmented subgraph scheduling instructions are issued to each region for execution, and real-time data on actual throughput, latency, cost, and utilization are collected. If deviations from expectations occur, such as SLA violations, re-optimization or degradation strategies are triggered.

[0054] In an optional embodiment, scheduling decisions are sent to the underlying container orchestration platform to create or adjust corresponding training tasks, inference services, replica counts, and affinity / anti-affinity policies. When the network quality in a certain area deteriorates significantly or resources become scarce, task migration or replica redistribution is triggered to achieve cross-domain load balancing and fault avoidance.

[0055] The AI ​​large model cross-domain hybrid training and hybrid inference collaborative scheduling method provided in this application integrates cost, throughput, latency, and SLA into the optimization framework, achieving Pareto optimality of cost and performance while ensuring SLA. In addition, the resource utilization rate ≥ threshold is set as a hard constraint, which requires the optimizer to select an allocation scheme that can fill the idle GPU, thereby significantly improving the global resource utilization and avoiding local overload and global idleness.

[0056] In an optional embodiment, the AI ​​large model cross-domain hybrid training and hybrid inference collaborative scheduling method further includes: In the training task, gradients are layered according to importance. The gradients of the first layer are synchronized in each iteration step. The synchronization frequency of the gradient of the i-th layer is greater than that of the gradient of the (i+1)-th layer, where i is a positive integer.

[0057] In this embodiment, gradients are hierarchically classified according to their importance. Gradients with a greater impact on convergence are synchronized in each iteration step, while minor gradients are synchronized in a longer period or asynchronously. The model is divided into multiple pipeline stages, with each stage executed in different regions. Intermediate activations and gradients are transferred across domains to form a pipeline, thereby improving GPU utilization and reducing the impact of single synchronization latency.

[0058] Here, stratifying gradients by importance refers to dividing all gradients into multiple priority levels, such as layer 1, layer 2, ..., layer K, based on the influence of model parameters on the loss function, convergence sensitivity, or historical update magnitude. Layer 1 gradients are the most critical parts for model convergence, such as the classification head and attention output weights, which change drastically or have a significant impact on the loss. Layer i+1 gradients are relatively less important, such as the lower embedding layers and redundant channels, which update more slowly or have high information redundancy. The criteria for determining gradient importance may include, but are not limited to, the magnitude of the gradient's L2 norm, the diagonal elements of the Fisher information matrix, the sensitivity of parameters on the validation set, and dynamic evaluation during training.

[0059] Here, synchronization frequency refers to how many training iterations (steps) it takes for a certain type of gradient to complete a cross-node aggregation in distributed training. A high synchronization frequency means synchronization every 1 step, while a low synchronization frequency means synchronization every K steps (K>1), or asynchronous background aggregation. For example, the first layer synchronizes every 1 step, the second layer every 2 steps, the third layer every 5 steps, and the higher the layer, the sparser the synchronization.

[0060] In the specific implementation process, the scheduling control plane divides the model parameters into K importance levels according to the model structure and preset rules, or through online analysis, and configures an initial synchronization frequency for each level. All nodes perform local calculations normally, generate complete gradients, and split the gradients into K subsets according to the predefined levels. If the current step is an integer multiple of the synchronization period of the corresponding gradient level, the gradient of that level is synchronized; otherwise, the gradient of that level is cached in the local error accumulation buffer.

[0061] For gradient layers that need to be synchronized, they are sent to the aggregation node via a cross-domain link. After aggregation, the updates are broadcast to all participating nodes. The local model only uses the synchronized gradients to update the corresponding parameters, while unsynchronized layers retain their original values.

[0062] The AI ​​large-scale model cross-domain hybrid training and hybrid inference collaborative scheduling method provided in this application performs layered synchronization based on gradient importance. On the one hand, it reduces the gradient synchronization frequency, significantly reducing the total amount of cross-domain communication and alleviating bandwidth bottlenecks. On the other hand, it ensures that critical gradients are synchronized at every step, ensuring that the backbone parameters of the model remain consistent. Although secondary gradients are sparsely synchronized, information loss can be compensated through error feedback mechanisms, thereby maintaining the model convergence quality and avoiding divergence caused by asynchronous updates. In addition, high-frequency synchronization is prone to triggering timeouts or retransmissions when RTT jitter occurs. This scheme sparsifies most of the communication, keeping only the critical path high-frequency. Even if a cross-domain synchronization fails, it only affects secondary parameters, and backbone convergence is not disturbed, improving the system's robustness to network fluctuations.

[0063] In an optional embodiment, the synchronization frequency and gradient compression ratio of each layer gradient are updated in real time based on the network state.

[0064] Here, gradient synchronization frequency refers to how many training iterations a gradient of a certain layer completes cross-node aggregation, and gradient compression ratio refers to the ratio of the original gradient data volume to the compressed data volume.

[0065] In this embodiment, when starting the training task, the gradient is divided into K layers according to the model structure. An initial synchronization frequency and compression ratio are configured for each layer. The scheduling control plane collects the RTT, bandwidth, packet loss rate, current cross-domain traffic throughput, and queue congestion status reported by the white-box switch at fixed intervals for the private cloud-public cloud link. An adaptive policy engine based on rules or machine learning models is invoked to update parameters according to the following logic: if RTT increases or bandwidth decreases, the compression ratio of all layers is increased, and the synchronization frequency of non-critical layers is reduced; if the packet loss rate increases, the compression ratio is increased (reducing the amount of retransmitted data), or a more robust compression algorithm is switched; if the inference SLA pressure is high, the gradient synchronization of layers 2 and above is temporarily frozen to release bandwidth; if bandwidth utilization is low, the compression ratio is reduced, and the synchronization frequency is increased. Furthermore, the updated synchronization frequency and gradient compression ratio are broadcast to all training nodes, and the new policy takes effect at the start of the next iteration step. Data such as training throughput, convergence speed, and communication volume after policy adjustment are recorded, and the policy mapping function is continuously improved using reinforcement learning or Bayesian optimization.

[0066] The AI ​​large model cross-domain hybrid training and hybrid inference collaborative scheduling method provided in this application maintains training throughput stability in a dynamic network environment by dynamically adjusting the gradient synchronization frequency and gradient compression ratio, minimizing communication costs while ensuring convergence, and avoiding training failure due to sudden network deterioration.

[0067] In an optional embodiment, the AI ​​large model cross-domain hybrid training and hybrid inference collaborative scheduling method further includes: The traffic from the training task is labeled as a first-class business flow; The traffic from inference tasks is labeled as a second-category business flow; The priority of the second type of service flow is higher than that of the first type of service flow.

[0068] Here, "service flow" refers to a class of network data flows with the same service attributes and Quality of Service (QoS) requirements. The first type of service flow is traffic generated by training tasks, including gradient synchronization, parameter broadcasting, and checkpoint uploads. The second type of service flow is traffic generated by online inference tasks, including user requests, model responses, and health checks.

[0069] Here, priority control refers to allocating traffic to different priority queues based on traffic tags and employing corresponding scheduling strategies. For example, high-priority queues are strictly prioritized, sending traffic as long as it is not empty, while low-priority queues are either weighted fair queues or best-effort queues.

[0070] In this embodiment, the AI ​​private cloud and the operator / public cloud establish multiple long-distance links through methods such as leased lines, Multi-Protocol Label Switching (MPLS), Software-Defined Networking in a Wide Area Network (SD-WAN), and Virtual Private Network (VPN). White-box switches or programmable switches are deployed in the middle or at both ends of the links to perform traffic classification, queue management, link telemetry reporting, and ECN (Explicit Congestion Notification) marking. In the white-box switches on the cross-domain links, traffic is classified according to packet characteristics: training traffic such as gradient synchronization and parameter transmission is marked as Class I service flow, such as CoS1; online inference request and response traffic is marked as Class II service flow, such as CoS0; and non-critical traffic such as logs, monitoring, and backup is marked as ordinary service or low-priority traffic. Different service flows correspond to different queues and scheduling strategies. For example, higher priority and minimum bandwidth guarantee are allocated to inference traffic, while a larger but compressible bandwidth share is allocated to training traffic.

[0071] Optionally, a Quality of Service (QoS) identifier field can be written into the header of the network data packet for subsequent device identification and processing.

[0072] In an optional embodiment, a programmable data plane is used to embed telemetry information such as queue length, hop count, and delay into the packet header, and ECN bits are used to mark congestion. The end-side transmission module dynamically adjusts parameters such as sending window, parallelism, data block size, and retransmission strategy based on the telemetry and ECN information in the received packets.

[0073] In optional embodiments, a custom congestion control algorithm is implemented at the transport layer or application layer: based on the estimated BDP, the amount of data in transit is controlled to remain within a reasonable range to avoid excessively low bandwidth utilization or excessively long queues; for training traffic, parameters such as gradient compression ratio and synchronization frequency are appropriately adjusted according to network conditions; for inference traffic, low latency and stability of the traffic are prioritized, and the training traffic rate is actively limited when necessary.

[0074] In an optional embodiment, GPU nodes in the AI ​​private cloud and the carrier / public cloud are incorporated into a unified resource pool to allocate resources uniformly for training and inference tasks. Based on the business SLA, strategies are set such as prioritizing inference tasks and prioritizing critical stages (such as the last few rounds of convergence) in training tasks.

[0075] In an optional embodiment, during peak inference service periods, the system automatically guarantees the number of GPUs and resource quotas for inference instances on the operator / public cloud side, and temporarily reclaims some GPUs from training tasks when necessary; during off-peak inference service periods, the system automatically downgrades or shuts down some inference instances and allocates the released GPUs to training tasks for incremental training, distillation, or model evaluation.

[0076] The AI ​​large model cross-domain hybrid training and hybrid inference collaborative scheduling method provided in this application gives strict priority to inference traffic. Even if the training stream is being transmitted, it can immediately preempt the link to avoid large-scale training from occupying most of the bandwidth, thus ensuring inference service and significantly improving the SLA achievement rate of inference tasks.

[0077] In an optional embodiment, the main model is trained on the AI ​​private cloud side, and multiple lightweight student models are deployed on the operator / public cloud side for online inference; the system periodically transmits the latest parameters or intermediate features of the main model to the operator / public cloud side through cross-domain links to retrain or distill the student models; through differential parameter transmission and hot update mechanism, update costs are reduced and the update latency of the inference model is shortened.

[0078] In an optional embodiment, when a network or computing power failure occurs in a certain area, the system can launch additional inference replicas in other areas to replace the services in the failed area; when the network is severely restricted, degradation modes such as compression models and low-precision inference are enabled to ensure basic service capabilities.

[0079] The following describes the AI ​​large-scale model cross-domain hybrid training and hybrid inference collaborative scheduling system provided in the embodiments of this application. The system described below corresponds to the method described above. In a specific deployment implementation, the above-mentioned global computing power view construction module, task graph modeling module, region partitioning module, scheduling decision module, cross-domain communication control module, and execution feedback and optimization module can be deployed separately or in combination in the AI ​​private cloud cluster, public cloud cluster, and cross-domain network control node.

[0080] like Figure 3 As shown, the system includes a global computing power view construction module, a task graph modeling module, a region partitioning module, a scheduling decision module, a cross-domain communication control module, and an execution feedback and optimization module.

[0081] The global computing power view construction module is deployed on the master control node of the AI ​​private cloud. It is used to collect GPU resource information in the AI ​​private cloud and one or more public clouds, abstract physical GPUs from different manufacturers and regions into logical computing power units, and maintain the regional label, computing power, memory capacity, communication capabilities and software stack compatibility information of each logical computing power unit.

[0082] The task graph modeling module is used to convert a training task or inference task into a task graph after receiving the task, where nodes represent independently schedulable computational subtasks and edges represent data dependencies or control dependencies.

[0083] The region segmentation module is used to combine the RTT, bandwidth, packet loss rate, bandwidth-delay product of cross-domain links, as well as the inter-node communication volume and latency sensitivity in the task graph to segment the task graph into regions, generate multiple segmented subgraphs, and determine the target region label for each segmented subgraph.

[0084] The scheduling decision module is used to select logical computing units that meet resource requirements and compatibility constraints within the corresponding region based on the target region labels of each segmented subgraph, and generate execution plans for training and inference tasks. In an optional embodiment, the scheduling decision module can also combine measured benchmark results to determine the adaptation relationship of different types of GPUs to different operators.

[0085] The cross-domain communication control module is used to determine the gradient synchronization frequency, gradient compression ratio, sending window size, number of concurrent connections, and priority strategies for training traffic and inference traffic on cross-domain links such as leased lines, MPLS, SD-WAN, and VPN between AI private clouds and public clouds, in conjunction with telemetry information and ECN tags provided by white-box switches or programmable switches.

[0086] The execution feedback and optimization module continuously collects operational metrics such as GPU utilization, memory usage, training throughput, inference latency, network RTT, packet loss rate, and bandwidth utilization. Based on rule-based, Bayesian optimization, or reinforcement learning methods, it dynamically adjusts task mapping relationships, parallelism parameters, synchronization frequency, and compression strategies, and triggers task migration, degradation, or re-optimization in abnormal situations. In this embodiment, the AI ​​private cloud cluster deploys core dataset storage, a large model parameter repository, a main training node, and some inference service nodes; it primarily uses NVIDIA GPUs, but may also include some domestic GPU nodes; the main control node deploys global control and scheduling components, including a global computing power and task scheduling controller, a long-distance perception scheduling subsystem, an automatic scheduling component, and an AIOps (Artificial Intelligence for IT Operations) engine.

[0087] In this embodiment, the operator / public cloud side cluster deploys an online inference service cluster for end users and a certain number of training / distillation nodes; it mainly uses domestic GPUs, but can also be equipped with some NVIDIA GPUs; it is responsible for undertaking low-latency inference tasks close to the user side, and participates in incremental training, distillation training and other operations during business downtime.

[0088] In this embodiment, the cross-domain network, i.e., the AI ​​private cloud and the operator / public cloud, establishes multiple long-distance links through leased lines, MPLS, SD-WAN, VPN, etc.; white-box switches or programmable switches are deployed in the middle or at both ends of the links, which can perform traffic classification, queue management, link telemetry reporting, and ECN marking, etc.; a control channel is provided for transmitting network status and monitoring data, and a data channel is provided for transmitting business data such as model parameters, gradients, intermediate features, and inference requests.

[0089] Figure 4 This is a schematic diagram of the cross-domain network and white-box switch deployment structure provided in this application, such as... Figure 4 As shown, the global control and scheduling components include: a global computing power and task scheduling controller: maintaining cross-domain resource views such as GPUs, CPUs, and storage, responsible for scheduling decisions and lifecycle management of training and inference tasks; a long-distance sensing scheduling subsystem: collecting inter-regional network status (RTT, bandwidth, packet loss rate, BDP), splitting tasks and scheduling them across domains; a cross-domain network transmission and congestion control subsystem: collaborating with white-box switches and end-side transmission modules to implement training / inference decoupling queues, congestion control, and automatic parameter tuning; a heterogeneous GPU abstraction and adaptation layer: abstracting GPUs from different vendors into unified logical computing units, providing multi-backend operator implementations and runtime adaptation for large models; a mixed training and inference runtime layer: responsible for scheduling various subtasks to run on specific nodes and GPUs during actual training and inference execution, managing cross-domain pipelines and task migration; and an automatic parameter tuning and AIOps engine: automatically adjusting and optimizing various system parameters based on collected runtime data.

[0090] In this embodiment, multiple backend implementation versions are provided for core operators in large models, such as Attention, Feed-Forward, Embedding, LayerNorm, Activation, and Convolution. For example, there are CUDA versions for NVIDIA GPUs, Ascend / HCCL versions for Huawei 910B, corresponding SDK versions for domestic GPUs like MU-X, and alternative implementations for general-purpose CPUs or other accelerators. At runtime, the heterogeneous GPU abstraction and adaptation layer selects the corresponding backend operator implementation based on the LCU type assigned to the task, ensuring that the same model can run in an adapted manner on different types of GPUs.

[0091] Furthermore, during the deployment phase or early operational phase, the system performs small-scale benchmark tests on different types of GPUs and operator combinations to measure actual performance metrics and build a GPU-operator-configuration capability model. When placing tasks, the scheduler bases its decisions not only on theoretical specifications but also on the measured performance data from the capability model to improve the accuracy of heterogeneous scheduling decisions.

[0092] Figure 5 This is a schematic diagram of the automatic parameter tuning and AIOps optimization process provided in this application, such as... Figure 5 As shown in this embodiment, the system continuously collects metrics such as GPU utilization, memory usage, task queuing time, training throughput, model loss curve and convergence speed, network RTT, bandwidth utilization, packet loss rate, ECN tagging ratio, inference QPS, P95 / P99 latency, and error rate, and aggregates them into a unified data storage. Based on these metrics, the automatic parameter tuning and AIOps engine searches and adjusts the following parameters using methods such as rules, Bayesian optimization, and reinforcement learning: training batch size, data parallelism / model parallelism / pipeline parallelism; cross-domain gradient synchronization frequency and compression strategy; transmission window size, number of concurrent connections, data block size; and the number and distribution area of ​​inference replicas. The engine performs anomaly detection on the training convergence process, network state, and inference latency. When gradient oscillations, large network jitters, or sudden increases in latency occur, it triggers automatic parameter tuning, task migration, or degradation strategies to achieve closed-loop self-healing.

[0093] The technical solution of this application will be further described below with reference to several embodiments.

[0094] Example 1: Cross-domain hybrid training and inference using A100 and Huawei 910B: Several training nodes and storage systems equipped with NVIDIA A100 GPUs are deployed on the AI ​​private cloud side; several nodes equipped with Huawei 910B GPUs are deployed on the carrier cloud side, which can be used for both training and inference simultaneously; a dedicated line connects the two data centers, and white-box switches are deployed in the middle or at both ends to classify and queue training and inference traffic, and configure ECN thresholds and telemetry reporting; the global scheduling controller divides the large model into multiple pipeline stages, with the first few stages scheduled to the A100 cluster for execution, and subsequent stages scheduled to the Huawei 910B cluster for execution, based on the differences in computational density, communication volume, and latency sensitivity of each stage. During training, forward activation and some gradients are transmitted between the two sides. The end-side transmission module dynamically adjusts the sending window, parallelism, and gradient compression ratio based on the telemetry information. When the training reaches a certain convergence, the model weights are transmitted to the operator's cloud side for inference services. If the network quality deteriorates or the operator's side load is too high, the scheduler can temporarily reduce the number of stages involved in cross-domain training or switch to single-domain training mode.

[0095] Example 2: Private cloud master model training + carrier cloud online inference and distillation: The AI ​​private cloud uses a centralized NVIDIA GPU cluster for pre-training and fine-tuning of the main model. The carrier cloud deploys multiple lightweight student model instances on a domestic GPU cluster to provide online inference services for end users. A unified scheduling system monitors inference request QPS (Queries Per Second). When the inference request QPS falls below a preset threshold and the carrier cloud's resource utilization is also below a preset threshold, some carrier-side GPU nodes are converted into distillation training task nodes, using intermediate features or labels from the private cloud's main model to distill the student models. After the private cloud's main model is updated, the system pushes incremental parameters or knowledge distillation data to the carrier cloud via cross-domain links. The automatic scheduling and AIOps engine adjust the distillation frequency and model update strategy based on inference quality metrics (such as accuracy, recall, user feedback, etc.) and network status, reducing transmission costs and update latency while ensuring inference performance.

[0096] Example 3: Cross-domain mixed training and push system jointly deployed by multiple operators in different regions: The AI ​​private cloud serves as the central training and management node, with each operator region (A and B) possessing its own GPU inference clusters. The central private cloud uniformly trains the main model and periodically pushes differential parameters or distilled models to each operator region. It also maintains a unified global computing power view and makes cross-regional scheduling decisions for training and inference tasks across operator regions. Different operator regions can also back up inference instances to each other based on network distance and load conditions. When one region experiences a failure or excessive load, other regions can provide inference services on its behalf. The global scheduling controller can schedule GPU nodes from some operator regions to participate in training tasks during training downtime, enabling multi-regional joint training and inference.

[0097] It should be noted that the AI ​​large model cross-domain hybrid training and hybrid inference collaborative scheduling device provided in this application embodiment can execute the AI ​​large model cross-domain hybrid training and hybrid inference collaborative scheduling method described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0098] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a collaborative scheduling method for cross-domain hybrid training and hybrid inference of large AI models, which includes: Construct a global computing power view and scheduling control plane between an AI private cloud and one or more public clouds. The global computing power view abstracts GPU resources from different cloud regions into logical computing power units. Each logical computing power unit records at least normalized computing power indicators, video memory capacity, region labels, communication capabilities, and software stack compatibility information. The scheduling control plane schedules the training and inference tasks of the large AI model based on the following steps: Task graphs are modeled for the training task and the inference task respectively, where nodes in the task graph represent independently schedulable computational subtasks, and edges represent data dependencies or control dependencies between subtasks. Based on network metrics such as latency, bandwidth, and packet loss rate of cross-domain links, and combined with constraints on inter-node communication volume, latency sensitivity, and resource compatibility in the task graph, the training task graph and inference task graph are divided into regions to obtain multiple segmented subgraphs, and target region labels are determined for each segmented subgraph. Based on the target region label of each segmented subgraph, logical computing power units that meet resource requirements and compatibility constraints are selected in the corresponding region, and each segmented subgraph is assigned to the selected logical computing power units for execution. During the execution of each segmented subgraph, the mapping relationship between the segmented subgraph and the logical computing unit, as well as the cross-domain communication strategy, are dynamically adjusted based on training throughput, inference latency, resource utilization, and network status.

[0099] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0100] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the AI ​​large-scale model cross-domain hybrid training and hybrid inference collaborative scheduling method provided by the above methods, the method including: Construct a global computing power view and scheduling control plane between an AI private cloud and one or more public clouds. The global computing power view abstracts GPU resources from different cloud regions into logical computing power units. Each logical computing power unit records at least normalized computing power indicators, video memory capacity, region labels, communication capabilities, and software stack compatibility information. The scheduling control plane schedules the training and inference tasks of the large AI model based on the following steps: Task graphs are modeled for the training task and the inference task respectively, where nodes in the task graph represent independently schedulable computational subtasks, and edges represent data dependencies or control dependencies between subtasks. Based on network metrics such as latency, bandwidth, and packet loss rate of cross-domain links, and combined with constraints on inter-node communication volume, latency sensitivity, and resource compatibility in the task graph, the training task graph and inference task graph are divided into regions to obtain multiple segmented subgraphs, and target region labels are determined for each segmented subgraph. Based on the target region label of each segmented subgraph, logical computing power units that meet resource requirements and compatibility constraints are selected in the corresponding region, and each segmented subgraph is assigned to the selected logical computing power units for execution. During the execution of each segmented subgraph, the mapping relationship between the segmented subgraph and the logical computing unit, as well as the cross-domain communication strategy, are dynamically adjusted based on training throughput, inference latency, resource utilization, and network status.

[0101] Furthermore, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the AI ​​large-model cross-domain hybrid training and hybrid inference collaborative scheduling method provided by the above methods, the method comprising: Construct a global computing power view and scheduling control plane between an AI private cloud and one or more public clouds. The global computing power view abstracts GPU resources from different cloud regions into logical computing power units. Each logical computing power unit records at least normalized computing power indicators, video memory capacity, region labels, communication capabilities, and software stack compatibility information. The scheduling control plane schedules the training and inference tasks of the large AI model based on the following steps: Task graphs are modeled for the training task and the inference task respectively, where nodes in the task graph represent independently schedulable computational subtasks, and edges represent data dependencies or control dependencies between subtasks. Based on network metrics such as latency, bandwidth, and packet loss rate of cross-domain links, and combined with constraints on inter-node communication volume, latency sensitivity, and resource compatibility in the task graph, the training task graph and inference task graph are divided into regions to obtain multiple segmented subgraphs, and target region labels are determined for each segmented subgraph. Based on the target region label of each segmented subgraph, logical computing power units that meet resource requirements and compatibility constraints are selected in the corresponding region, and each segmented subgraph is assigned to the selected logical computing power units for execution. During the execution of each segmented subgraph, the mapping relationship between the segmented subgraph and the logical computing unit, as well as the cross-domain communication strategy, are dynamically adjusted based on training throughput, inference latency, resource utilization, and network status.

[0102] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0103] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for cross-domain hybrid training and hybrid inference collaborative scheduling of large AI models, characterized in that, include: Construct a global computing power view and scheduling control plane between an AI private cloud and one or more public clouds. The global computing power view abstracts GPU resources from different cloud regions into logical computing power units. Each logical computing power unit records at least normalized computing power indicators, video memory capacity, region labels, communication capabilities, and software stack compatibility information. The scheduling control plane schedules the training and inference tasks of the large AI model based on the following steps: Task graphs are modeled for the training task and the inference task respectively, where nodes in the task graph represent independently schedulable computational subtasks, and edges represent data dependencies or control dependencies between subtasks. Based on network metrics such as latency, bandwidth, and packet loss rate of cross-domain links, and combined with constraints on inter-node communication volume, latency sensitivity, and resource compatibility in the task graph, the training task graph and inference task graph are divided into regions to obtain multiple segmented subgraphs, and target region labels are determined for each segmented subgraph. Based on the target region label of each segmented subgraph, logical computing power units that meet resource requirements and compatibility constraints are selected in the corresponding region, and each segmented subgraph is assigned to the selected logical computing power units for execution. During the execution of each segmented subgraph, the mapping relationship between the segmented subgraph and the logical computing unit, as well as the cross-domain communication strategy, are dynamically adjusted based on training throughput, inference latency, resource utilization, and network status.

2. The AI ​​large-scale model cross-domain hybrid training and hybrid inference collaborative scheduling method according to claim 1, characterized in that, The step of assigning each of the segmented subgraphs to the selected logical computing units for execution includes: The first type of segmented subgraph is assigned to the logical computing unit on the AI ​​private cloud side for execution; The second type of segmented subgraph is assigned to the logical computing unit on the public cloud side for execution; Specifically, the computational density of the first type of segmented subgraph is greater than that of the second type of segmented subgraph; the communication volume and frequency of the first type of segmented subgraph are less than those of the second type of segmented subgraph; the user relevance of the second type of segmented subgraph is greater than that of the first type of segmented subgraph; and the latency sensitivity of the second type of segmented subgraph is greater than that of the first type of segmented subgraph. The first type of segmented subgraph and the second type of segmented subgraph are determined according to a preset classification rule or classification model.

3. The AI ​​large-scale model cross-domain hybrid training and hybrid inference collaborative scheduling method according to claim 1, characterized in that, The step of assigning each of the segmented subgraphs to the selected logical computing units for execution includes: Under the conditions of satisfying the service level agreement and resource utilization threshold constraints of the inference task, the execution region of each segmented subgraph is determined with training throughput, inference latency and execution cost as joint optimization objectives. For each segmented subgraph, a logical computing unit is selected and executed in the corresponding execution region; Determine the topology and frequency for cross-domain communication.

4. The AI ​​large-scale model cross-domain hybrid training and hybrid inference collaborative scheduling method according to claim 1, characterized in that, The AI ​​large-model cross-domain hybrid training and hybrid inference collaborative scheduling method also includes: In the training task, gradients are layered according to importance. The gradients of the first layer are synchronized in each iteration step. The synchronization frequency of the gradient of the i-th layer is greater than that of the gradient of the (i+1)-th layer, where i is a positive integer.

5. The AI ​​large-scale model cross-domain hybrid training and hybrid inference collaborative scheduling method according to claim 4, characterized in that, The synchronization frequency and gradient compression ratio of each layer's gradient are updated in real time based on the network state.

6. The AI ​​large-scale model cross-domain hybrid training and hybrid inference collaborative scheduling method according to claim 1, characterized in that, The AI ​​large-model cross-domain hybrid training and hybrid inference collaborative scheduling method also includes: The traffic from the training task is labeled as a first-class business flow; The traffic from inference tasks is labeled as a second-category business flow; The priority of the second type of service flow is higher than that of the first type of service flow.

7. A collaborative scheduling system for cross-domain hybrid training and hybrid inference of large AI models, characterized in that, include: The global computing power view building module is used to abstract GPU resources in the AI ​​private cloud and one or more public clouds into logical computing power units; The task graph modeling module is used to perform task graph modeling for training tasks and inference tasks respectively; The region segmentation module is used to segment the task graph into regions based on network indicator data and combined with task graph features, to obtain multiple segmented sub-graphs and determine the target region labels. The scheduling decision module is used to determine the execution area and target logical computing unit of each segmented subgraph based on the target area label and the logical computing unit capability information. The cross-domain communication control module is used to determine the cross-domain communication topology and communication strategy during the execution of segmented subgraphs; The execution feedback and optimization module is used to adjust scheduling parameters based on training throughput, inference latency, resource utilization, and network status.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the AI ​​large model cross-domain hybrid training and hybrid inference collaborative scheduling method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the AI ​​large model cross-domain hybrid training and hybrid inference collaborative scheduling method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the AI ​​large model cross-domain hybrid training and hybrid inference collaborative scheduling method as described in any one of claims 1 to 6.

Images