A cloud-native containerized AI infrastructure software development method and system

Abstract

This invention relates to the field of cloud-native containerized software technology, and discloses a cloud-native containerized AI basic software development method and system. The method includes: collecting iterative time-series data of all AI training containers on the same GPU node to generate a container stage time-series record table; calculating the GPU computation stage time overlap rate and the communication stage time overlap rate of each container pair within the node to generate a two-dimensional stage conflict matrix; injecting the communication time fluctuation amplitude into the peak-shifting scheduling scheme of the integer programming constraint solution stage and executing it; generating a replica pair communication heat matrix, detecting communication mode change points and extracting the current effective communication mode snapshot; introducing two-dimensional stage conflict evaluation constraints to generate a container migration scheme; and re-converging scheduling and topology after migration.

Description

Technical Field

[0001] This invention relates to the field of cloud-native containerized software technology, specifically to a cloud-native containerized AI infrastructure software development method and system. Background Technology

[0002] In a cloud-native containerized AI infrastructure software platform, multiple distributed AI training tasks are scheduled to run on GPU nodes in the cluster in a containerized form. Each training container alternately executes the local GPU computing phase and the cross-node communication phase during the iteration process. The AI ​​infrastructure software is responsible for allocating resource quotas for multiple containers on the same GPU node and maintaining the communication routing mapping between container replicas.

[0003] Existing cloud-native containerized AI infrastructure software executes intra-node GPU resource scheduling and cross-node communication topology management as independent functional modules. The intra-node resource scheduling module, when coordinating the GPU usage sequence of multiple containers, treats the communication phase as a fixed-duration non-GPU phase, failing to recognize the uncertainty in phase duration caused by dynamic routing fluctuations in communication latency. This leads to frequent failures of off-peak scheduling schemes during communication phase fluctuations. The cross-node communication topology management module, when optimizing inter-replica communication links, does not assess the GPU phase overlap of existing containers on the target node. This may migrate communication hotspot replicas to nodes with severely contested GPU resources, introducing computational contention bottlenecks while eliminating communication bottlenecks. The lack of joint coordination between these two modules, operating independently, prevents cloud-native AI infrastructure software from effectively addressing the intertwined problems of intra-node resource contention and cross-node communication bottlenecks simultaneously. Summary of the Invention

[0004] This invention provides a cloud-native containerized AI infrastructure software development method and system, which solves the technical problems in related technologies, such as mutual interference between the GPU computing stage and the cross-node communication stage in AI training clusters, reduced training efficiency due to resource contention between containers, and mismatch between communication topology and load distribution.

[0005] This invention discloses a joint optimization method for resource scheduling and communication topology, comprising: collecting iterative time-series data of AI training containers on GPU nodes, marking the start and end timestamps and resource consumption of the GPU computation phase and cross-node communication phase in each iteration of each container, and generating a container phase time-series record table; Based on the container phase timing record table, calculate the GPU computing phase time overlap rate and communication blocking time overlap rate of each container pair to generate a two-dimensional phase conflict matrix. The fluctuation amplitude of the cross-node communication time sequence of each container is obtained, and the fluctuation amplitude is injected as an uncertainty parameter into the peak-shifting scheduling scheme in the integer planning constraint solution stage. The corresponding controllable waiting time is injected before the data prefetching queue of each container is dequeued. The amount of cross-node communication data between containers is recorded iteratively to generate a replica pair communication heat matrix, and a snapshot of the current effective communication mode is extracted through change point detection. Based on the matching of the communication mode snapshot and the cluster interconnection topology, the evaluation constraints of the two-dimensional stage conflict matrix are introduced to screen candidate migration target nodes and output container migration schemes. After the migration is performed, the two-dimensional stage conflict matrix is ​​updated for the affected nodes and the stage staggered scheduling scheme is solved again.

[0006] Furthermore, the generation of the two-dimensional stage conflict matrix includes: computing the node content container. With container GPU computation phase time overlap rate : in, and Containers In the The start and end timestamps of the GPU computation phase in each iteration. To count the number of iterations within the statistics window; calculate the container. Communication phases and containers Overlap rate of communication blocking time between GPU computing stages : in, and Containers In the The start and end timestamps of the cross-node communication phase in the next iteration; each element in the two-dimensional phase conflict matrix is ​​composed of... and A binary tuple.

[0007] Furthermore, the peak-shifting scheduling scheme in the solution phase includes: applying an exponentially weighted moving average algorithm to the cross-node communication time sequence of each container to output a smoothed estimate of the communication time of each container. and fluctuation range value ,in The exponentially weighted standard deviation of the difference between the actual communication time and the smoothed estimate is used to solve for the micro-latency of each container with the objective of minimizing the peak instantaneous total demand of the node-level GPU. The objective function is: in, The total number of containers on the node. For containers Apply delay After the time GPU resource requirement function, To optimize the time window length, the constraints include: and It is an integer; , For containers Maximum allowed end-to-end time increment; , This represents the uncertainty tolerance coefficient. For containers The duration of a single iteration cycle, For containers The time consumed during the GPU computing phase.

[0008] Furthermore, the uncertainty tolerance coefficient Adaptive adjustment based on node communication fluctuation levels: Calculate the average fluctuation amplitude of all container communication time on the node; when this average exceeds a preset fluctuation threshold, increase the value. The value of is reduced when the mean is lower than the fluctuation threshold. The value of .

[0009] Furthermore, the replica pair communication heat matrix is ​​a symmetric matrix, and its elements... Indicates a copy With copies The communication heat value between them is processed by an exponentially weighted moving average. and The cluster-level container replica index is used for the change point detection, which adopts a dual-threshold confirmation strategy: after the change point is detected for the first time, a preset number of confirmation iterations are waited for. If the communication heat matrix of the replica continues to deviate from the pattern before the change point within the number of confirmation iterations and the deviation exceeds the confirmation threshold, the change point is officially confirmed as valid and a snapshot of the current valid communication pattern is extracted.

[0010] Furthermore, the evaluation constraint of introducing a two-dimensional stage conflict matrix to screen candidate migration target nodes includes: identifying replica pairs with the highest communication popularity based on the communication mode snapshot, and determining high-popularity replica pairs that cross high-latency links as candidate migration objects; for each candidate migration object, selecting a set of nodes in the cluster interconnection topology diagram that are within the range of low-latency links with its communication peer replica as candidate migration target nodes; for each candidate migration target node, simulating the insertion of the stage time series data of the replica to be migrated into the existing time series of the node, recalculating the elements of the two-dimensional stage conflict matrix involving the replica, and excluding candidate migration target nodes whose stage conflict increment in any dimension exceeds the corresponding threshold; and selecting the target node with the lowest communication cost between the replica to be migrated and its high-popularity communication peer from the remaining candidate migration target nodes to generate a container migration scheme, wherein the communication cost is calculated based on the link hop count and link bandwidth.

[0011] Furthermore, it also includes a migration benefit assessment: calculating the expected reduction in cross-node communication latency after migration and comparing it with the training interruption cost caused by container pauses during migration; executing the container migration scheme only when the cross-node communication latency benefit exceeds the training interruption cost, otherwise keeping the current topology unchanged and marking the replica pair as an observation state.

[0012] Furthermore, the step of updating the two-dimensional stage conflict matrix and resolving the stage staggered scheduling scheme for the affected nodes after the migration includes: re-collecting the container stage timing record tables for both the migration target node and the migration source node; recalculating the two-dimensional stage conflict matrix for the migration target node based on the updated container stage timing record tables; determining whether there are any elements in the updated two-dimensional stage conflict matrix that exceed the conflict threshold, and if so, resolving the stage staggered scheduling scheme for the migration target node; and the new stage staggered scheduling scheme taking effect uniformly when all affected containers complete the current iteration and enter the iteration gap.

[0013] Furthermore, the injection point of the controllable waiting time is the dequeue operation trigger point of the container data prefetch queue. When the container data prefetch queue is ready to dequeue a batch of training data to start the next iteration of GPU computing, the dequeue operation is intercepted and the corresponding micro-delay waiting time is inserted. After the micro-delay waiting time ends, the dequeue operation is executed normally.

[0014] This invention discloses a resource scheduling and communication topology joint optimization system for executing the above-mentioned resource scheduling and communication topology joint optimization method, including: a container stage time series acquisition module, used to acquire iterative time series data of AI training containers on GPU nodes, mark the start and end timestamps and resource consumption of the GPU computing stage and cross-node communication stage in each iteration of each container, and generate a container stage time series record table; The two-dimensional conflict matrix generation module is used to calculate the GPU computing phase time overlap rate and communication blocking time overlap rate of each container pair based on the container phase timing record table, and generate a two-dimensional phase conflict matrix. The robust off-peak scheduling module is used to obtain the fluctuation range of the cross-node communication time sequence of each container, inject the fluctuation range as an uncertainty parameter into the off-peak scheduling scheme in the integer programming constraint solution stage, and inject the corresponding controllable waiting time before the data prefetching queue of each container is dequeued. The communication pattern monitoring module is used to iteratively record the amount of cross-node communication data between containers to generate a copy pair communication heat matrix, and extract the current effective communication pattern snapshot through change point detection; The joint optimization migration module is used to match the communication mode snapshot with the cluster interconnection topology map, introduce the evaluation constraints of the two-dimensional stage conflict matrix to screen candidate migration target nodes, and output container migration scheme. The post-migration coordination and convergence module is used to update the two-dimensional stage conflict matrix of the affected nodes and resolve the stage peak-shifting scheduling scheme after the container migration scheme is executed.

[0015] The present invention has the following beneficial effects: In solving the phased peak scheduling scheme, the fluctuation range of communication latency is introduced as an uncertainty parameter and injected into the integer programming constraints to form a robust margin. Since the phased peak scheduling scheme has reserved a tolerance space for the fluctuation of communication phase duration during the solution phase, when the communication latency changes due to dynamic routing fluctuations in actual operation, the allocated micro-latency can still ensure that the GPU-intensive phases of each container do not overlap in time, overcoming the problem of frequent failure of the phased peak scheduling scheme caused by treating the communication phase as a fixed latency.

[0016] A two-dimensional stage conflict assessment constraint is introduced into the communication topology migration decision. The stage conflict increment after adding the replica to be migrated is calculated for each candidate migration target node, and candidate migration target nodes with excessively high stage conflict increments are excluded. Since the selection of target nodes in the container migration scheme considers both communication costs and the degree of competition for computing resources, it avoids the problem of introducing new bottlenecks by migrating replicas to nodes where GPU resources are already severely contested during communication optimization.

[0017] By re-triggering the timing data acquisition of the affected nodes after migration, updating the two-dimensional stage conflict matrix, and resolving the stage staggered scheduling scheme, the intra-node scheduling and cross-node topology optimization remain in a coordinated state after the migration event occurs, avoiding the problem of the two types of optimization gradually deviating in the dynamic operating environment. Attached Figure Description

[0018] Figure 1This is a flowchart of the resource scheduling and communication topology joint optimization method for cloud-native containerized AI infrastructure software provided in this embodiment of the invention; Figure 2 This is a schematic diagram of the temporal distribution of each container of node A in the third iteration stage provided in an embodiment of the present invention; Figure 3 This is a schematic diagram showing the comparison of GPU resource usage in each container of node A during the third iteration, provided in an embodiment of the present invention. Figure 4 This is a schematic diagram of a two-dimensional stage conflict matrix heatmap (node ​​A, K=5 iterations) provided in an embodiment of the present invention; Figure 5 This is a schematic diagram comparing the smoothed estimate of communication time of each container with the fluctuation amplitude provided in the embodiments of the present invention; Figure 6 This is a schematic diagram of the micro-delay allocation of the off-peak scheduling scheme provided in the embodiment of the present invention; Figure 7 This is a schematic diagram of the cross-node communication heat distribution of replicas provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the communication heat iteration sequence and change point detection between replica A and replica D provided in an embodiment of the present invention; Figure 9 This is a schematic diagram illustrating the calculation of overlap rate changes during the scheduling convergence process of node C after migration, provided in an embodiment of the present invention. Detailed Implementation

[0019] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0020] At least one embodiment of the present invention discloses a method for joint optimization of resource scheduling and communication topology for cloud-native containerized AI infrastructure software, see [link to relevant documentation]. Figure 1 This includes the following steps: Step 1: Collect timing data for the container phase.

[0021] Acquire iterative time-series data of all AI training containers on the same GPU node, label the start and end timestamps of the local GPU computing phase and cross-node communication phase in each iteration of each container, as well as the resource consumption of each phase, and generate a container phase time-series record table.

[0022] It should be noted that each record in the container phase timing record table includes the container identifier, iteration number, phase type marker, phase start timestamp, phase end timestamp, and GPU memory usage and GPU computing power utilization within that phase. The phase type marker distinguishes between GPU computation phases and cross-node communication phases. The granularity of timing data collection is at the phase level within a single iteration; that is, each container generates two records in a complete iteration, corresponding to its computation phase and communication phase, respectively.

[0023] Step 2: Generate a two-dimensional stage conflict matrix.

[0024] Based on the container phase timing record table, the time overlap rate of the GPU computing phase of each container pair within the compute node, and the time overlap rate of the communication phase of each container with the GPU computing phase of other containers on the same node, a two-dimensional phase conflict matrix containing the computing overlap dimension and the communication blocking dimension is generated.

[0025] It should be noted that calculating the overlap dimension means: for any two containers on a node... and Its GPU computing phase time overlap rate The definition is as follows: in, and Containers In the The start and end timestamps of the GPU computation phase in each iteration. and Containers In the The start and end timestamps of the GPU computation phase in each iteration. To count the number of iterations within the window, The iteration number and , and This is the index of the node's contents. The numerator takes the larger value than zero for each iteration to ensure that the contribution is zero when there is no overlap between two phases. Both the numerator and denominator are in time units; the result of dividing them is... It is a dimensionless ratio.

[0026] The meaning of the communication blocking dimension is: for containers Communication phases and containers Time overlap between GPU computing stages The calculation method is as follows: container in formula The time interval is replaced with the start and end timestamps of its communication phase. and ,container The time interval is maintained as the start and end timestamps of its GPU computation phase. and The denominator is replaced with a container. Cumulative value of communication phase time ,Right now: in, and Containers In the The start and end timestamps of the cross-node communication phase in this iteration are both derived from the corresponding records in the container phase timing record table marked as cross-node communication phases. The two-dimensional phase conflict matrix is ​​generated by... and Together, they form a two-dimensional stage conflict matrix, where each element is a binary tuple.

[0027] Step 3: Solve the staggered peak scheduling scheme.

[0028] An exponentially weighted moving average algorithm is applied to the cross-node communication time series of each container to obtain the fluctuation range of communication time. The fluctuation range is then injected into the integer programming constraint as a stage duration uncertainty parameter to solve the stage staggered scheduling scheme with robust margin.

[0029] Step 3 may specifically include: Step 301: Extract the cross-node communication stage time sequence of each container in the container stage time sequence record table for continuous iterations. Apply the exponential weighted moving average algorithm to the cross-node communication stage time sequence to output a smoothed estimate of the communication time of each container. and fluctuation range value .in, For the exponentially weighted moving average algorithm for containers Smoothed output value of the communication latency sequence. This is the exponentially weighted standard deviation of the difference between the actual communication time and the smoothed estimate. Index the contents of the node.

[0030] Step 302: With the objective of minimizing the peak instantaneous total demand of the node-level GPU, and constrained by the upper limit of the end-to-end time increment for each container iteration and the tolerance for communication time uncertainty, solve for the micro-latency of each container. The objective function and constraints of the integer programming are as follows: Objective function: in, The total number of containers on the node. To be assigned to a container The micro-delay (integer time units). Let be the set of decision variables consisting of the micro-latencies of all containers. For moments on the timeline, To optimize the time window length, the duration of a single iteration cycle for all containers is taken. The maximum value, i.e. , For containers Applying delay After the time The GPU resource requirement function, whose value is: when time Falling into the container During the GPU computing phase interval, It equals the GPU computing power utilization of the container at this stage; otherwise Take zero.

[0031] Constraints: in, Represents a set of integers. For containers Maximum allowed end-to-end time increment The container obtained in step 301 Communication latency fluctuation range This represents the uncertainty tolerance coefficient. For containers The duration of a single iteration cycle, For containers The time consumed in the GPU computing phase For containers A smoothed estimate of communication time. All units in the above constraints are time units. Since the coefficients are dimensionless, dimensional consistency is guaranteed. The third constraint ensures that even if the communication phase is extended... The magnitude of the delay is such that the container, even after applying a micro-delay, does not exceed its iteration period boundary. Solve this integer programming problem and output the micro-delay for each container. The phased staggered scheduling scheme is constituted.

[0032] In this embodiment of the application, in order to ensure that the phased peak-shifting scheduling scheme remains effective when communication latency fluctuates significantly, the uncertainty tolerance coefficient is... It can adaptively adjust based on the overall level of communication fluctuations at the current node. Specifically, it calculates the average fluctuation amplitude of communication time across all containers on the node; when this average exceeds a preset fluctuation threshold, it increases... The value is chosen to expand the range of the robustness margin; when the mean is below the fluctuation threshold, the range is narrowed. The value of is chosen to reduce unnecessary scheduling constraints.

[0033] Step 4, staggered scheduling during the execution phase.

[0034] Based on the micro-latency allocated in the phased off-peak scheduling scheme, a controllable waiting time is injected before the dequeue operation of each container's data prefetch queue, so that the GPU-intensive phases of each container on the same GPU node are alternately distributed on the time axis.

[0035] It should be noted that the controllable waiting time is injected at the dequeue operation trigger point of the container's data prefetch queue. When the container's data prefetch queue is ready to dequeue a batch of training data to start the next iteration of GPU computation, the scheduling module intercepts the dequeue operation and inserts the corresponding micro-latency waiting time. After the micro-delay waiting period ends, the dequeue operation executes normally, and the container enters the GPU computing phase. This method does not modify the training logic inside the container; it only achieves phase staggering by controlling the iteration start time.

[0036] Step 5: Generate a copy pair communication heat matrix.

[0037] The amount of cross-node communication data sent by each training container to the remote replica is recorded iteratively. The data is accumulated and statistically analyzed according to the container replica pair dimension. The communication volume sequence within the continuous iteration window is smoothed by applying an exponentially weighted moving average algorithm to generate a replica pair communication heat matrix.

[0038] It should be noted that the replica pair communication heat matrix is ​​a symmetric matrix, and the row and column indices of the replica pair communication heat matrix correspond to the identifiers of all container replicas participating in distributed training in the cluster. Elements in the replica pair communication heat matrix... Indicates a copy With copies The smoothed communication heat values ​​between them, of which and This is a cluster-level container replica index. The input to the exponentially weighted moving average algorithm is replica pairs. The output of the original communication data sequence in continuous iterations is the smoothed communication heat value of the replica pair. .

[0039] Step 6: Detect communication mode change points.

[0040] A time-series change point detection algorithm is applied to the replica pair communication heat matrix to identify the time points when structural changes occur in the communication heat distribution. When a change point event is detected, the replica pair communication heat matrix within the stable window period after the change point is extracted as a snapshot of the current effective communication pattern.

[0041] It should be noted that the input of the time series change point detection algorithm is the matrix sequence formed by the replica pair communication heat matrix as it progresses through iterations, and the output is the time point marker of structural changes. The stable window period is the interval of several consecutive iterations after the change point. When the change amplitude of the elements of the replica pair communication heat matrix is ​​lower than the stability threshold within this interval, the mean of the replica pair communication heat matrix within this interval is taken as the snapshot of the current effective communication mode.

[0042] In this embodiment of the application, in order to avoid unnecessary topology migration triggered by short-term communication fluctuations, the change point detection adopts a dual threshold confirmation strategy: when a change point is detected for the first time, the subsequent migration process is not triggered immediately, but waits for a preset number of confirmation iterations. If the communication heat matrix of the replica continues to deviate from the pattern before the change point within the number of confirmation iterations and the deviation exceeds the confirmation threshold, then the change point is officially confirmed to be valid and a snapshot of the current valid communication pattern is extracted.

[0043] Step 7: Generate a container migration scheme that optimizes both topology and load.

[0044] The current effective communication mode snapshot and cluster interconnection topology graph are input into the incremental graph matching algorithm. A two-dimensional stage conflict assessment constraint is introduced in the candidate migration target node screening, and a container migration scheme with joint optimization of topology and load is output.

[0045] Step 7 may specifically include: Step 701: Identify communication heat value based on the current valid communication mode snapshot. The top-ranked replica pairs are identified. The physical node locations of these high-latency replica pairs are determined, and those crossing high-latency links are selected as candidate migration targets.

[0046] Step 702: For each candidate migration target, select a set of nodes in the cluster interconnection topology graph that are within the low-latency link range of its communication peer replica as candidate migration target nodes.

[0047] Step 703: For each candidate migration target node, based on the current container phase timing record table and the two-dimensional phase conflict matrix of the candidate migration target node, calculate the phase conflict increment generated after adding the replica to be migrated. The phase conflict increment is calculated as follows: simulate insert the phase timing data of the replica to be migrated into the existing timing of the candidate migration target node, recalculate all elements involving the replica in the two-dimensional phase conflict matrix, and compare the sum of the newly calculated overlap rate and the sum of the communication blocking rate with the corresponding thresholds respectively. Exclude candidate migration target nodes whose phase conflict increment in any dimension exceeds the corresponding threshold.

[0048] Step 704: Among the remaining candidate migration target nodes, select the target node with the lowest communication cost between the replica to be migrated and its most frequently communicated counterpart, and generate a container migration plan. The communication cost is calculated based on the number of hops and the link bandwidth from the source node to the target node in the cluster interconnection topology diagram.

[0049] In this embodiment, to reduce the impact of migration frequency on training stability, step 7 further includes migration benefit evaluation: calculating the expected reduction in cross-node communication latency after migration and comparing it with the training interruption cost caused by container pause during migration. The container migration scheme is executed only if the cross-node communication latency benefit exceeds the training interruption cost; otherwise, the current topology is maintained and the copy pair is marked as an observation state.

[0050] Step 8: Reconverge scheduling and topology after migration.

[0051] After executing the container migration scheme, the phase time series data of each container on the affected nodes are re-collected under the new topology, the two-dimensional phase conflict matrix is ​​updated, and the phase off-peak scheduling scheme is re-executed for nodes whose conflicts have intensified due to the addition of new replicas, so that the node-level phase off-peak scheduling and cross-node topology optimization converge to a coordinated state after the migration event.

[0052] Step 8 may specifically include: Step 801: After the container migration is completed, the collection operation of step 1 is re-executed on both the target node and the source node to obtain the container phase timing record table after migration.

[0053] Step 802: Based on the updated container phase timing record table, re-execute the calculation operation of step 2 on the migration target node to update the two-dimensional phase conflict matrix of the migration target node.

[0054] Step 803: Determine if there are any elements in the updated two-dimensional stage conflict matrix that exceed the conflict threshold. If so, re-execute the solution operation in step 3 for the migration target node to obtain a new stage peak-shifting scheduling scheme and execute it according to step 4. If not, maintain the current scheduling state of the migration target node.

[0055] In this embodiment of the application, in order to avoid conflicts between the rescheduling process after migration and the ongoing training iteration, the new phase staggered scheduling scheme in step 803 is switched uniformly when all affected containers have completed the current iteration and entered the iteration gap, rather than being forcibly changed in the middle of the iteration execution.

[0056] This invention introduces the communication latency fluctuation amplitude as an uncertainty parameter in the solution of the phased peak scheduling scheme, injecting the communication latency fluctuation amplitude into the integer programming constraints to form a robust margin. Since the phased peak scheduling scheme has reserved a tolerance space for communication phase duration fluctuations during the solution phase, when the communication latency changes due to dynamic routing fluctuations in actual operation, the allocated micro-latency can still ensure that the GPU-intensive phases of each container do not overlap in time, overcoming the problem of frequent failures of the phased peak scheduling scheme caused by treating the communication phase as a fixed latency.

[0057] This implementation introduces a two-dimensional stage conflict assessment constraint in the communication topology migration decision. It calculates the stage conflict increment after adding the replica to the candidate migration target node and excludes candidate migration target nodes with excessively high stage conflict increments. Since the target node selection in the container migration scheme considers both communication costs and the degree of competition for computing resources, it avoids the problem of introducing new bottlenecks by migrating replicas to nodes where GPUs are already heavily contested during communication optimization.

[0058] This implementation method re-triggers the timing data acquisition of the affected nodes after migration, updates the two-dimensional stage conflict matrix, and re-solves the stage staggered scheduling scheme. This ensures that intra-node scheduling and cross-node topology optimization remain in a coordinated state after the migration event occurs, avoiding the problem of the two types of optimization gradually deviating in the dynamic operating environment.

[0059] On the other hand, this invention also proposes a joint optimization system for resource scheduling and communication topology of cloud-native containerized AI infrastructure software, including: The container phase timing acquisition module is used to acquire iterative time series data of all AI training containers on the same GPU node, mark the start and end timestamps of the local GPU computing phase and cross-node communication phase in each iteration of each container, as well as the resource consumption of each phase, and generate a container phase timing record table.

[0060] The two-dimensional conflict matrix generation module is used to calculate the GPU computation phase time overlap rate of each container pair within a node and the time overlap rate between each container communication phase and the GPU computation phase of other containers on the same node based on the container phase time sequence record table, and generate a two-dimensional phase conflict matrix.

[0061] The robust off-peak scheduling module is used to obtain the fluctuation range of communication time for cross-node communication time sequence of each container, inject the fluctuation range as an uncertainty parameter into the off-peak scheduling scheme in the solution stage of integer programming constraints, and inject the corresponding controllable waiting time before the dequeue operation of the data prefetch queue of each container.

[0062] The communication pattern monitoring module is used to iteratively record the amount of cross-node communication data sent by each training container to the remote replica and generate a replica pair communication heat matrix. The time series change point detection algorithm is applied to the replica pair communication heat matrix to identify structural changes in communication patterns and extract a snapshot of the current effective communication pattern.

[0063] The joint optimization migration module is used to perform incremental graph matching between the current effective communication mode snapshot and the cluster interconnection topology. It introduces a two-dimensional stage conflict assessment constraint in the selection of candidate migration target nodes, excludes candidate migration target nodes whose stage conflict increment exceeds the threshold, selects the target node with the best communication cost among the remaining candidate migration target nodes, and outputs the container migration scheme.

[0064] The post-migration coordination and convergence module is used to re-collect the phase time series data of the affected nodes after the container migration scheme is executed, update the two-dimensional phase conflict matrix, and re-solve the phase peak-shifting scheduling scheme for nodes with intensified phase conflict increments, so that node-level scheduling and cross-node topology optimization can re-converge to a coordinated state.

[0065] By way of example, the system proposed in this invention may also include various features and combinations thereof in the method embodiments, which will not be elaborated here.

[0066] See Figures 2-9 The following application examples are proposed: An AI computing cluster hosts multiple large-scale distributed training tasks. On a GPU node (Node A) within the cluster, four training containers (Container 1 to Container 4) run simultaneously, each carrying a different replica shard of the same distributed training task. Each container alternates between local GPU computation and cross-node communication phases in each iteration. The platform performs joint optimization on the four containers on Node A, while simultaneously monitoring the communication heat between Node A and its replicas on other nodes in the cluster (Node B and Node C) to eliminate the intertwining issues of intra-node GPU resource contention and cross-node communication bottlenecks.

[0067] The platform collects time-series data of containers 1 to 4 on node A during continuous iterations. Each container generates two records per iteration, corresponding to the GPU computation stage and the cross-node communication stage, respectively, recording the start and end timestamps, GPU memory usage, and GPU computing power utilization for each stage. Taking the third iteration as an example, the collection results are as follows.

[0068] Table 1. Container Phase Timing Record Table (3rd Iteration, Node A) As shown in Table 1, the GPU computing phases of containers 1 to 3 started at similar times and overlapped significantly, providing the initial data foundation for subsequent conflict matrix calculations. The time-series data of container 4 was collected synchronously, forming a complete input together with the records mentioned above.

[0069] Based on the container phase timing record table, the container pairs on platform compute node A... (Calculate the overlap dimension) and (Communication blocking dimension). Taking containers 1 and 2 as examples: the GPU computation phases of both overlapped in the third iteration from millisecond 5 to 118, lasting 113ms; container 1's computation phase took 120ms, indicating a high overlap rate. The communication phase of container 1 (from millisecond 120 to 175) and the GPU computation phase of container 2 (from millisecond 5 to 118) had no time overlap, resulting in a zero communication blocking rate. Summary statistics window ( After several iterations, the results of the two-dimensional stage conflict matrix are as follows.

[0070] Table 2 Two-Dimensional Stage Conflict Matrix (Node A, (next iteration) The matrix shows that the computational overlap rate between containers 1 and 3 all exceeds 0.85, which is a region of severe computational contention. The communication blocking rate between container 4 and other containers is relatively high, indicating that the communication phase of container 4 frequently overlaps with the GPU computation phase of other containers, causing interference to GPU computation.

[0071] The platform extracts the time sequence of cross-node communication stages for five consecutive iterations of each container, applies an exponentially weighted moving average algorithm, and outputs a smoothed estimate for each container. and fluctuation range value .in Indicates the first A smoothed estimate of the communication time of each container. This indicates the corresponding fluctuation range. All containers in the current node A. The average value is 9.2ms, which exceeds the preset fluctuation threshold of 8ms, therefore the uncertainty tolerance coefficient is... Adaptively adjusted to version 1.5. The platform adapts to each container's... , Calculation phase time Iteration cycle and maximum allowable delay Given the input, solve for the constraint using integer programming. micro delay Output the off-peak scheduling scheme for each container, and the results are as follows.

[0072] Table 3. Communication Time Statistics and Peak-Shaving Scheduling Scheme After solving the integer programming problem, the micro-latency of containers 1 to 4 is staggered by about 18ms, so that the GPU computing stages of the four containers are distributed sequentially on the time axis, avoiding simultaneous competition for GPU resources.

[0073] The platform injects controllable waiting times of 18ms, 36ms, and 54ms respectively at the data prefetching queue dequeue trigger points of containers 2, 3, and 4. After injection, container 1 starts GPU computing first, container 2 starts approximately 18ms after container 1's computing phase, and containers 3 and 4 follow in sequence. No modifications are made to the training logic of each container; only the iteration start time is controlled and delayed. The peak instantaneous total GPU demand of node A is reduced from almost simultaneous full load of all four containers to a gradual increase in the computing power utilization of a single container, effectively suppressing GPU resource competition within the node.

[0074] The platform iteratively records the amount of communication data sent by each container on node A to other replicas in the cluster. This data is accumulated along the replica pair dimension and smoothed using an exponentially weighted moving average to generate a replica pair communication heat matrix. The cluster involves a total of 6 replicas (replicas A to F, distributed across nodes A, B, and C). Some elements of the heat matrix are shown below.

[0075] Table 4. Replica Pair Communication Heat Matrix (Partial, Unit: MB / iteration) The highest heat value (312.40MB / iteration) is between replica A and replica D, and they belong to node A and node B respectively. The results of subsequent change point detection need to be monitored.

[0076] The platform applies a time-series change point detection algorithm to the iterative sequence of the replica pair communication heat matrix. Around the 47th iteration, a structural jump occurs in the heat value between replica A and replica D (from approximately 180MB / iteration to over 312MB / iteration). The platform activates a dual-threshold confirmation strategy, waiting for 8 confirmation iterations. Within the confirmation window, if the heat value continuously deviates from the pre-change pattern and the deviation exceeds the confirmation threshold, the change point is officially confirmed. The platform extracts the mean of the replica pair communication heat matrix from the 47th to the 55th iteration window as a snapshot of the current effective communication pattern, marking replica A and replica D as a high-heat cross-node replica pair, and proceeds to the next step.

[0077] The platform inputs a snapshot of the current effective communication pattern and the cluster interconnection topology graph into an incremental graph matching algorithm. Replica A is currently located at node A, and its high-traffic communication peer, replica D, is located at node B. The link between the two has 3 hops, which is a high-latency link. The candidate migration target is node C (the link between node C and node B has 1 hop).

[0078] The platform performs a two-dimensional phase conflict incremental assessment on node C: it simulates and inserts the phase time series data of replica A into the existing time series of node C, and recalculates the conflicts involving replica A. and The results are as follows.

[0079] Table 5. Assessment of Conflict Increment at Candidate Migration Target Nodes The conflict increments in both dimensions of node C did not exceed the corresponding thresholds, so it was retained as a valid candidate target node. The platform further evaluated the migration benefits: after replica A migrated to node C, the link hop count with replica D decreased from 3 hops to 1 hop. The expected communication latency benefit exceeded the cost of pausing the container migration, so the migration plan was confirmed to be executed, and replica A was migrated from node A to node C.

[0080] After replica A is migrated, the platform re-collects container phase time-series data for both the target node C and the source node A. Because node C has added replica A, the two-dimensional phase conflict matrix shows the relationship between replica A and the existing containers. The conflict rate is 0.52, exceeding the conflict threshold of 0.40, triggering a staggered scheduling scheme for the re-solution phase. The platform waits for all affected containers on node C to complete their current iteration and enter the iteration gap before uniformly switching to the new low-latency allocation scheme, thus redistributing the GPU computation phase of node C. Node A, due to the departure of replica A, has all its conflict matrix elements below the threshold, maintaining its current scheduling state.

[0081] Table 6. Scheduling convergence status of node C after migration The entire data flow starts from the container phase time-series record table collected in step 1. Step 2 transforms this into a two-dimensional phase conflict matrix that quantifies conflict intensity. Step 3 generates a micro-latency scheme with robust margins by utilizing communication latency fluctuations. Step 4 injects the micro-latency into the data prefetch queue to achieve intra-node peak shifting. Simultaneously, steps 5 and 6 continuously monitor cross-node communication heat and generate effective communication pattern snapshots after confirming structural changes. Step 7 combines the snapshots with the cluster topology map and uses the two-dimensional conflict matrix as a constraint to filter target nodes, outputting a migration scheme that balances communication cost and computational load. Step 8, after migration, uses updated time-series data to drive the conflict matrix and scheduling scheme to reconverge, ensuring that intra-node scheduling and cross-node topology optimization remain coordinated throughout the entire dynamic operation.

[0082] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A cloud-native, containerized AI infrastructure software development method, characterized in that, include: Collect iterative time-series data of AI training containers on GPU nodes, label the start and end timestamps and resource consumption of the GPU computing phase and cross-node communication phase in each iteration of each container, and generate a container phase time-series record table. Based on the container phase timing record table, calculate the GPU computing phase time overlap rate and communication blocking time overlap rate of each container pair to generate a two-dimensional phase conflict matrix. The fluctuation amplitude of the cross-node communication time sequence of each container is obtained, and the fluctuation amplitude is injected as an uncertainty parameter into the peak-shifting scheduling scheme in the integer planning constraint solution stage. The corresponding controllable waiting time is injected before the data prefetching queue of each container is dequeued. The amount of cross-node communication data between containers is recorded iteratively to generate a replica pair communication heat matrix, and a snapshot of the current effective communication mode is extracted through change point detection. Based on the matching of the communication mode snapshot and the cluster interconnection topology, the evaluation constraints of the two-dimensional stage conflict matrix are introduced to screen candidate migration target nodes and output container migration schemes. After the migration is performed, the two-dimensional stage conflict matrix is ​​updated for the affected nodes and the stage staggered scheduling scheme is solved again.

2. The cloud-native containerized AI infrastructure software development method according to claim 1, characterized in that, The generation of the two-dimensional stage conflict matrix includes: Compute node contents With container GPU computation phase time overlap rate : in, and Containers In the The start and end timestamps of the GPU computation phase in each iteration. This represents the number of iterations within the statistics window; Computational Containers Communication phases and containers Overlap rate of communication blocking time between GPU computing stages : in, and Containers In the The start and end timestamps of the cross-node communication phase in the next iteration; each element in the two-dimensional phase conflict matrix is ​​composed of... and A binary tuple.

3. The cloud-native containerized AI infrastructure software development method according to claim 1, characterized in that, The peak-shifting scheduling scheme for the solution phase includes: Applying an exponentially weighted moving average algorithm to the cross-node communication time series of each container, a smoothed estimate of the communication time of each container is output. and fluctuation range value ,in It is the exponentially weighted standard deviation of the difference between the actual communication time and the smoothed estimate; The goal is to solve for the micro-latency of each container by minimizing the peak instantaneous total demand of the node-level GPU. The objective function is: in, The total number of containers on the node. For containers Apply delay After the time GPU resource requirement function, To optimize the time window length; The constraints include: and It is an integer; , For containers Maximum allowed end-to-end time increment; , This represents the uncertainty tolerance coefficient. For containers The duration of a single iteration cycle, For containers The time consumed during the GPU computing phase.

4. The cloud-native containerized AI infrastructure software development method according to claim 3, characterized in that, The uncertainty tolerance coefficient Adaptive adjustment based on node communication fluctuation levels: The average value of the fluctuation in communication time across all containers on the compute node is used to calculate the value. When this average value exceeds a preset fluctuation threshold, the value is increased. The value of is reduced when the mean is lower than the fluctuation threshold. The value of .

5. The cloud-native containerized AI infrastructure software development method according to claim 1, characterized in that, The replica pair communication heat matrix is ​​a symmetric matrix, and its elements are... Indicates a copy With copies The communication heat value between them is processed by an exponentially weighted moving average. and For cluster-level container replica indexes; The change point detection adopts a dual threshold confirmation strategy: after the change point is detected for the first time, it waits for a preset number of confirmation iterations. If the communication heat matrix of the replica continues to deviate from the pattern before the change point within the number of confirmation iterations and the deviation exceeds the confirmation threshold, then the change point is officially confirmed as valid and a snapshot of the current valid communication pattern is extracted.

6. The cloud-native containerized AI infrastructure software development method according to claim 1, characterized in that, The evaluation constraint for screening candidate migration target nodes by introducing a two-dimensional stage conflict matrix includes: Based on the communication mode snapshot, the top-ranked replica pairs in terms of communication popularity value are identified, and high-popularity replica pairs that cross high-latency links are determined as candidate migration targets. For each candidate migration target, select a set of nodes in the cluster interconnection topology graph that are within the low-latency link range of its communication peer replica as candidate migration target nodes; For each candidate migration target node, the phase time series data of the replica to be migrated is simulated and inserted into the existing time series of the node. The two-dimensional phase conflict matrix elements involving the replica are recalculated, and candidate migration target nodes whose phase conflict increment in any dimension exceeds the corresponding threshold are excluded. Among the remaining candidate migration target nodes, the target node with the lowest communication cost between the replica to be migrated and its high-frequency communication counterpart is selected to generate a container migration scheme. The communication cost is calculated based on the link hop count and link bandwidth.

7. The cloud-native containerized AI infrastructure software development method according to claim 6, characterized in that, It also includes a migration benefit assessment: Calculate the expected reduction in cross-node communication latency after migration and compare it with the training interruption cost caused by container pause during migration; execute the container migration scheme only if the cross-node communication latency benefit exceeds the training interruption cost, otherwise keep the current topology unchanged and mark the replica pair as an observation state.

8. The cloud-native containerized AI infrastructure software development method according to claim 1, characterized in that, The process of updating the two-dimensional phase conflict matrix of the affected nodes and resolving the phase peak-shifting scheduling scheme after the migration includes: Re-collect the container phase timing record table for both the target node and the source node during migration; The two-dimensional phase conflict matrix of the migration target node is recalculated based on the updated container phase timing record table. Determine whether there are any elements in the updated two-dimensional stage conflict matrix that exceed the conflict threshold. If so, resolve the stage staggered scheduling scheme for the migration target node. The new phased off-peak scheduling scheme takes effect uniformly when all affected containers complete their current iteration and enter the iteration gap.

9. The cloud-native containerized AI infrastructure software development method according to claim 1, characterized in that, The controlled waiting time is injected at the dequeue operation trigger point of the container data prefetch queue. When the container's data prefetch queue is ready to dequeue a batch of training data to start the next iteration of GPU computing, the dequeue operation is intercepted and the corresponding micro-delay waiting time is inserted. After the micro-delay waiting time ends, the dequeue operation is executed normally.

10. A cloud-native containerized AI infrastructure software development system, used to execute the cloud-native containerized AI infrastructure software development method according to any one of claims 1 to 9, characterized in that, include: The container phase timing acquisition module is used to collect iterative time series data of AI training containers on GPU nodes, mark the start and end timestamps and resource consumption of the GPU computing phase and cross-node communication phase in each iteration of each container, and generate a container phase timing record table. The two-dimensional conflict matrix generation module is used to calculate the GPU computing phase time overlap rate and communication blocking time overlap rate of each container pair based on the container phase timing record table, and generate a two-dimensional phase conflict matrix. The robust off-peak scheduling module is used to obtain the fluctuation range of the cross-node communication time sequence of each container, inject the fluctuation range as an uncertainty parameter into the off-peak scheduling scheme in the integer programming constraint solution stage, and inject the corresponding controllable waiting time before the data prefetching queue of each container is dequeued. The communication pattern monitoring module is used to iteratively record the amount of cross-node communication data between containers to generate a copy pair communication heat matrix, and extract the current effective communication pattern snapshot through change point detection; The joint optimization migration module is used to match the communication mode snapshot with the cluster interconnection topology map, introduce the evaluation constraints of the two-dimensional stage conflict matrix to screen candidate migration target nodes, and output container migration scheme. The post-migration coordination and convergence module is used to update the two-dimensional stage conflict matrix of the affected nodes and resolve the stage peak-shifting scheduling scheme after the container migration scheme is executed.

Images